I don't think it is wrong at all to not want to visit your loved one's gravesite, no matter what the reason. It is obviously a very uncomfortable thing to do for many people. Please do not feel bad about it. It's ok. The most important thing is that your loved one is in your thoughts, and that is all that matters. I always keep my late mother in my thought every single day of my life. The day she died, I swore to myself that I would NEVER forget her. And I haven't. I keep a picture of her and me in my wallet, from Christmas 1984. I'll scan it and upload it and put it on here tomorrow.
My mother died on June 28th, 1990, two months shy of my 10th birthday. I have not visited her grave since the summer of 2001 just before I moved out of Wisconsin. That was the first time in about 10 or 11 years that I had visited her grave since Mother's day in 1991. I have not visited her grave ever since. I wish she had been cremated instead, so that I would have an urn I could take anywhere with me when I move anywhere. I am in Texas and am about 3,000 miles away from her grave, so I cannot visit her grave. Also, it is rather uncomfortable for me anyway.
When she died, my heart broke into a million pieces. I miss her to this day with all of my heart, and not a day goes by that I don't think of her.