That does sound frustrating and I can completely understand why you wish your vision would deteriorate.
I have a friend who is legally blind (her visual acuity is "counts fingers") and as she likes to say, "I have just enough vision to get me into trouble." She has often told me how she wished she were totally blind so she wouldn't have to deal with fluctuating vision -- especially when moving from light to dark environments. She also expresses frustration at explaining her vision loss to others. Some people she meets expect her to be able to see everything while others assume she's totally blind.
That's where I'm "lucky." When you tell someone you're totally blind, people understand what that means. When you tell them you're legally blind, it's a bit more difficult to explain since that can mean so many different things and can be impacted by the environment (sunny/cloudy, color contrast/size of objects, lack of depth perception, etc.).
