No, it doesn't annoys me, it happens for a reason, being deaf and you cannot hear the words you're trying to say. There's nothing to be ashamed about or be embarrassed about.
Sometimes there are words that I can't pronounce correctly but my step-mom had helped me with whatever I'm having trouble pronouncing. It's all good. 
