Yeah, most of the time I let the "I'm sorry" slide, but you can tell when someone is giving fake pity. So, I say, "It's not your fault."
"I'm deaf."
"Oh, I'm so sorry." *Fake pity sorry*
"Why? it's not your fault."
They usually don't know what to say after that.