I see it as the hearing that's impaired, not the person. Just as I see the words "visually impaired" to mean that the vision is impaired, not the person. Yet I don't see people with visual impairments go up in arm over the term "visually impaired." I see it as much ado over nothing. Maybe a cultural thing that's running amok?
There's just as much controversy over terminology in the blind community. Some people take issue with the term "visually impaired" for a lot of parallel reasons, like thinking it indicates they have more vision than they do (or any at all if they're total). On the flip side, NFB members who would be considered visually impaired by the medical community often call themselves "blind" because they function as a blind person, in the sense they rely on non-visual techniques to do everyday tasks instead of trying to rely on a small percentage of vision.