It may be that you've encountered many more hearing people who are already familiar with Deaf culture than I have and who see deafness as a cultural choice rather than as a physical or medical situation or disability.
I'm not saying I have more or better experience, just different: but in my experience before, and in my encounters with hearing people since the happy moment when my deaf daughter came into my life, and in what I've read, I've found that the majority of hearing people are unaware that a Deaf culture exists, unaware that there's been a choice made, much less seeing the use of ASL as a rejection of hearing culture. They consider deafness to be a physical, not a cultural state.
It seems that discrimination is made much less on the basis of a cultural choice, and much more on the faulty perception that deaf people are "unable" to perform job duties, to interact as equals, somehow inferior because they are deaf. Not because they are Deaf.
It is common to find cases where hearing aid users are not provided equal access to jobs or opportunities, or that those with typical hearing consider themselves superior to an oral deaf HA wearer -- even if that deaf person is not integrated into Deaf culture or does not self-identify as Deaf. I think audism is as Tom Humphries defined it in the 70s: when a person thinks that one is superior based on one’s ability to hear or behave in the manner of one who hears. It can and does happen even when neither the victim nor the person discriminating or oppressing have a familiarity with or participate in Deaf culture and even to deaf people who don't use ASL.