this happens all the time and I don't feel disrespected nor do I feel they are talking about me. I work in a highly diverse workplace where we do have many people from different cultures and several of them use their native langage and switch between that and English and personally I don't see a problem with it.
I'm going with how the original poster described the situation, which is that hearing people switch languages not because they're more comfortable in another language, but because they want to talk about others in their presence.
I live in New York, and people switch languages all the time. That's life.
In my field, it's assumed that people speak more than one language. Our common language is English, but if, say, two people who speak Spanish happen to end up in the same spot, they might switch to Spanish because they're comfortable with that. And if someone has a problem with that, they really need to get over themselves!
Everyone in the room can tell the difference between sharing common ground and deliberately excluding others. (It's body language -- pointing at other people, or looking pointedly at them, for example. Also, you learn the insulting words. I don't speak Spanish, but I now the insults, and if I hear my name and the rude words, I know what's going on.)
And there's another thing. Some -- by no means all -- but some hearing people do treat deaf/HOH people as lesser than them. How? By deliberately waiting until the deaf/HOH person is not looking to make fun of them. By deliberately leaving deaf people out of the conversation by speaking quickly or muttering. Or by saying something rude, then claiming they said something else.
When I was fully hearing, I saw and heard people do that to my deaf relatives. I have students who do that to me now that I'm HOH.
AND there is the whole awful history of oralist schools, where it was assumed that hearing people were better teachers because they could hear. The OP didn't say, but I wonder if these hearing teachers think that too.
So there is:
a. a double standard involved, as switching into another language specifically to talk about others in their presence is rude among hearing people, and doesn't become less rude in a deaf environment
b. valid reason for deaf/HOH people to be suspicious of some hearing people deliberately excluding them
c. a long and painful history of hearing people being automatically assumed to be better teachers than deaf people.
I think the original poster has good reason to question these people's behavior.