I seriously doubt he was attempting to bully you. He has too much of a heart, even he couldn't bully a fly.
What he was trying to point out is that a lot of people put the deaf in an incorrect frame. We as a collective group should be changing that by questioning the others on what they mean when they say these things.
I have totally no doubt if you wanted - you could meet the Boulder boys and kick it back, talk about reframing.
We won't bite.
Using your logic, we could deduct that with the social norms that a deaf can't bully a hearing person because the hearing counterpart automatically has more power than the deaf, therefore the deaf can't really be a bully.
You can read this with the thought in the back of your head; it requires power to be a bully. If you have no power, you cannot be a bully.
Hearing over deaf.
Men over women.
Light skin over dark.
Just to start.