That's how it usually is for those who are deaf. They cannot always pick up differences in sounds. So, they grow up accustomed to the English language (or the language they're used to).
I had a couple deaf friends who were Hispanic. They grew up with Hispanic families. They speak and understand Spanish very well, but they had a hard time understanding English unless it was in ASL. The reason for that was because they grew up getting used to accents and the way the Spanish language is spoken.
This also applies to a few deaf Asians I knew as well.
Oh yeah, the most common voice that pretty much everyone is accustomed to is... their mom's voice (or their dad's). I have a friend who can only understand her mom's voice on the phone and no one else.
