Right. If I talk on the phone with people, it's with people I whose voices I am familiar with, like parents and some friends. But it is hard to hear them on the phone, too. The longer a telephone conversation goes on, the harder it is for me to understand them, because it goes outside the range of predictability.
How predictable the conversation is, helps me to understand them on the phone. For example, if my mother calls, it is probably for a couple things... to invite me to dinner on such-and-such day, that the weather is bad (tornado coming, etc), or someone died. The range of words within those topics are fairly predictable, so I can guess at a lot of the words she's saying. Once I have context, it's easier to figure the rest out.
But let's say she starts talking about her neighbor and the choice of paint they picked out for their basement office that their husband is going to use for his photography business. Whoa. Way outside range of predictability now and I have no clue what she's saying on the phone. I might happen to hear "paint" and "photography." But without context, these words are meaningless.
But my family and friends know by now that texting me or email is the best way. That's usually how they contact me. I haven't actually talked to my parents or friends on the phone for a long time now. I prefer not to talk on the phone if I can avoid it.