Oh, LOOKIE!! I'm the first one to respond to this!! 
*ahem* Seriously, Fly...
You raise a good point. That's actually something I never thought about before, but you're right. For the most part, we don't look at people when we speak to them. And, I guess, that could be construed as being rude, couldn't it?
I suppose that a hearing person might take for granted that what they are saying is being understood, so therefore, they don't feel the need to make eye contact with the person to which they are speaking to? 
Interesting to note, my parents are also hearing. However, I believe they are both slooooooowly going deaf or HOH. My Dad is particularly HOH, at times. Because I know this, I always look at them when I speak to them. This is also something I never thought about before. It's just something I've done.
Interesting.
Thank you for making this thread. This definitely gave me something to ponder.
Mucho glacias!
Fly