When talking to deaf people, I make sure I'm facing them, voicing or signing. It's habit--if I know they're deaf, I face them.
For hearing people, it's totally different (unless I'm practicing signing with them). I will look all over the room and rarely make eye contact with them. I often stare at the floor or at something else in the area. This has been something that's true almost all my life because of my autism (the looking around the room/area) and the understanding that deaf and hearing people have different needs.