I lipread and I speak too.... my entire family is hearing... and I was raised oral.
I, too, do experience a lot of frustration when interacting with some people.. facial hair, accents, mumbling (some of my friends do that so I smack their heads every now and then... j/k), the occasional distraction (broccoli in teeth)... j/k! And sometimes just plain, old fatigue...
Anyway, I will probably always have to rely on lipreading and speech -- ASL is soo much easier... and I can follow big group conversations w/ that!
Oh.. also.. I tend to "interpret" for my Deaf and hearing friends -- I speak for the Deaf, and I sign for the hearing... a lot of Deaf friends, at first, thought I was just HoH... nope!