I don't think there's any one I've ever had the easiest time speech reading with. I have to admit though that I'd probably be better at it if I had any real training. I think I started doing it when I lost the hearing in my left ear. I could still hear with my right, but I think I started picking up speech reading then, and not even realized what I was doing.
Usually I rely on both, sound and visually seeing what they're saying. There are definitely people I can understand better than those, but that boils down to sound really, not how well they form the letters on their lips. Some people's voices are just in a better frequency for me to understand I think, and some people's voices carry better than others.
I have the hardest time speech reading/hearing people with really bad teeth, because they either mumble really bad covering their teeth with their lips because they're embarrassed by them.....or they don't care and they let them all hang out and I just distracted thinking about their teeth rather than paying attention to what they're saying.