I used to be around people like that at the deaf school where I attended. I didn't reject them for that reason, I did my best to be very patient with them. I am in contact with one of them, older than I on regular basis through E-mail and texting - she knows not to be afraid to ask me to make it easier for her.
I would rather be around people, deaf or not like that than to have to deal with non-English speaking people on daily basis. This is something I have to deal with now.