I know totally what you mean. Since I lost my hearing, it seems that the people who knew me while I wore HA's don't have the time for me since I can't communicate with them. I am still learning ASL, but they don't know it and won't take the time to try to communicate with me.
I think the pen and paper idea is by far the best solution until you can learn a little ASL. It won't seem rude or presumptuous on your part expand on your customer service skills.
