I don't mind people throwing in some sign if they know it. It often helps confirm that what I am speech reading is correct, or I am way off base!
What does bother me is "OH! You're Deaf! I know some ASL!" and then they proceed to show me every sign they know and struggle their way through the ASL alphabet. That bothers me. If it is someone I know well it doesn't bother me...like the other day my audiologist showed me all the signs she knows (which is 4 and I taught her 2-3 of them) that wasn't the same as some random stranger doing it though.
I am sure other people here know what I mean!