You might have been brought up that way, to me it's not the case. I have no problem using paper and pen as a means of communication.
LuciaDisturbed did mention that the people that react like that are hearing people that try to learn sign. That's why they don't want to have things written down. (Which is silly, since it will help them learn...)
