If I were in your shoes, I would put him a block. It will be more peaceful on my mind. If that person has no patient then block him or her. If that person really wants to talk with me very badly. Just simple to email me. I do not need to be annoyance from anyone.
I need to focus on my own time and family. Chatting sometime bores or not important. I did not want to waste of my time. I have 55 people on my list in my IM, they do most respect of my time.