Should you? I can't tell you what you should feel. Do you feel personally attacked when you see a Deaf person without CI? If you felt personally attacked by a Deaf person without CI, that Deaf person would probably feel judged. I'm not talking about what people should and shouldn't feel, just trying to offer a new perspective into why people do feel the way they do. Judgment arises from strong feelings, and it helps to see why others judge when you look at the strong feelings behind those judgments.
I can't speak towards justification. Who's to say what's justified and what's not? I can totally see how it comes to you out of the blue.
It's worth noting that people are offended by just the sight or sound of something all the time because of what it represents for them in their minds. That's why swear words get bleeped out in TV shows and certain ways of dressing get banned in certain countries, because even if the person saying the swear word or wearing the outfit isn't saying anything about what others should do, the fact they are saying or wearing it is enough to offend other people. They don't have to start talking about what others should do, the fact they live the way they do is enough to offend other people.