I honestly don't think it was deafness that bothered them. If it was deafness, then the family would have disowned Felix long before this unfortunate Pawn shop incident.
Just my 2 cent.
I am sure it bothered the parents according to this quote ."He told me his parents were ashamed of having a child that could not hear so they did not want anyone else to know." Also why did Felix lived with his grandmother instead of his parents?