I get the "you talk too fast" complaint from people who have never met a deaf person or have never heard one speak. I also sometimes get that from people I know very well and who are used to my form of speech.
As far as I can remember, I have never gotten that from family members for the simple reason that they're so used to the way I speak. Think of your oral speech as another language and people who are intimately familiar with your deaf speech (like family members) will be able to follow you no matter how fast you speak.
However, the caveat is that your deaf speech MUST be close enough to normal speech so that it isn't impossible for family members to follow you at any speed. Otherwise, it all becomes mumbling/gibberish. Speech therapy does help to some extent.