Hearing children are very exposed to hearing world. It is very rare for them to live in a sheltered life. I don't think deaf people can keep them from hearing when they go out shopping, watch tv, visit friends or family or church, etc. Children can pick up language easily that way. But it is the opposite for deaf children. they can not pick up language easily unless someone show them.
My grandma's sister have two CODA (one of them was married to a hearing son of a deaf teacher) They grew up in ASL and I haven't noticed any problems. She is very adaptable to the hearing world without early intervention. speaking of that, I should ask my grandma for her email.