It's a challenge, yes, but it can be done. Hearing children can learn how to sign. No one has to be "voice off" all the time but when the deaf child is present he or she should be included in the conversations. How you go about it would probably vary, depending on the circumstances. If it's a short utterance, such as, "dinner is ready" that could be done simultaneously in ASL and English. If it's more complicated, a consecutive response might be better. It also depends on the ages involved, degree of formality of the conversation ("Baby Snookums, come meet Daddy's boss"), urgency ("Don't stick your finger in the outlet!"), etc. Also, within the family context, members can establish cues and shortcuts for casual conversations that will make things easier.
In families where both parents are Deaf but some or all of the children are hearing, they work it out, so where both parents are hearing and some or all of their children are deaf, the family can work it out.
One would hope that parents could model fluency in the language they use but regardless, casual home chat and formal school language instruction aren't usually the same thing for any kids, hearing or deaf.