I disagree. A separate, shared language can be a part of a culture and a cultural identity, but it's not a requirement for all cultures. Most people from the U.S., England, Canada, and Australia speak English, but they all have their own distinct cultures (and there are also various regional cultures within each of those countries). Religions can have their own culture that crosses languages. For example, Catholics from all over the world share a common cultural identity as far as their religion, even if they don't all speak the same language. In fact, the original language of Catholicism was Latin, and hardly anyone speaks that anymore. There are many cultures where no one speaks their original language because everyone who spoke it died; that doesn't mean their culture also died out. There are also many countries where there's more than one official language (such as Canada: English/French), but people who speak those different languages would still think of themselves as sharing a common national culture.
Cultural identities and languages overlap and cross into one another. Sharing a language isn't necessarily a requirement for sharing a culture. If members of Deaf culture want to exclude non-signing deaf/HOH people as being part of their culture, then that's their decision, but that ignores all the other things besides language that also make up culture, such as common experiences and beliefs (which Deaf, deaf, and HOH people sometimes do share). Some Deaf people may say, "But we often don't believe the same things or have the same experiences!" Well, not all people who sign share the same beliefs and experiences, either.