I believe your experience parallels mine and my son's. I learned sign from hanging out at the local deaf club, he learned much of his there, too. From the two of us, my parents and my brother, and my brother's boys learned. The kids in the neighborhood learned simply from having play time contact with my son.
Today, even my nephews' children sign. So it has spanned 4 generations, because I was blessed to have the type of family that puts all of the children's needs first and is willing to do whatever is necessary to meet those needs. They go so far as to not just sign to my son, but to sign to each other in his presence.