I agree that it's probably a fluke, especially since the "loss" keeps switching ears. It's more likely that the ear is a little stopped up. Up until children are 18-24 months old, their eustachain (sp?) tube -- that tube between the ear and nasal passages -- is straight across; it doesn't slope towards the nose as it does in adults. That's one reason babies get so many ear infections.
I think it would make sense to begin learning ASL and teaching it to Adam. That way you're prepared if he does have a hearing loss, or if you're hearing should worsen. Best case scenario: his hearing is fine, and you have enriched both your lives by learning a new language.
I too relied exclusively on lipreading for many years, and it's still my main method of communication with my family and old friends. But because my hearing loss is progressive, I started learning ASL several years ago, and now have made new friends within the Deaf community. We've been teaching my grandson (whose hearing is fine) ASL signs; he knows the signs for milk, more, finish, and bed. It makes dealing with him soooo much easier. Now, instead of throwing his food on the floor when he is finished eating, he signs "finish" and we don't have to go thru all the disciplining and cleaning up his mess.