I think you handled it well by making other plans. Your actions said more than you could communicate it, "You have a life and although you are interested in checking US out, your life doesn't stop for him, at least not at this time".
If it were me I would be as casual as possible and see how the ASL classes go and how that helps the relationship for better communication. I'd take it slow, treat him as if he were one of my friends.
How would you respond to one of your friends if this had happened with your girl friend? Maybe you'd be a little annoyed, but wouldn't have read into it too much.
I guess if it becomes a habit then you might confront things.
I'd say he's interested just like you, so give it some time and keep the pressure off for yourself and for him while you work out these little knots.
Good luck cause you just don't know, you may have the ingredients for a really great long-term relationship.
How do you Know if you don't take the time and check it out?