Had just that thing happen more than once. I answered, "All students--not just the few who disrupt. Many here tell me and other faculty they appreciate classes starting on time and limiting disturbances."
Every time I got applause from serious students who wanted their money's worth.
Way to go. That's the way I like it too. It's highly rude to talk back to an educator in that manner. Just like it is rude to tell a police officer that you put the food on his/her table by paying taxes.
A professor's job is to educate the students and the rules of his/her classroom should be followed, no matter what. If they don't like it, they can drop out for all I care.