I know what you mean. I don't wear any make up when I'm outside sweating and doing yard work. Dirt, tree leaves, mulch, all kinds of stuff blowing on my face.
I do recommend protecting your skin outdoors. You might not notice it now but the results will show up years later. Cleansing your face deeply but
gently each night. Using sunscreen and appropriate moisturizer during the day. (Even oily skin needs moisture; moisture is not the same as oil; moisture is water.) Reapply sunscreen during the day if you're outside all day or if you sweat a lot. Use extra sunscreen for your lips, and wear UVA/UVB sunglasses to protect your eyes (it will also reduce future crow's feet that result from squinting).
For all the ladies:
Don't forget to scrub off the rough elbow and heel skin every day, and use lots of body moisturizer on those areas. Sometimes we focus on just the face and forget the other "rough" areas.
Also, treat your neck and upper chest skin like face skin, not like body skin. If you don't believe me, notice some old ladies' necks. The necks wrinkle worse than the faces. Be especially careful about that "V" shape of tan that you get at the neckline. Maybe it looks good now but it will look like a plucked chicken skin later, if you don't take care of it.