How do you let them pass you? Do you make it obvious that you're letting them pass you or do you look for something to give you a reason to stop and let them pass you?
I do that sometimes. If I'm walking and I think there's someone walking behind me, I'll look around for something to stop and look at. For instance, I'll be at a mall. While walking, I think there's someone who I thought looked at me weirdly like he/she knew me or something. So, I walk along like nothing's wrong... then see a game store. So, I stop and go into the game store to check out their demo. (Their demos are facing outside through the window. That way, I can watch the person walk past me.