Hah, same thing for me! Except, people usually think I'm from Spain or Austria. Since I'm originally from California but now living in Midwest, when I tell people I'm from California, they just figure my "accent" is Californian. In California, if people ask about my speech, they just ask if I have a lisp or something. I like being in a different part of the country, because I'd rather think of it as an "accent" than a lisp anyway.
I have a bit of an "accent" but most people think/assume that I'm just not from the Southern US. I was born and raised in the south but I have absolutely no southern accent. I thank speech therapy for that.
I work at a major research hospital where there are workers from all over the world. When people meet me they usually spend a few minutes trying to guess where I'm from. Their guesses are (in order of frequency)- Ireland, Canada, and "not from here".
The funny thing is that my grandparents were Irish and most people assume that I'm from Ireland because of the way I talk.