I've been to ireland already. Love it. Want to move there one of these days, but never quite get around to it.
What's really cool is that if your family is irish, you can get irish 'dual' citizenship relatively easy. (ie, if your parents, grandparents or in some cases great-grandparents were irish citizens, you get irish citizenship quickly, and keep your US citizenship.)
Link to embassy info
My father has been halfway thinking of getting that, and retiring there.
The Castles in ireland really aren't much. Most of them are ruins. (which is why you might think they are haunted). For castles, going elsewhere in Europe might be best.
But DO contact any relatives you might have there, its very enlightening. The irish are friendly, and used to having immigrants coming back. (My great-aunt had relatives from australia, US, NZ, and South africa all come to visit in ireland)
Just DO it. Ignore the cost, its really very worth it.