Pray for the Poor Souls and there will be no reason for them to bother you. That's what they're after.
I've heard numerous anecdotes of "noises upstairs" and what not, and after the Rosary was said for them (sometimes repeatedly), the noises went away.
But if it's a demon, you'd need an exorcist. A traditional one, of course. The devil laughs at the modern rite of exorcism.
Either way, keep a good prayer life, keep yourself right with God, and you'll have nothing to fear in this life or the next.