It could also be Satan trying to get you to stay away from the Church. A similar thing happened to me as a recent covert, only the repercussions were more immediate, like, in the middle of Mass, I'd have panic attacks and need to leave. Once, I had such a horrible dream that I awoke barely able to breathe, terrified that if I merely got out of bed, I'd trip, fall, hit my head and die because I'd forgotten to confess a mortal sin from my distant past in confession the day before. I called the rectory at 4:45 in the morning and to my amazement, I got the priest in person, no recording. He heard me out and simply said, "It's a temptation." Instantly, the fear and panic left. He gave me a blessing and it was over.
You should definitely talk to a priest, however. Is it possible your baptism was invalid? Mine was fine because it was in 1956.