What if you only think you committed a venial sin but it was actually a mortal sin, but you did not confess it because you weren't sure or thought it was "only" a venial sin?

Yes. As I said, how is one supposed to correctly self-diagnose the very nature of a thought, especially without even thinking about it for fear of committing (or repeating) a sin.

What if you only think you committed a venial sin but it was actually a mortal sin, but you did not confess it because you weren't sure or thought it was "only" a venial sin?

If you're scrupulous, your Confessor may tell you not to confess such doubtful sins.  If you sincerely THINK it was venial and didn't confess it, even if it HAPPENED to be mortal, you're still absolved of your sins ... just as if you had simply forgotten to confess a mortal sin.  So long as you don't deliberately and consciously withhold confessing something you know to be a mortal sin.

Your words actually reflect precisely how a scrupulous mind works ... tormented by all the "what if"s.  I think it's venial, but what if I'm wrong, and what if it's really mortal, and what if I don't confess it and then I compound the sin by a bad confession, etc. etc. etc.  You do the best you can.  If you're scrupulous, you can be pretty sure that if you commit a mortal sin there'd be zero doubt about it.  Scrupulosity is a spiritual disease that God wants people to be rid of because it's founded in a lack of belief in the Mercy of God.  God doesn't wish to go around tormenting souls with fear, uncetainty, and doubt.  Do the best you can and if you're not deliberately withholding something you know to be a mortal sin, you can be assured that you're forgiven of all your sins in Confession.

You're right, I was aiming more at the OP. While we do not need to confess venial sins, per the OP who does not seem to know whether the sin is venial, mortal or even a sin at all - he should confess it - hopefully the priest will council him what is a sin and what to confess for next time.

Yes, a good confessor will easily detect scrupulosity and would advise the person accordingly.  If, on the other hand, the confessor detected laxity, the person would be encouraged to confess doubtful sins.  So this advice on this thread must be taken with a grain of salt; only a confessor (a good one) can provide the correct guidance for any specific individual.

It is difficult to find a good confessor today. In my opinion, a lot of this talk of scrupulosity comes from modernist confessors who want to control you, and tell you not to confess, which can only worsen the situation.

Nevertheless, scrupulosity can real as it could be a form of OCD, but OCD could result from a poor diet of junk foods, OTC vitamins and drugs, and/or environmental factors like bad drinking water and agricultural chemicals. Having a diet of Pepsi and Coke can cause OCD as the sugar or artificial sweeteners can cause serious health problems. When we eat a poor diet or take harmful drugs, we are poisoning our body, which is a temple of the Holy Spirit --- and that is a sin.

We should be confessing all thoughts, words, and deed that miss the mark. Missing the mark means that our thoughs, words and deeds are not of God, but come from the world and its concerns, ourselves (our flesh), or the devil. St. Paul tells us to pray unceasingly, so every time our thoughts are not of God, in essence, we are sinning as we are failing to pray unceasingly.

My priest told me that whenever I am having a sinful or distracting thought which is not of God, then I should pray immediately:

Lord Jesus Christ, I reject these thoughts, help me and save me.

This has been of tremendous spiritual help to me. My dreams are now more peaceful too.

When we pray, the devil is most clever in tempting us to worry about our parent's health, about our finances, about our country and the direction it is taking, or worse, to think about some indecent ad that we heard over the radio, TV, or Internet. We must reject these thoughts immediately, focus on Christ, and return to our prayers. Looking at a Holy Icon of our Lord or of His Most Pure Mother can help us tremendously to refocus on Christ.

A simple statement, such as, "Lord Jesus Christ, forgive me a sinner for I have sinned in thought, word, and deed" could be a good way to begin your confession after the customary beginning prayers. Then any sinful thoughts (that are not mortal) would have been confessed. My priest encourages me to begin my confession with those words.

By the way, Father Robert Bishop of the Claretians is an excellent priest and confessor. He says the Traditional Latin Mass and is biritual (Ruthenian).