My take on it is: if I miss Confession I miss Holy Communion. I do this whether it's venial sins I'm confessing or (God forbid) mortal sins I'm confessing. Better to be safe than sorry.
Remember, the priest gives you absolution twice during Mass so your venial sins are forgiven. Why would you pass up the wonderful graces you would receive in Holy Communion?
It's one of my very few scruples I have. I've always linked the two Sacraments together. It's definitely something I have to pray about.
Please do not let the devil stop you from receiving Holy Communion because of a venial sin. Please do reflect on your sins, and ask yourself whether you committed a mortal sin, or a venial one. There are specific things that are required for a mortal sin.
Mortal sin is a sin of grave matter
Mortal sin is committed with full knowledge of the sinner
Mortal sin is committed with deliberate consent of the sinner
If you are missing one of these elements, it's not a mortal sin. Please remember this, and do not become overscrupulous. You cannot accidentally commit a mortal sin.