He was speaking to the Apostles, who had the means to partake in his Body and Blood. He said "YOU".
Now compare this with St. John 3:5 "Unless a man," he was referring to mankind in general.
The Church has used this verse from Scripture in an infallible decree, at the Council of Florence, using words that, when taken literally (and why should we not? the words are infallible), insist on the absolute necessity that each person who will be saved is baptized in water:
"Holy baptism holds the first place among all the sacraments, for it is the gate of the spiritual life; through it we become members of Christ and of the body of the church. Since death came into the world through one person, unless we are born again of water and the spirit, we cannot, as Truth says, enter the kingdom of heaven."
Furthermore, the Holy Catholic Church has never used the verse you quote in any way to describe the absolute necessity that every person receive the Eucharist for salvation.
The only time that this quote has been used in an infallible Council was at the Council of Ephesus, defending the natures of Christ:
"For being life by nature as God, when he became one with his own flesh, he made it also to be life-giving, as also he said to us: "Amen I say to you, unless you eat the flesh of the Son of man and drink his blood" . For we must not think that it is the flesh of a man like us (for how can the flesh of man be life-giving by its own nature?), but as being made the true flesh [vere proprium eius factam] of the one who for our sake became the son of man and was called so.
"For we do not divide up the words of our Saviour in the gospels among two hypostases or persons. For the one and only Christ is not dual..."
The Eucharist is held as a necessity of precept (the omission of which is a sin when it is able to be lawfully had at the hands of a Catholic clergymen, according to the precept of the Church), whereas the Sacrament of Baptism is held as a necessity of means (the omission of which renders the effects impossible to attain, whether such omission is culpable or inculpable).