How do you know that there's no holiness at all in the visible and material structure of the church? The negative elements that you describe above do exist, but can you be quite sure that nothing of the Catholic Faith remains?
Communicatio in Sacris/Divinis with heretics and/or schismatics is heresy. Those in the material structures of the Church are heretics for either a) being heretics themselves or b) for being in Communion with heretics.
Pope Leo XIII, Satis Cognitum (#9): “No one who merely disbelieves in all can for that reason regard himself as a Catholic or call himself one. For there may be or arise some other heresies, which are not set out in this work of ours, and, if any one holds to one single heresy he is not a Catholic.”
III Council of Constantinople, 680-681: “If any ecclesiastic or layman shall go into…the meetinghouses of the heretics to join in prayer with them, let them be deposed and deprived of communion. If any bishop or priest or deacon shall join in prayer with heretics, let him be suspended from communion.”