I found this quote attributed to St. Athanasius in a letter to his flock during the Arian Crisis.

"Thus, the more violently they try to occupy the places of worship, the more they separate themselves from the Church. They claim that they represent the Church; but in reality, they are the ones who are expelling themselves from it and going astray. Even if Catholics faithful to Tradition are reduced to a handful, they are the ones who are the true Church of Jesus Christ."

As true today as it was then.

I got this from here.
http://www.olrl.org/snt_docs/athnasus.shtml