You don't need the Fathers for that. Someone must have the final say when Patriarchs have a dispute. The Pope has always been the one everyone turned to when there were disputes.
All the Eastern Orthodox have is quotes that indicate that Rome only intervened when asked, not that he didn't if asked. Every list of Apostles has Peter first and even the EO admit it.
Common sense