Quite agreed. At the end of the day, that's all BOD is, speculation.
Indeed, St. Augustine admitted as such when he floated the speculation. While he's always cited as THE authority behind the "teaching," if you read what he actually wrote without the ellipses generally inserted by the BoD-promoters, you'll find that he said that he floated the opinion after he had gone back and forth on the question, so, something along the lines of, "Having gone back and forth on the matter in my mind, I find that ..." There's no assertion that this was received teaching or that he was teaching it with authority, but that he was clearly speculating and opining based on HIS personal thinking ("I find ..."). Lesser known is the fact that he later retracted this youthful speculation, issuing some of the strongest ANTI-BoD statements in existence, after he had cut his teeth battling the Pelagians. St. Augustine barely ever had a thought he didn't write down, to the point that as the years went by, he felt compelled to issue an entire book of "Corrections". There's St. Ambrose but I believe that the oration for Valentinian is seriously misinterpreted, probably on purpose. He said that he HOPED that Valentinian could receive some grace from his confession and zeal, similar to the martyrs, but then added that even the martyrs who die without Baptism are not crowned even if they are washed. That sentence generally has escaped everyone's notice. That's your sum total of Patristic authority for BoD. Meanwhile you have 6 or 7 Church Fathers who explicitly rejected it.
There's no need for it either. If I promote disbelief in it, that makes it all the MORE likely that someone would ardently desire the Sacrament and possibly be saved by it (if that is in fact part of God's salvific providence). Yet if I promote BoD, it actually could make individuals more complacent that they could be saved without it and therefore desire it less ardently. There's no benefit. If I'm wrong, God will save this soul regardless of my opinion. If I'm right, then some souls might be lost due to an indifference to receiving the Sacrament.
St. Thomas wrote that explicit faith in the Holy Trinity and the Incarnation are necessary for salvation, and said that if there's some poor ignorant native who had not placed obstacle (from other sin) in the way, God would send an angel if necessary (or else interior enlightenment) to convert the soul ... and then that same angel could easily baptize the individual or God could bilocate a Catholic, such as He had done with Mary of Agreda to the Native Americans.
See, the BoDers claimed that we "limit" or "constrain" God by the Sacraments. Hogwash. We're trying to lean what constraints God has imposed on us. Quite to the contrary, the BoDers limit God by "impossibility", a heretical notion since the Holy Ghost teaches us that with God all things are possible. So St. Augustine later declared that you can't accept that God could be prevented from bringing the Sacrament to His Elect ... if "you wish to be Catholic". God arranges all things according to His Providence for the good of His elect. Period. And no kind of "impossibility" could ever prevent bringing Baptism to His elect being simple childsplay for God.