Based on my reading of the facts, I do believe that Honorius was objectively heretical in some of his statements. Where there's doubt is regarding his pertinacity. Certainly the matter had not been explicitly defined by the Church at the time he wrote those things to Sergius. And the letter to Sergius does not meet the notes of infallibility, since it was not a teaching to the Universal Church deciding/imposing a matter of faith (it's very similar to some of those Pope Innocent letters cited incorrectly by St. Alphonsus as making the BoD de fide).
Some Church Fathers promoted opinions that were later explicitly condemned as heretical, but they're still listed among the catalog of saints.
Now, the Church's judgment that he was a heretic is in fact very strong, but again it doesn't strictly meet the notes of infallibility. And it's a bit problematic to "remove" from the Church someone who's already dead. If Honorius had been in heaven, a member of the Church Triumphant, would he be pulled out of there and thrown into hell by virtue of this declaration?
I don't believe that Honorius was guilty of pertinacious manifest heresy that would cause him to have lost the office, but I do believe he was materially heretical on the point, at least at on time during his reign.