Speculation is unverified or unsupported notions, whereas an accusation is the allegation of a wrongdoing. An accusation may be speculative.
Yes, an accusation may be speculative, but speculations are not necessarily accusations. You're assuming that every speculation is an accusation ... and indeed an accusation rooted in speculation would be unjust. At best it may be labeled a suspicion. But you keep claiming that people speculating about various activities are making accusations, when they are not, or, rather, not all of them are.
Am I ACCUSING Pablo of using witchcraft? No. Am I speculating that he might be? Absolutely. That's the difference.
SPECULATION->SUSPICION->ACCUSATION
So, for instance, we see Joe Biden fondling all kinds of young girls. He's under suspicion of perversion due to speculation that his motives maybe something other than affection and insensitivity to "private" space.