The fact is, people without religion have bought into a bona-fide CULT complete with dogmas, high priests, inquisitions, shaming of "infidels", you name it.
It's not shameful to worship God, to give due honor and praise to Him. That is the purpose of religion. God made everything, and we can't give Him anything He didn't first make and give to us -- including our very being.
But when the same practice of religion is rendered to a false god, a man, a lifeless idol -- it becomes debasing, ludicrous and shameful.