I used to agree with the "Sons of Shem" theory, but having read the Patristic sources posted by Pax (thank you), I have changed my mind.
I believe that there were a lower form of angel that had physical / corporeal nature. This is fascinating really.
I think that we have looked back upon these texts with a post-scholastic bias. By the scholastic era, Angels were defined as "pure spirit". Consequently, it made no sense how they could possibly mate with women or give in to "lust".
Pope St. Clement:
changed themselves into the nature of men, in order that, living holily, and showing the possibility of so living, they might subject the ungrateful to punishment, yet having become in all respects men, they also partook of human lust,
Pope St. Clement was very early and likely had direct contact with disciples of the Apostles.
He writes here that they "changed themselves into the nature of men".
Lactantius added some more detail. He said that after men began to become corrupt, God sent these "angels" down to earth in order to help keep men from being thoroughly corrupted. Of course, it backfired an they themselves became corrupted. This dovetails with the term used by the Book of Enoch, "Watchers".
Athenagoras:
Some, free agents [angels], you will observe, such as they were created by God, continued in those things for which God had made and over which He had ordained them; but some outraged both the constitution of their nature and the government entrusted to them: namely, this ruler of matter and its various forms, and others of those who were placed about this first firmament
I'd love to get a hold of the Greek and see what is translated as "free agents". So he says that some of them continued on doing the things God had made them for. But there were a group (a lower choir) that were put in charge of "matter and its various forms" and hanged around the "first firmament" ... which is the physical firmament.
So, the Ancient Aliens people do a good job of digging up all kinds of cultural legends of these beings that came from the sky and brought them knowledge (of the gods). These guys of course call them aliens (from other planets).
But what if God had created this lowest choir of angels that were closest to physical nature, to govern it and and interact with it, and they had a partly-corporeal nature precisely so they could interact with nature. So perhaps not ALL of the angels were pure spirit, but there was a lower form of angel that had a partially-material nature, perhaps along the lines of what our risen bodies will eventually be like. Evidently their physical nature was compatible with human nature, but a step above it, where they were capable of fertile relations with human women, and then eventually succuмbed to lust.
Based on these sources, I do not think that we're talking here about either "pure spirit" angels or the "sons of Seth" vs., say, perhaps, the descendants of Cain.
Fathers say these were in fact angels. Angels by definition, and by etymology, are creatures that are in between God and man, above them in a hierarchy. Whereas the higher choirs of these angels are pure spirit what if there was a lower choir of angels that was in fact partly material, who were not fashioned from the clay of the earth, but rather from a more pure matter. And there was some kind of genetic component to them that caused their offspring to be gigantic.
Perhaps all those legends of "gods" that came down from the sky (bringing knowledge with them) and the begat a generation of "heroes" and "giants" were not mere legends but a reference to these very creatures. Perhaps the Ancient Aliens people are right ... except that these creatures do not originate from other "planets" but from the "first firmament" directly above earth.