Will they fall into eternal fire due to their pastor?
I was taught that it is no sin on our soul to follow the instruction of our spiritual advisor, that any sin he encourages us to do falls squarely on his shoulders.
If we know it to be a sin and still do it then we are also culpable, but he is no less culpable.
In my understanding, that's mostly correct. As long as what we follow from our Spiritual Advisor is not a sin in and of itself then we must follow but we can never be obedient in sin even if we are ignorant of it, although they will be more culpable than you you would still be culpable. Ignorance of sin is not an excuse and we will still be judged by it but less intensely than if we know for sure something is a sin.
If he tells us to touch his "member" to be obedient then we cannot, even if we somehow don't think it's a sin. This is why proper catechesis and instruction is so important and that responsibility falls on the parents at first and then the proper religious instructors next. Remember also that God writes the Natural Law on our hearts.
Much like ignorance of the law is not an excuse same mostly applies here but one is less culpable than if they had full knowledge.