Why do you think it "worldly" to go outside, or walk in the woods, or take innocent pleasure in a funny movie, or enjoy, in a wholesome manner, the company of a beautiful woman?
Are these things the same as time spent in mental prayer, the use of the discipline, etc? No, but the bow and string cannot always be stretched. Eventually they will break.