Looking at the foot of the word..."will"ful.  It is in the will.  For instance you pray but you would rather be thinking about what you're going to eat later and how good it will taste, etc. while you're uttering words to prayers (or trying to meditate with mental prayer.)   It would be willful distraction if you just think about whatever while praying wilfully.  If you don't want or desire the distraction then it's not willful.  I know at the beginning when I started praying I used to purposely think about other things while praying and those were willful distractions.  If you're praying without the intent to be distracted and your mind just wanders (and it will) then it's not a willful distraction.  

Reminds me of a joke --

During spiritual direction, a Jesuit asked his confessor if it was OK to smoke while praying. The confessor told him "no".

The Jesuit came back the next day and asked if it was OK to pray while he smoked.