Yes, I don't want the Reputation score to reflect "What Ethelred thinks of this person" -- I'd rather have it reflect the forum membership at large.
The more people using the voting system, the better.
Otherwise, guys like Ethelred will decide who has a good or bad reputation. That wouldn't be good, would it? Even if you like Ethelred. It would be best for the reputation to be a total of EVERYONE's votes for and against the person's posts.
So, the moral of the story is,
Everyone, get out and vote!