The Reputation score is mostly worthless, but is only useful in a broad-brush, vague context. It is a measure of How Well a User Fits In On CathInfo, it could be their Trad Orthodoxy Score, Good Poster Score, or some combination of those things.
A few downvotes aren't going to change the big picture. Just the fact that you got 7000 upvotes says a lot about your glorious past posting history. Even if someone managed to downvote you to 7,000 downvotes, I'm not going to look into banning you or something. I'm not stupid.
If I connected the downvote system to automatic warnings/temporary bans/posting privileges or other computer/software-based moderation then I would FULLY agree with your objections.
But they are just a guideline for the Moderator when he looks into a given user. When someone has 10 upvotes and 120 downvotes, I take notice and look at his posting history. And guess what? Those numbers almost NEVER LIE. Usually when someone is thus lopsided the person is very rough around the edges at best, heretical at worst, etc.
And I want to remind everyone again: the more participation in the voting system, the better and more accurate everyone's score is. Because guess what? The troublemakers and loudmouths certainly don't hold back on voting. If you leave all the voting to them, the forum's posts and Rep scores will reflect only THEIR tastes! Don't let that happen.
And I should also remind everyone (again) that part of the reason the system "works" is because I restrict how much "inflating" or "damage" you can give someone's score. Mary is only allowed to be 16% of John's downvotes. Once she hits that limit, she will get an error message until enough people downvote John so she can downvote him (1) more time. And so on.
If John has 100 downvotes, Mary can only be 16 of them. But if John gets up to 200 downvotes, then Mary's total 16 downvotes is only 8% instead of 16% of John's total, so she is allowed to downvote him again.
When a user is a real jerk, or a total misfit here, the downvote limits don't really limit anybody since enough OTHER people are also downvoting him.
But it makes it harder to gang up on someone and damage his score.