Be more specific.
If you're reasonably ABLE to drive 200 miles---have the time, energy, finances---then I'd say yes, but you should then ask the priest how often you should attend.
If you simply DON'T WANT to make the sacrifice, even a few times a year, ALTHOUGH ABLE to do so, then you are in the wrong. Contact the priest for advice.
If this involves SERIOUS INCONVENIENCE and would likely have a DETRIMENTAL EFFECT on your ability to perform your duty of state, then, no, you're not obliged. Again, at least let the priest know you exist and of your situation.
If, by the word "want," you really meant "obliged by Church law," then, no. A 200 mile drive is above and beyond what is required.
I believe Church law specifies an hour by vehicle if you're able to avail yourself, or a walking/biking distance of a mile, if physically able. Both presume no other impediments, ie, physical inability, essential employment, need to attend to the sick, elderly, very young, adverse weather conditions, etc. Make your dispensation official by contacting the priest.
My guess is that a Pharisaical person or persons have been harassing you. If that is the case, tell them calmly, but firmly that Fr. N. has dispensed you, or quote Church law to them, then change the topic or say, "Please do not bring up this subject again," and walk away. Do not try to justify yourself if God and Church have already done so, and NEVER argue, raise your voice, or show anger.
This is one of Satan's devices to drive you away from holiness. Get on with the work saving your soul. By all means, pray for those persons. They are likely guilty themselves and instead of repenting and confessing, they're attempting to lessen their feelings of shame by dumping them on you instead of going to Our Lord.
Hope this helps.