Living on a small plot will not yield a satisfactory livelihood.
That being said, it's crazy to settle with the idea of raising food on marginal land.
If you're going to pick a small plot to farm, find a good location with good soil and ample water available, preferably by a stream.
There are plenty of people out there who are doing just that on less than 3-4 acres, and I've met a few of them. And they are making about the same as a conventional farmer with 100 subsidized acres of corn and a job on the side. The key is a diversity of crops/products, stacking and overlapping the functions of the elements in your system, and using nature mimicking techniques to fertilize and build the soil that at the same time minimize your monetary inputs.
I'm not saying that just about anyone could do it, but they can if they know what they are doing. The possibilities of what you can do with each acre is limited only by your knowledge base.
5 acres is enough to support a Trad size family and some relatives.
You make a good point about water. I thought that was a no brainer actually, but you can settle in a dry area so long as you have a good aquifer to tap into.