Libertarianism is (1) false and (2) impracticable.
(1) First, it is based on the false premises of political individualism, economic reductionism, and social contract theory. Catholics believe that the family is the basic social group, that man’s end is spiritual, that the state was created by God, and that it is obliged to profess Catholicism. Right off the bat we can see the incompatibility.
(2) In practise, libertarianism would create a vacuum of power, which is of course unnatural and could not be maintained long except through external intervention (which would be an internal contradiction). My brother, who is basically an anarcho-capitalist, inadvertantly expressed this contradiction when he once said to me that in a libertarian society all children would have to be thoroughly educated in libertarianism in order to resist the tendency for power structures to 'ossify'. Human nature being what it is, I think that within a generation such a society would regroup into a primitive feudalism.
It might be objected that there are many different kinds of libertarianism, some of which might be compatible with Catholic teaching, etc. I’ve read Hans-Hermann Hoppe before, and his vision seemed somewhat conservative and appealing. Ultimately I think the social doctrine of the Church is best expressed in Christian Corporatism, as expressed in writings like Quadragesimo Anno, and that in terms of secular thought the perspectives of men like Karl Polanyi (The Great Transformation) or even Alisdair MacIntyre are more valuable than the Libertarians.