Smith is describing how the average person is more inclined to be fair, of greater moral standing, and less inclined to pursue elite social status. If he thinks the "every man" is more suited for governing, who does he think is less suited for governing?
