William Ellison was one of the richest men in the South, besides being a black ex-slave. I had cotton ginners, plantations and 68 slaves. At the height of slavery in the United States, a large number of free blacks owned black slaves; in fact, in numbers disproportionate to their representation in society in general. In 1860 only a small minority of whites owned slaves. According to the US Census report of the previous year of the Civil War, there were almost 27 million whites in the country.