You are looking at it from a Northern perspective. The majority of Southerners were not slave owners, but when the slaves were freed the Southern economy collapsed and their living standards suffered.
You are also wrong about the South starting the war, it was not illegal for the Southern States to secede - it was simply not provided for in the Constitution. The North refused to let them go, which was a political decision, not an ideaological one based on whether or not they should own slaves.
Bookmarks