Quite, actually. Many a soldier fought for their state simply out of loyalty. The most famous example would be Mr. Lee, who Centurion has already pointed out. Lincoln himself was rather racist, although quite a moderate given his contemparies, and his goal was to strengthen the North, not to free slaves. Not to say that race wasn't an issue, but most Northerners fought because the South just seceded from the USA. The South obviously found the loss in plantation revenue alarming, but the poor white workers with no land didn't have much to do with that.
Bookmarks