Some might have wanted to move. Not be forced and shipped off to Africa, which, if memory serves, wasn't exactly the best place to live at that time.
That might not have been as bad as a slave holder, but it is certaintly racist, and he is IMO undeserving of the great rap he gets about being the slave free-er.

Alex, I don't know what he could have done better. But it seems to me he didn't care much at all about slaves, rather just outlaw slavery because it's bad, not so much for the slave's sake. And if he wanted to free them, he wouldn't have left them in the boder states.