The founding Fathers were more diverse than that. You can find it in the language that many wished to suppress or remove slavery. The constitution gave slavery protection for only twenty years at which point congress banned the import of new slaves as soon as the protection expired. These compromises and the actions of early congress show more nuance than you give them.