I try to get a pact with one of my neighbors so I can finish off the rest of my neighbors and then smash them.

But as far as I remember the computer doesn't really honor alliances so you still need a garrison and you'll never really have any sort of long-lasting alliance. Feel no regret, they will have a better life under your benevolent rule.