If you highlight a province, not just a stack, and press the "m" key it will merge the various units in the various stacks into one or more stacks, and it will sort the units when it does this. It's a little funky, though - if you have more men than will fit in one stack then it makes two (or more), and tries to distribute the unit types evenly between them. You can manipulate what gets combined by moving units/stacks you don't want to merge into the castle or an adjacent province before you hit the "m" key.

(I don't know if it makes a difference, but I always play with "auto-tidy" turned off.)