Well, you can't build forts...

-next to other faction's armies or cities
-right next to another fort (your own faction's or another's)
-Not sure on this, but, I don't think you can even build them next to other faction's diplomats or assassins.

That's about it.