Yes but, you should have been kicked out of the fort if it belonged to a friendly faction. Only enemies can occupy towns, ports, or forts. You were stuck in-between.