Not sure anymore, so correct me if I am wrong ....

You can however not use the walls, meaning you cant put your troops on the walls and have the AI siege the fort. (or vice versa)