When I first startied playing, I saw a Mosque on the battlefield and use my arrow to destroy it. Can't remember if you them lost it of the province building list.

A Castle drops to a level, the port usually goes and some other buildings. What determines when a castle drops a level?