You dont have to destroy enemy temples. As far as i know you can build the next level of your temples over the ones that belong to another faction.