PS: Stupid question, but do you totally own the province or is the castle still in enemy hands? If so, then you can't build anything in that province...

May not be so stupid, it seems as the only explanation available, unless he's not reconstructing the infrastructure as it should.