problem can be had as the almos, where you have to build a citadel in Grenada. You have to build it after you get the goal to get the points.
You'll also lose the points if you upgrade to a fortress before the time for this goal has run out.