Just some possibilities here.

While wilderness may be wilderness, another possibility is that the terrain needs to be a minimal amount level to build a fort. While one area may appear level from the campaign map, it may be too steep upon which to build a fort. Cross the border to the next territory, and there, perhaps the ground is level enough. We cannot really see how steep or level it is unless there is a settlement or fort there and we can view it on battle map mode. The only way we can view around a fort on battle map mode, is if it's actually attacked, and we sally forth. Without a fort, we can only see the surrounding terrain if we get into a battle there, but if there is nobody to do battle with, that option is off the table.

Another observation I've made is that in one turn, I cannot build a fort. The next turn, same place, I can.

Fickle isn't it?

Strength and Honor

Celt Centurion