Yeah, I´ve been pretty pissed by the hundreds of times there´s been a little castle just next to my starting position, the gate is open, and I´d really need some defenses against the heavily outnumbering enemy. Yet there is no way for me to enter the castle. I can walk up to the open gate, but through it? No way.
Stupid, why put them there in the first place if you can´t use them.

Little farms with mills and wells can actually be pretty useful for dividing your enemy´s forces into two when they´re crossing the area. But still annoying that you can´t garrison them or anything.
At least it appears this sort of actions will be possible in ETW. Better late then never.