Then what's the purpose of a granary anyway?

It increases pupulation growth and reduces trade, two thing you wouldn't usually want anyway, and on top of that it doesn't even help you out when being besieged.