You should be fine, just wait and see if they attack. If they don't and your about to fall, just rush out with your generals and keep your crossbowmen on the walls. Then bait them toward the gates of your castle.