Withdraw or rout from battle may be considered as bad and could be penalized with bad vice. However, retreat to castle should not be penalized because that is the main reason for building the castle.