I would actually say don't build farms because they increase population growth and you'll end up with 30k+ population in settlements and have massive PO problems or you'll have to employ full stacks of militia to keep PO in check. Build at most landclearance, nothing more in towns.
In castles you can build full farming as the order there is usually no problem. Of course the income isn't as great, but still, in the long run it pays out.