Oh lol, if a castle has negative happines at end turn it will receive a -25 wealth...
Bankruptcy will give minus wealth too, which will grow if not fixed...
Other than that only random events can give malus afaik...