Well I agree with that people say about the grayed out/flashing thing.

Look at screen, queue up a farm, grayed out farm income higher than old one appers, unqueue farm, it goes away.

"I've constructed Watchtowers and scoured the place with spies, assassins, diplomats and even army pieces. I can easily see if there are bandits,enemy spies, diplomats, and army nearby"

The bandits can be anywhere on the road between your town and the one you are trading with. I'd have trade gutted by a bandit army that appeared on the doorstep of my allies town and they were too inept to drive them off so I had to march way over there and take them out.