It has been a problem.

There are some old threads you can look into on the same subject.

You must wait at least one turn after breaking the agreement before you can attack without penalty but the game has a way of sticking you with it in some odd ways.