Nasty rumors were stock in trade for Roman elections. No libel or slander laws of substance back then. Attack ads are NOT a new invention.
Didn't this impact also the fact that he was dating Cleopatra? Her being so famous made the rumours even more spicier for the common Roman folk.