Nah, you won't lose a point for not voting if you lost a duel.