Just think of what adulterous thoughts entail - perversion of the institution of marriage, cheating on your own wife, disrespecting and objectifying another woman, fuelling your base desires, corrupting your own perception of relationships, rebellion towards God, lukewarmness in your faith.
Not to mention the reproach you bring to the name of Christ if you follow through with the thoughts. Sadly, many Christians fail to think about that, or they do not care enough to put aside their own lusts. Any action begins with a thought. That is why we are supposed to guard our thoughts.