Is there a downside to eating Almonds?

[Ice]

Of course it makes a difference. But the difference is only relevant if your diet is so messed up to begin with that it actually matters what you eat.
Your body amortises its dietary needs: that you need 2500 calories or something today doesn't mean you'll die when you take only 1500. That you need 2500 doesn't mean your body will suddenly inflate to triple its normal size when you have a hefty Christmas dinner .... All those “health” things with food are like that assuming your body is in halfway working order.

As a result healthy people get a varied diet without even trying, just like people simply stop eating when they've had enough. Hence, to simply eat what you like is actually the sane thing to do, assuming what you like amounts to a reasonably healthy over the long term. Picky eaters is another issue entirely: that's a psychological/brain issue.

That's a pretty big if. In the situation you described, I'd agree. However, the vast majority of people do not do this. Therefore checking nutritional information is often a good idea.