The increase of firepower between 24pdr and 32-36 was relatively marginal. A 24pdr would smash almost anything on it's path anyway. Though I don't doubt for shooting away thick masts the big guns had an edge.
But SOL were not required to only fight ships. In a shore action the bigger guns would be of significant advantage over smaller cannon against fortified gun batteries, both in stand off range and effect on target.