Until Thirty Years' War pikemen by default marched and fought in huge squares (or, so I've read, sometimes triangles) so catching them "off guard" with flanking moves wasn't really an option - those huge formations can at least in principle fight in all directions simultaneously.

Shooting the whole densely packed buch to bits is an option, and a good one, especially if you have decent field artillery available (the dreadful carnage Swedish guns wrought on the Imperial tercios at Breitenfelt was one of the more sobering object lessons to the deep formations' vulnerability). Gaining an upper hand in a regular missile fight, that is to say one fought with bows, handguns and crossbows, would have been dependent on the number and quality of missile troops each side could field. AFAIK the Swiss made a point of having around a third of their field armies made up of skirmishers...

The sad fate of the Scottish spear-hedges stemmed mostly from the fact that they were static formations - they lacked the drill required to maneuver without losing cohesion and breaking ranks, which the English exploited by threatening them with cavalry and thus forcing them to stay put and sending the longbowmen in to do the killing. The Swiss didn't have this weakness - indeed, their tactics emphasized aggressive maneuver, speed and shock impact.

Aside from pikemen of your own any solid heavy infantry seems to have done at least passably against the Swiss at least. They could pretty much brush off mounted men-at-arms with impunity, but if these elite troops left their horses aside (as was almost the norm by the later half of Hundred Years' War already) and fought on foot they could cause the lightly equipped pikemen some serious trouble. Odds are they didn't use their lances overmuch for this kind of combat - in a push of pikes the massive pikeman formation would have the advantage of weight and ranks. AFAIK they just took up their swords, axes and polearms and hacked their way through the pike-shafts, relying on their shields, armor and sheer skill for protection.