I ranked Imagawa second in difficulty because I don't usually opt to take the "traditional" route with them (or the Takeda, the other split clan) when playing the Sengoku campaign. I abandon Kyushu (after milking its three nice provinces for koku as long as I can) and consolidate in central Japan. Same with Takeda--I abandon the west in order to take the east. This tends to lead to a more interesting game, though it isn't the "ideal" strategy.

I prefer to steadily expand (but not "blitz"). I've found that this tends to counter the big pile of money, fancy infrastructures, and elite troops that tend to result in 1.0x if you turtle. Hence, the initially poor factions (Shimmies and Mori) are more challenging for me; often I cannot expand as quickly as I'd like with them unless I manage things just right and get a bit lucky with my harvests.

I'd say the clan "specials" affect difficulty too. Cheap archers or cheap cavalry from the get-go make playing the Uesugi or Takeda easy for me. On the other hand, the Mori monk bonus is great, but can't really be leveraged significantly until income level is raised significantly. The Imagawa shinobi bonus may seem insignificant, but against the rebel-infested Oda early in a game they can be used to great effect.

I would imagine unit size makes a difference too, but I have always played with 60-man units so I can't really compare that to larger unit games.

I don't use the canned formations either, preferring to set up my own.