Before I start, I must say that large armies are not the only way to challenge the civilised world. You can take civilisation apart with small and numerous raids just as well as with large campaigns. Now on the the main topic as I see it.

Population pressures would be one reason for the barbarians being able to get together large armies. They were actually much more numerous than the civilised armies because while the civilised armies (i.e. Roman) were institutionalised and very highly-trained and therefore smaller in numbers, the 'barbarians' came with entire populations of manhood, not merely a percentage, and these were not trained in any formal way except in the school of hard knocks. Their equipment was simple, and hence they didn't cost much to field compared to the high-logistics civilised armies. I have in mind here the Germanic peoples, since at the time the Roman Empire collapsed, the Celts had already been Romanised and looked just as good in lorica segmentata as any Italian, maybe even better. This is continental warfare as I see it.

With regards to Britain, the barbarian armies that attacked the island were not even armies, but hordes of raiders. These struck at the cohesion of the civilised society, not at any representative presence e.g. an army. They came and went so fast armies were practically useless against them, as well as most navies.

Furthermore, there was the most insidious method of challenge which is from within. Many of the armies that caused the fall of the Western Romans were actually settled and present within the Empire as foederatio, allies of sorts. Essentially it's a case of letting the wolf into your bedroom because you know your door isn't strong enough to keep it out. They took apart the Empire from within.

It isn't true that armies grew smaller post-Roman Empire, not immediately anyway. But as the Dark Ages set in, you do get forces numbering in the thousands instead of the tens of thousands like before.