I could see how the theory might be appealing to small-government advocates if it weren't such an obviously flawed idea. If the beast can borrow money then it will do just that, and you'll end up paying for those expenses sooner or later with interest to boot.

I recall that in the 2011 debt ceiling standoff Obama said to the public that if the ceiling wasn't raised in time, social security payments + medicare + whatnot would all have to be suspended. This is exactly what "starving the beast" ultimately means in practice, but of course the republicans accused him of fear mongering. Did republican congressmen ever threaten Reagan or Bush junior* with the debt ceiling? I'm guessing not.

(*Bush sr. actually raised taxes to increase revenue, so he does not belong here with those other two)