The trouble with infrastructure is that it breaks down and needs to be repaired, it wears out and needs to be replaced, and it gets destroyed and needs to be rebuilt. All that requires energy, resources, labor and money.
Conceptually, here's the problem we face. The bigger we make any part of our infrastructure--roads, pipelines, electricity grids, water and sewer systems--the more expensive it becomes just to keep it in operating order. The same is true for our industrial plant, transportation system, commercial buildings and private homes. Things fall apart over time; entropy makes sure of that. To keep things from degrading to the point where they cannot function requires resources, labor and money--all of which cannot be spent on new infrastructure or productive investment, that is, all of which must go to maintain what we have rather than grow the economy.
The ancient Romans came face to face with this reality. Expansion of the empire had been paid for with booty seized from conquered populations. But once the expansion stopped, so did the booty. The Romans increasingly had to tax themselves in order to pay for large armies to protect the now very long border and for the necessary improvements in roads and other infrastructure to maintain their administrative and military presence throughout the empire.
It didn't last. Eventually, the Romans had to pull back. They had to shrink the empire.