The cloud is cost-effective when you start, because you can start using a cheap or even a free plan, and get many pre-configured services.
The cloud is cost-effective when your load is highly spiky, so you can spin up a ton of capacity during the spike, but not pay for it during the lull. (I worked in a company for which this was a principal case.)
If you are a relatively large company and want compute mostly for internal IT needs, especially for sensitive data, it may be cheaper to run your own server room and a private cloud. It won't be cheap in absolute terms though.
I work as an IT Infrastructure Architect for VMware and Windows Based Enterprise workloads and a lot of the time it's just more cost effective to have the servers in house for many of the workloads. It isn't the case for ALL workloads, for example we move almost all exchange email services to office 365, in combination with Azure AD for active directory service. As far as datacenter space and power usage, a lot of the time the server room is located in the utility room / closets, they already have the space and the building and the power infrastructure. Typically they have vendors that require VM's because the vendor software is not cloud based, many of these enterprise software can consume quite a lot of CPU/RAM/DISK and are very latency sensitive. Having a couple of powerful vmware hosts and an all NVMe SSD SAN is really really nice for performance and speeding up these applications and not paying 4K+$ a month for equivalent hardware in the cloud. You can even argue against the capital investment cost because a lot of this gear you can lease directly from DELL, and have full service contract for 7 years 24x7 4 hour SLA on hardware.
Depending on what your company does even if it starts out and gets free credits through the AWS startup program it still might not be cheaper.
But it's more flexible and a lot of startups often pivot a view times.
Through if you have high compute and somewhat longer term predictable levels of compute then e.g. renting a rack in a server center and placing your hardware in can be notifiable cheaper (when your book keeping splits the initial buying costs across the years you run the hardware)
And then there is one huge lie about cloud, that it's way less work. It's not. It's a mess. I mean seriously we have a few docent years now of experience about what is user friendly and what is not when it comes to API/interface design and I would argue ~90% of cloud design from AWS is just plain terrible. That costs time, and time is money.
1) a lot of startups were never about building a sustainable business but to live the "startup founder" lifestyle and provide cushy jobs for your friends. In this case, complexity is a feature as it gives you an excuse to raise a lot of money, hire a lot of people and give those people tons of busywork to do (which they wouldn't have with a simpler stack).
2) even for startups who do want to build a sustainable business, VCs are often close-minded and anything that doesn't fit the typical "engineering playground" mold described above may fail technical due diligence despite it being better from an engineering point of view. So cloud and overengineering it is.
> "startup founder" lifestyle and provide cushy jobs for your friends
I have yet to see a startup for which that is the case, like at all. Idk if it's a SV thing but in most of the world this won't fly.
Most important you are missing the most important reason why startup go with the cloud.
Flexibility.
Startups often pivot multiple times in various aspects before they find a way approach substainability. Like common are changes in monetization model and changes in target audience. This often also comes with major changes in requirements for compute/hardware. This isn't just about amount of resources but also kind of them. Furthermore it also often comes with changes in architecture where services provides by AWS and similar can make such changes much faster/easier.
This means that for startups setting up their own hardware is a form of self-locking which startups can not always afford, and can even lead to a startup failing.
Naturally once a company has found a promising way to reach profitability the cloud vs. non cloud decision needs to be reconsidered.
A bit less cynically, a startup is a company designed for enormous growth. If not, it's just a sparkling regular business.
If your user base grows 100x in a year, there's no easy way to scale using your own hardware. But this is exactly the (rare) case the VCs are after. So a startup is required to build their tech in the cloud, even if in 99% of the cases it's an overkill and an extra expense.
If you have enough hard technical changes due to changes in monetization model or similar it might not even have any extra expenses. I think the company I currently work for might have been bankrupt by now (after due to a marked change our original monetization strategy started to dry up) if we hadn't taken advantage of the flexibility of the cloud. (But then we still might not make it, but at least we had a chance to make it which I don't see us having had we tried to have our own dedicated hardware.)
Then there are also some highly managed/opinionated "cloud providers" (e.g. not aws), while they tend to have even more absurd prices for compute power they tend to actually safe quite a bit man power (in difference to e.g. AWS where the person time savings seam to be completely eaten up by subtle complexity and need for optimizing deployments in ways you otherwise would not need to). Which can make them attractive for very small startups which very low compute requirements. Through such startups also tend to not have calssical VC.
Are you going to run your production from an old desktop colocated at the cheapest DC nearby?
You must know how to run a really scrappy startup!
All of a sudden, some post on Reddit brings in a wave of visitors and a large bunch of signups; your hardware barely manages. What's your next action? (Note: `kubectl scale` is not it.)
The cloud is only cost-effective if your workload fits into their loss leaders where they're actually giving you reasonably-priced services. Anything else and you're still getting scammed. It may be that the amount is too low to matter and the pragmatic decision is to keep going for the time being, but you're still getting scammed.
The fast scaling/up down is a good argument but you need to proceed with caution:
1) is your application actually structured to be able to do this? Would the overheads of developing your application in this way negate the savings (a fully distributed service with queues/etc has more development/management overhead than your basic monolithic MVC app, which may eat up all your savings)?
You need to do your own numbers here and be pragmatic. You'll have plenty of resume-driven people who see the potential of an engineering playground and will immediately tell you whatever it takes to make this happen, regardless of whether it will benefit the business.
2) how much does it actually cost to run your application at full scale outside of the cloud? I've seen places that do the whole autoscaling thing (and have lots more moving parts as a result) and yet a fully-scaled-up equivalent on bare-metal would cost less than the baseline scale on the cloud, thus removing the business case for it.
Keep in mind that the cloud effectively ate all the hardware improvements we've had in the last decade so switching to bare-metal hosting means you'll need a lot less hardware. A cloud "vCPU" is more like a pocket calculator when compared to a real CPU, and same with storage.
The cloud is cost-effective when your load is highly spiky, so you can spin up a ton of capacity during the spike, but not pay for it during the lull. (I worked in a company for which this was a principal case.)
If you are a relatively large company and want compute mostly for internal IT needs, especially for sensitive data, it may be cheaper to run your own server room and a private cloud. It won't be cheap in absolute terms though.