The Real Risks And Costs Of Using Cloud Services: The Freedesktop.org Inside Scoop
Benjamin Tissoires, who works for IBM subsidiary RedHat, shared some interesting and shocking insights into the very real cost of cloud hosting a huge free software community like Freedesktop.org in his presentation "Status of freedesktop.org gitlab/cloud hosting" at the The X.Org Developers Conference 2020. For them, a initially very small monthly cloud hosting bill grew over time and eventually became gigantic. The effect of getting "free" cloud hosting for a period of time were specially interesting: They did not notice that the "sponsored" and temporarily hidden monthly bill was growing rapidly and got a huge wake-up call when monthly bills returned and they turned out to be massive.
written by 윤채경 (Yoon Chae-kyung) 2020-09-20 - last edited 2020-10-11. © CC BY
"Cloud" is just a marketing term for CPU-time on someone else's server somewhere. The artwork in this picture, taken in Norway's capital Oslo in 2006, has absolutely nothing to do with this story beyond there being very different kinds of clouds depicted in this work of art.
"Cloud hosting" is just another word for resources someone else's computer somewhere. Those resources are usually priced and billed based on actual use, not a fixed amount. It differs from traditional dedicated server hosting where the monthly fee tends to be fixed. The server hosting this website, as a simple down-to-earth example, has a fixed $50 monthly fee attached to it. It makes zero difference if it is completely idle or running at a 100% CPU load the entire month, the bill's the same at the end of the month regardless of what that server has been doing. That is not so with "cloud" hosting, a months bill for a cloud service could be $5 or $5000 depending on how many resources it has been using.
Enter Cloud Hosting[edit]
freedesktop.org moved to cloud hosting in December 2017. They started out with a tiny $20 monthly cloud hosting bill that first month. The bill increased to $348 in January 2018. It stayed manageable between $350 and $400 until August 2018. Several big free software projects, namely Mesa, were added to their cloud-hosted services at that time.
freedesktop.org's monthly cloud hosting bill rose to $2043 in November 2018. That increased to $3039 in March 2019.
The Freedesktop.org projects monthly cloud hosting bill saw very rapid growth between September 2018 and March 2019.
The sharp rise in the Freedesktop.org projects cloud hosting bill did not stop there. It got worse. Much worse.
Google Sponsorship To The Rescue[edit]
The Freedesktop.org got a $30K grant from Google in March 2019. It was very kind of them to give Freedesktop.org a "grant" in the form of sponsored cloud hosting. Google would kept hosting the Freedesktop.org's services and paid the rapidly creasing monthly cloud hosting bill using said "grant". The result was that nobody at Freedesktop.org got to see how much Google was "billing" them using the "grant" money.
Free sponsored cloud hosting looks fine until the "grant" money runs out and bills return and you realize it's grown to 4K per month.
The Freedesktop.org bill had grown to $3869 in November 2019 and that rose to a whopping $6317 in January 2020.
"The growth was exponential, we were using the cloud more and more, and the costs were piling up every month. It was completely out of control and nobody was actually watching it."
$6317 per month is substantially more than the comparatively tiny $350 monthly bills the Freedesktop.org project got the first half of 2018.
Taking Measures[edit]
Panic set in within the Freedesktop.org project when they realized that a $6000+ monthly bill would lead to a gaping hole in their yearly budget.
Benjamin Tissoires, who works for IBM subsidiary RedHat, and others involved with the Freedesktop.org project, begun to analyze the now very huge monthly bills to see where all that was going.
"I don't know if anyone listening to the stream has ever had a cloud bill, but it's really complex to get through it. This is an extract, it is actually more condensed than what you got on your bill because I summarized everything as a table but you got all of these objects, and you don't know what you're paying for."
A breakdown of Freedesktop.org's $6317 cloud hosting bill for January 2020.
A big effort and lots of time invested in reducing usage of the more expensive service resulted in a lower monthly bill hoovering steadily around the $3000 mark the the second half of 2020.
The freedesktop.org cloud hosting bill has fallen down to "only" $3000 per month thanks to huge efforts by the freedesktop.org infrastructure team..
The Alternative To Using "Cloud" Providers[edit]
Cutting a $6317 bill down to slightly less than half and ensuring that the bill remains that that level is sort-of impressive. We should be impressed. But we're not. We are, in fact, unimpressed and we find the whole ordeal to be ridiculous. And here's why:
| Dedicated Root Server (full access) | Servers | Price |
|---|---|---|
|
1 | $80/month |
| 38 | $3040/month | |
|
1 | $40 |
| 76 | $3040/month |
$1600, or half of what freedesktop.org is paying, gets you 20 dedicated Skylake-grade servers or 40 dedicated Haswell-grade servers with Gigabit networking at your average dedicated server hosting company. We're not talking about prices who include rebates you'd likely get if you're Freedesktop.org and you say you are willing to have the hosting providers logo on your website, $80 and $40 is what these kinds of servers will cost you if you go ask for one. You'll probably get a better deal if you ask for a quote on 20.
Reducing a $6317 bill to around $3000 sounds very impressive if you only look at those numbers and ignore any broader context. The, to me, obvious question is: What kind of requirements would they have that couldn't be covered by 10 dedicated servers on unmetered connections in a well-connected data-center? $80*10=800, not $3000.
"How do we reduce our cloud hosting bill" is the question the Freedesktop.org project jumped on when their cloud hosting bill reached $6000 per month. Another perhaps less obvious question worth asking is: "Is cloud hosting, with a variable cost depending on utilization, the right solution?
Consider how it looks if you simply ignore where their bill topped out at and compare the monthly bill when they started using cloud hosting in 2018 with the current one: That's $350 pr month the first half of 2018 vs $3000 per month the second half of 2020. That's a 8.5x increase in two years. It's also $3000*12 = $36000 per year just for some cloud computer power and some bandwidth.
Do note that a general disgust for cloud services is likely coloring my unimpressed point of view. I have personally never liked the idea of using any kinds of "we can't give you a price up front, you will know what the bill is when you get it" services. And that's what all cloud hosting contract looks like. Dedicated server contracts are more predictable, if it's $50/month then the bill will say $50 no matter how much work it's done.
It must, of course, also be noted that those who actually manage freedesktop.org's infrastructure have a much better idea what their real needs are, I've only got an outside view of what some of their services look like based on the X.org developers conference presentation and a general idea of how those would be set up and managed. Hosting a GitLab-based service isn't black magic, it's just software with slightly higher than average I/O and bandwidth requirements.
Benjamin Tissoires: "Status of freedesktop gitlab cloud hosting" At XDC 2020[edit]
Benjamin Tissoires's "Status of freedesktop gitlab cloud hosting" presentation at the X.Org Developers Conference 2020, embedded below, is worth watching if you are interested in how cloud services are billed or just want to learn more about the freedesktop.org project in general. It's about 40 minutes including the Q&A at the end.
Copyright X.Org Developers Conference 2020. Licensed under Creative Commons CC-BY.
You can learn more about the freedesktop.org project and their many huge contributions to free software and the free software community at freedesktop.org.
Enable comment auto-refresher