The AlmaLinux Dilemma: When Upstream Suddenly Disappears

“I wanted to know about how the folks at Alma felt about the recently formed cartel that includes never to be trusted Oracle; SUSE, which is recovering from a case of Stockholm syndrome that was brought about by several decades of abuse by Novell and the other owners that followed; and Kurtzer’s Rocky Linux, which might be a fox or might be a lamb, I haven’t yet decided.”

AlmaLinux had been on something of a three-year roll, when suddenly a move by Red Hat upset the apple cart.

I wondered what it felt like to have an open-source project that was young but essential, getting lots of uptake from both home users, but especially from the enterprise, and to suddenly have the upstream open-source project that was your source rip the rug out from underneath you by denying you access to your source.

“The initial reaction was frustration, because we finally felt like we were really buzzing along and understood what we were doing — understood how we fit into the ecosystem — and we had ideas for what to do next,” benny Vasquez told me in a recent talk on Zoom. Vasquez spells her given name all lower case. No, I don’t know why, and really don’t care. You shouldn’t either.

“This definitely shifted priorities and caused a complete change in where we were focusing, because when your entire pipeline gets blown up, you have to try something different,” she added.

Vasquez has been the board chair at the AlmaLinux Foundation since October 2021, about a half-year after AlmaLinux’s first release. She said there were signs that something was amiss even before Mike McGrath, Red Hat’s VP of core platforms, dropped a bombshell on June 21 and revealed in a blog post that the company would be removing the source code from its flagship platform, Red Hat Enterprise Linux, from git.centos.org, where it had always been freely available to open-source developers.

“We noticed that the updates weren’t shipping,” she said, and noted there wasn’t much concern over that, “because it’d happened before.”

“It was always considered a bug before,” she explained. “Things would make it into Red Hat before they made it into the Git at centos.org. We would just open up a bug report, and they’d be, ‘Oh, yeah, you’re right,’ and then they’d ship it. This time that didn’t happen, and by the time they put out that blog post we were already a little bit behind, because we didn’t see this coming at all.”

She thought about that for a moment, then corrected herself.

“Well, we didn’t have notification that it was coming.”

The RHEL Clone Ecosphere

Both AlmaLinux and Rocky Linux came on the scene in 2021 as a result of Red Hat upsetting another apple cart, when it announced in late 2020 that future new releases of CentOS would no longer be positioned as downstream clones of RHEL, but would be moved upstream, meaning future releases would no longer feature one-to-one compatibility with RHEL and so couldn’t be used as a free drop-in replacement for Red Hat’s bread and butter product

In the same breath, Red Hat said that CentOS 8, the clone of the RHEL 8 series that had been released about a year earlier and which had been scheduled to be fully supported through 2029, would lose all support at the end of 2021, a mere two years after its initial release. However, Red Hat wasn’t going to touch the timeline for CentOS 7. The clone of RHEL 7 would still be supported through June 30, 2024 as originally promised.

This announcement literally sent a shock wave through every corner of enterprise IT. In many ways, CentOS was even more important to the day-to-day operations of data centers and public clouds than the Red Hat distro on which it was based. This was especially true in the hosting arena, where many companies had been offering CentOS as their default operating system since the release of CentOS 3 in 2004.

The announcement of CentOS’s upcoming demise as a drop-in RHEL replacement quickly led to the creation of AlmaLinux and Rocky Linux to fill the void.

AlmaLinux was started by CloudLinux, then a 10-year-old startup whose flagship product is an eponymous commercial Linux distro that’s basically a RHEL clone that’s been security hardened to meet specific needs and requirements of web hosts. It was also built using CentOS as its source, which prodded CloudLinux’s founder and CEO, Igor Seletskiy, to quickly announce that the company would be spending at least $1 million to build a CentOS replacement distro, and begin putting together a community of developers from among his web hosting customers, who also needed to find a replacement.

Vasquez, who joined AlmaLinux as a board member in May 2021, just after the project’s first release, referenced those early days when I asked her to define AlmaLinux’s place within the RHEL arena as it exists outside of Red Hat.

“Our place has always been, honestly, to serve a community of people who used to use CentOS and still wanted to use that Red Hat downstream clone,” Vasquez answered. “It was really easy for us to see the void that was left after CentOS 8x was discontinued. As users of CentOS, we understood all too well how important it was to have that downstream clone. Thankfully, the people that were involved very early had all of the expertise needed to spin it up quickly. They’d been building Red Hat for a decade, so they knew what they were doing.”

Those years of experience building off of CentOS was also why the Alma Linux team was able to roll the first production ready version of its distro out the door almost exactly two months ahead of Rocky Linux, the other CentOS replacement that actually had a head start, having been been announced before Seletskiy announced Alma. Rocky Linux was the brainchild of Gregory Kurtzer, who created a RHEL clone called Caos in 2002, which was eventually folded into CentOS.

These days, Kurtzer is the founder and CEO of the enterprise high performance computing company, CIQ. He also lists himself as the president and chairman of the board of Rocky Enterprise Software Foundation, the organization behind Rocky Linux, but he’s actually more than that. The foundation is a public-benefit corporation, formed in Delaware, that is owned by Gregory Kurtzer. It’s also legally not a nonprofit, although the organization has declared itself to be “a self imposed not-for-profit organization.”

“I think realistically the biggest differences between us comes down to our structures,” Vasquez said when asked about Rocky. “We’re solving the same problem in different ways. The thing that we saw as the ultimate problem with CentOS is that it was controlled by a corporation and there were no checks and balances in it. So, what we set up was very intentional, to make sure that that didn’t happen with Alma. You can’t buy a spot on the board; no one company can have control of the board at all. We’re very intentional about making sure that the board doesn’t have more than one person per company.”

The AlmaLinux Foundation is also a 501(c)(6) non-profit organization, which unlike Rocky’s public-benefit corporation status means no ownership. This means that if AlmaLinux were to cease operations, it would be required to distribute its assets among other non-profits, unlike Rocky where Kurtzer would be able to bank any assets.

To make sure that the new foundation was laying the proper groundwork to assure an open organization, AlmaLinux took on Simon Phipps, currently the standards and policy director at Open Source Initiative, who served as that organization’s president from 2012-2015 and again from 2017-2019. During the earlier term, Phipps helped transition the organization towards a membership-based governance structure, and was credited with putting OSI on a firm financial footing in a way that assured that the organization wouldn’t find itself dependent upon funding from a single corporate sponsor.

“Honestly, he’s been instrumental,” Vasquez said of his input, “both from like making sure we don’t make really common early mistakes, and from understanding how things work in the open-source world — from its long history. That’s the kind of stuff that has made it much easier for me. Having Simon in my corner has certainly made me not make the mistakes that I definitely would have without being able to reach out to somebody who knows so much.”

Life After Red Hat

Not long after Red Hat said it was going to lock down its source code and make it unavailable to Alma, Rocky, and other RHEL clones, Rocky Linux issued a statement that basically said that they didn’t care what Red Hat wanted, that there are ways to access Red Hat’s source code — such as through its availability on public clouds like Amazon Web Services or through the Universal Base Images that Red Hat makes available for use in containers. Going forward, the organization said that it will use those to continue to create versions of Rocky Linux that are bit-by-bit copies of RHEL.

Although Red Hat makes the code available with the caveat that it can’t be redistributed, Kurtzer says that’s not in keeping with the GPL and that Red Hat can’t stop him.

“The clouds do not require a click through agreement to obtain the sources, neither do the UBI images,” Kurtzer told me in a comment on LinkedIn back in July. “The Rocky Linux legal advisors reviewed our plan and it is not ‘fraught with legal issues.'”

About the same time, Dirk-Peter van Leeuwen, Germany-based SUSE’s new CEO and a 17-year Red Hat veteran, said that he was spending $10 million of his new boss’s money to develop a line-by-line clone of RHEL and to create a foundation that will make the code freely available to whomever wants to use it. Then, just a week or so before Vasquez and I talked, SUSE, Rocky Linux, and Oracle (which markets a RHEL clone with a modified kernel called Oracle Linux) announced a partnership to make van Leeuwen’s vision a reality.

The board at the AlamaLinux Foundation decided to go in another direction, and announced in July that going forward its distro will no longer seek to be bug-by-bug compatible with RHEL, but would instead seek ABI compatibility with RHEL — which basically means that it will look, act, walk, and quack exactly like RHEL although some patches and a few lines of code might be slightly different.

Considering that everyone else in the Red Hat clone space is insisting on keeping line-by-line compatibility, I asked Vasquez why AlmaLinux decided to walk in this new direction.

“Red Hat has made very clear what they want from us and how they’re willing to work with us,” she said. “Attempting to subvert them in the way that they want us to work with them is only going to cause, from my perspective, insecurity in my community. I don’t want to try and pick a fight and have the community have to deal with the instability that comes with that. That’s why we ultimately ended up going ABI instead of trying to continue to be one-for-one with RHEL.”

According to Vasquez, this change isn’t drastic, and most users — even those who use a mixture of machines running RHEL and AlmaLinux — will notice no change.

“The people who are going to be most impacted are developers and people running really low level sort of stuff,” she said. “If you’re running a website, it’s not going to impact you at all.”

In fact, Vasquez says that in ways that matter most to the installed base, this new approach is going to be an improvement. For example, most bugs and vulnerabilities should get fixed quicker, since Alma’s maintainers will no longer have to wait for Red Hat to release patches.

“We’re pulling in patches from anywhere that we can legally get them,” she said. “We’ve pulled in stuff from CentOS stream, because it typically is pretty quick. We’ve pulled in some stuff from Oracle; we’ve pulled in some stuff from other random places where we can find the patches.”

“It’s still pulling RHEL patches, but from wherever we can legally get them,” she added. “That’s the biggest thing for us, we’re making intentional choices about what we’re doing, to continue to build the operating system in the most stable way.”

“Do you lose anything?” I asked. “There has to be some cost to not being able to be one-on-one anymore.”

Vasquez paused long enough to consider the question.

“Here’s the thing,” she said at last. “I would have said we’re going to lose vendor support, but the way that this is playing out and the fracturing that we’re seeing, I think we’re not going to lose vendor support. I think we’re gonna see vendors come along. They’re going to step forward and say, ‘OK, so RHEL, but also RHEL compatible.’ They’re going to start expanding what they support, because it’s going to be worth it to them to continue to be able to support alternative operating systems.”

Server Linux Through a Crystal Ball

I’ve been covering this story since before there was an AlmaLinux. Actually, longer than that. I’ve been covering this story since CentOS, as a downstream version of RHEL, seemed to have a long future. I have been covering this story since the days when CentOS’s existence was proof that Red Hat understood that the vision that gave birth to free software, FOSS, and open-source was about much more than computer software, but was born out of a cultural awakening that transcended the brief revolutionary renaissance that was the 1960s, an awakening that went all the way back to the mystery schools that were still around at the end of the dark ages to protect knowledge from the coming inquisition.

That’s how I wanted to frame my questions. That’s how I almost always want to frame my questions, but I know better.

So instead, I wanted to know about how the folks at Alma felt about the recently formed cartel that includes never to be trusted Oracle; SUSE, which is recovering from a case of Stockholm syndrome that was brought about by several decades of abuse by Novell and the other owners that followed; and Kurtzer’s Rocky Linux, which might be a fox or might be a lamb, I haven’t yet decided.

“Rocky’s thrown its hat in the ring with SUSE, which is going to spend $10 million to come up with a permanent solution and then sort of build their RHEL clone,” I said. “What do you feel about that? And is AlmaLinx going to take part in that?”

I figured she would think I was nuts. If she did, she didn’t show it.

“I will say there are lots of conversations happening,” she answered without hesitation. “I don’t want to talk about any specifics yet, but there are certainly a lot of conversations happening, because RHEL has historically been the standard, right? RHEL has historically held the like — ‘standard’ is the only word to put there — for what Enterprise Linux is. I think ultimately, what we’re gonna see come out of this is a bunch of different options for enterprise Linux. We’re gonna redefine what enterprise Linux is.”

“Some of those options have been there all along,” I said, my voice betraying an uncertainty about whether I was making a statement or asking a question. “RHEL has a reputation of being, if you want solidity, if you want state of the art, if you want uptime, this is the way to go. So, you think, maybe, Red Hat has shot themselves in the foot in a way with this — other than losing the faith and trust of a lot of people in the open source community?”

“It’ll be interesting to see how it plays out from a business perspective,” she answered, “because even if we just take the Netflix example, the reality is people will get mad about a change, but the dollars are what the company cares about. If the dollars don’t go away, then they feel justified in whatever decision they made. Netflix is my favorite example right now, because Netflix made that login tracking change that said if you’re sharing logins, they’re gonna start cracking down. And everybody was like, ‘Well, I’m leaving, I’m not gonna put up with this.’ But they also signed up something like five-and-a-half-million new users in the last quarter. So…”

Once Upon a Server

Truthfully, this whole thing with Red Hat moving CentOS upstream, and now trying to separate itself from thousands if not hundreds of thousands of users it once encouraged to use clones of its distro has become something of an existentialist crisis for me. Four years ago, in 2019, when I attended Red Hat Summit for the first time — the last Summit presented by an independent Red Hat — I looked up to the company and thought it represented the promise of open-source.

Even though I was past being old enough to know better, I really believed that the day when the deal closed and Red Hat officially became part of IBM would end up being a red letter day in the history of tech. I was convinced that Big Blue had opened its eyes and understood that open-source and openness within the organization was what could save IBM from irrevelancy, and that when IBM’s CEO Ginni Rometty left the building to enjoy her retirement that she would be replaced by Jim Whitehurst, who would bring the open organization to IBM, which would forever be owned by Red Hat.

Yes, I’m a dreamer.

“Do you think that Red Hat would have done this if, say, Jim Whitehurst was still still running the show and if the sole stockholder wasn’t IBM?” I asked.

“I honestly don’t know,” Vasquez said, surprising me by taking my question seriously, because that’s a question I ask often these days that no one takes seriously. “It’s one of those things that if we could look into a crystal ball and see an alternative future, it would be a very interesting thing. But the reality is that when Red Hat sold to IBM, it was the right move for them in that moment. If they hadn’t done that, I think we would see a completely different company. Do you know what I mean?”

“They probably would have been taken over by somebody else in some hostile takeover based on the pricing of their stock,” I said, sort of as if repeating the mantra of capitalist wisdom.

“Either that, or there would have been something else that happened that led us to a similar spot,” benny with a small ‘b’ said. “Because, we all see patterns in business, right? They happen over and over again, and this is another pattern that is being repeated. I don’t think there’s anything that could have drastically changed this trajectory.”