Fork of LXD Lands and Almost Immediately It’s Linux Containers’ Newest Project

About a month after Canonical pulled LXD from Linux Containers, with the ‘LXD community experiment’ evidently being labeled internally as a ‘failure’ by Ubuntu, the code is forked and almost immediately accepted as a project at Linux Containers.

The LXD story has turned into a bit of a soap opera, but one with a happy ending for those who support openness.

On Friday the LXD project that Ubuntu had removed from community development a month ago was forked as Incus, and then almost immediately the fork was accepted as a project by Linux Containers, LXD’s former home. While this was happening, the head honcho at Canonical/Ubuntu, Mark Shuttleworth, was commenting to a Hacker News post, in what seems to be something of a doomed attempt at damage control.

This was a neat cut and paste for the folks at Linux Containers. Almost all of the same community people who were in charge of the project when it was Ubuntu’s LXD are in charge at the project as Incus.

The Story so Far

You might remember that back in early July, Linux Containers announced in a statement that going forward it would no longer be hosting LXD because, “the creator and main contributor of the LXD project has decided that after over eight years as part of the Linux Containers community, the project would now be better served directly under Canonical’s own set of projects.”

When we reported the story on July 5, the reasons for Canonical’s removing the project from community control were unknown, but at least one reason came to light on July 10, when Stéphane Graber announced on his personal blog that he had resigned from Ubuntu.

Gabber had worked for Ubuntu for more than 18 years, and for the last eight years had been directly involved with the LXD project — first as the project’s technical lead, and since September 2018 as the projects engineering manager.

He indicated that Canonical’s decision to quit Linux Containers was at least partly tied to his resignation:

“Following the announcement of my resignation, Canonical decided to pull LXD out of the Linux Containers projects and relocate it to a full in-house project. That’s the news which we announced last week.

“I obviously wish that this particular change hadn’t happened, I strongly see value in having a project like LXD be run in a more open, community environment where everyone’s opinion is valued and everyone’s contribution, no matter the size, is welcome. Having the ‘LXD community experiment’ be labeled a failure within Canonical seems unfair to me and to everyone who contributed over the years.

“As for my particular involvement in Canonical’s LXD moving forward, I will definitely remain an active user of LXD and will likely still be filing issues and the occasional fix. However, I don’t intend to ever sign Canonical’s CLA, so should that become a barrier to contribution for the project, I will have to stop contributing to it”

Then on Friday, news began to break that LXD had been forked. Initially the reports were that the project had been forked twice, but it turned out that the first fork had been carried out by Gabber, who was aiding Aleksa Sarai, the person behind the “real” fork that’s now known as Incus and which is already a part of Linux Containers.

“I’ve just been helping out getting it to actually be functional and providing a bit of a laundry list of things I’d do differently should I be starting LXD from scratch now, which a fork like this now makes possible,” Gabber said as a comment on Hacker News.

Sari, the person behind the fork that is now the crowned heir to LXD, has been a senior software engineer at SUSE for nearly eight years. According to his LinkedIn bio, “I am the head engineer of the containers core team at SUSE, and am responsible for maintaining our container-related packages for SUSE Linux Enterprise (and openSUSE) as well as working as an upstream maintainer of several Open Container Initiative projects. I also work in the Linux kernel community to implement features that help improve the security and functionality of container runtimes.”

Things moved quickly after the fork was announced. Today, which is the next business day after the fork happened, Linux Containers announced that not only had Sari asked for the newly forked project to be considered for membership, but that membership has been granted:

“After some discussion with Aleksa and a fair bit of encouragement from our community, we have made the decision to take Incus under the umbrella of Linux Containers and will commit to it the infrastructure which was previously made available to LXD.

“The goal of Incus is to provide a fully community led alternative to Canonical’s LXD as well as providing an opportunity to correct some mistakes that were made during LXD’s development which couldn’t be corrected without breaking backward compatibility.

“In addition to Aleksa, the initial set of maintainers for Incus will include Christian Brauner, Serge Hallyn, Stéphane Graber and Tycho Andersen, effectively including the entire team that once created LXD.”

This was probably an inevitable outcome, even though Canonical evidently didn’t see it coming. With LXD being an important subset of LXC, a lot of LXC developers would naturally like to see LXD remain at Linux Containers where its development would be more unified with the development goals of the greater LXC ecosphere.

Shuttleworth Does Damage Control

This, of course, isn’t a good look for Canonical/Ubuntu, which is now in the position of appearing to have overreached, and as a result an upstream project it owns is now likely to become a downstream project of its fork.

Mark Shuttleworth seemed very aware of this on Saturday, even before Incus was accepted by Linux Containers, when he began posting to a thread on Hacker News after a commenter wrote:

“My take, given the optics of the change: Canonical is likely looking to bolster LXD integration within Ubuntu and drop support for other distros. Which is fair, in my opinion. They’ve done most of the maintenance and work, they are allowed to prioritize their use cases. But it also means there likely does need to be a fork available to make sure that wide distro support happens, and LXD doesn’t become a snap store special for Ubuntu.”

Shuttleworth replied:

“That speculation is simply incorrect. We’ve never had a discussion about dropping support for other distros. Open source is better when more people use it, and that means its better to have it used on other distros.

“We’ve always tried to be at the forefront of new kernel capabilities – especially security and container tech – and it helps that Ubuntu generally has very modern kernels. On Ubuntu we can make releases of the kernel and LXD that line up nicely. Other distros with older kernels have always been supported as well as possible, and I don’t see why that would not continue. There is certainly no plan at Canonical to inhibit that.”

I suspect it’s going to be difficult for Ubuntu to walk away from this misstep without taking some sort of a hit to its open-source credibility. It appears that Shuttleworth is aware of this, judging from his reply to a commenter who spoke of what he saw as the benefits of community developed software over software developed by commercial entities:

“It’s quite right that commercial entities tend to focus on commercial imperatives. But Canonical is unusual – I founded it precisely to support a more open approach to open source than I was seeing from the other enterprise Linuxes in the early 2000’s. We have a nearly 20-year track record of balancing community and company interests in Canonical and Ubuntu, in part because I have sufficient control of the company to stay true to that original vision.

“Of course, things may change at Canonical if I am no longer involved. That’s a reasonable risk to think about and have a plan for. Some paranoia is constructive. One of the nice things about open source is that you can fork it if you want to. But to do so just because Canonical might in future take a different view than we have to date seems like its paying too high a price for that paranoia 🙂 .”

LXD and Linux Containers

This might seem to be a lot of commotion for some software that most people haven’t heard about, but like much of the software that drives enterprise infrastructures these days, just because it isn’t famous doesn’t mean it’s not important.

LXD is a Linux container manager that’s built on top of LXC (which is also software that’s not above everybody’s radar), to provide a better user experience for those relying on Kubernetes and containers. Essentially, LXD is a container hypervisor providing an API to manage LXC containers. LXC is a popular Linux container runtime that consists of tools, templates, and library and language bindings, which DevOps types like because it’s flexible, and because it covers almost all features supported by the Linux kernel.

Linux Containers is the umbrella project behind LXC. In addition, it hosts a variety of other projects related to LXC, such as LXCFS (a userspace filesystem), distrobuilder (an image building tool for containers and virtual machines), libresource (a library of interfaces for obtaining system resource information), and lxcri (an LXC wrapper that’s used as a drop-in container runtime replacement).