When it comes to Linux and systemd, the one thing on which everyone can agree is that it’s complicated — some say overly complicated.
Devuan, the Debian fork without systemd, has a page called Init Freedom on its site. At first the title seems ridiculously over-dramatic, because freedom in free software generally refers to licensing and not to software itself, and feels out of place in a discussion of init systems. Here, however, the title seems to refer to the conviction that systemd was foisted upon unsuspecting users by a distro elite, especially in Debian.
Although the topic of systemd is often met with profound indifference in some circles, I confess to deep-seated ambiguity whenever the topic is raised. Moreover, the more I research, the deeper my ambiguity grows, although in the end I finally reached a position.
I generally ignore the claims that systemd is part of Red Hat’s long-range conspiracy to dominate the Linux desktop. Although Red Hat occasionally makes a gesture towards the desktop, it mostly appears to be indifferent to it. Anyway, how would you prove such a conspiracy exists? The closest I can find to evidence is the way that GNOME Boxes lists Fedora releases out of order and at the top of the list of distros to run in a virtual machine — which is a pretty feeble piece of evidence.
Likewise, I ignore the rejection of systemd that amounts to a hate campaign against Lennart Poettering, its chief original developer, even though like countless other Linux users I have had many occasions to curse Poettering’s other best known project, PulseAudio. In addition, as Linus Torvalds complained, Poettering has sometimes been slapdash about fixing bugs and usually sounds high-handed. However, much of the hatred against Poettering is homophobic, which means that it automatically disqualifies itself from serious discussion.
Once I turn to design considerations and technological benefits, however, I quickly bog down. The Unix philosophy of one utility to do one thing very well sounds admirable in theory, and perhaps on the administrative level it works well. Yet, I have heard few if any complaints that large apps like SE Linux or AppArmor violate the Unix philosophy, and on the desktop, apps like LibreOffice or Krita are clear violators. Could it be that some projects require a different approach to be effective?
As a user, my initial tendency is to applaude Init Freedom’s rejection of “this one-size-fits-all vision” that led to the general acceptance of systemd in the major distributions. Yet as I think further, I wonder how practical supporting multiple init systems might be.
In the recent Debian general resolution on systemd, both sides maintained that expecting maintainers to support multiple init systems would seriously add to the task of keeping packages current. The Devuan project itself appears to illustrate the difficulty, since it still uses Debian 9 as its basis and so far has been unable to add multiple init support itself. When finding maintainers is a constant problem for many distros, perhaps a limit on the number of supported init systems is sheer practicality.
My uncertainty continues when I start learning more about the technical aspects of systemd. My fellow Progeny alumni and Debian maintainer, John Goerzen, says “I don’t agree with every design choice they’ve made” in systemd. For instance, he states that “the learning curve is somewhat higher, what with files being spread out across /lib and /etc and the interactions between them sometimes complicated.”
Yet, he has no trouble mentioning some of systemd’s benefits:
- Systemd timers, for instance, can help avoid the thundering herd problem that one might otherwise see with their RandomizedDelayStart. They also have several other features that make them more useful than cron jobs in some situations.
- For daemons, the service monitoring and restart is quite helpful. The dependency ordering is more expressive and reliable than what we had in sysvinit, particularly in situations involving complex ordering of filesystem mounts (certain NFS scenarios were a thorn in people’s side for ages with sysvinit).
- Journalctl shows useful information, especially for daemons that crash or exit without logging to syslog. It also simplifies development of simple new daemons, which can just log to stdout if they don’t have copious output to produce.
He concludes that, “while systemd doesn’t strictly provide things that were impossible before, it streamlines a lot of things. [For instance] we no longer need to have separate watchers for most daemons [and] don’t have to write random delays into cron jobs.” His assessment sounds much like that of someone who has no particular stake in systemd, yet who has learned to live with it.
Still, what strikes me is Goerzen’s reservations. Systemd is a layer overlaid on an already working operating system. From what I have been told, it requires numerous small patches to integrate with the existing system — and in my experience, increased complexity means more space for things to go wrong. Asked for an opinion, developer Christopher Smith gave the opinion that systemd “became overly ambitious/complex. On the security front in particular, it seems to be the source of problems at least as much as the solution.” A search shows that memory leaks in particular have been a problem.
More importantly, I find that lists like Goerzen’s tend to focus on benefits for programmers, not for average users. Less knowledgeable users like me, who maintain only a few machines at home, generally use systemd only to start and stop services, to check their statuses, or read an occasional log — and for such purposes, systemd is serious overkill. I can get what I need from a more dedicated init system like runit, which is quicker to learn and has little beyond those basic options that I might actually use.
To my way of thinking, that’s the strongest argument against systemd. For average users, it has far more overhead than it needs. In that respect, it resembles SE Linux on Fedora, offering sophistication that is not needed and creating problems that should not be there. It does not seem like something that should be installed by default on every system.
Systemd seems to be here to stay for those who run major distributions. Still, it first appeared almost a decade ago, so its time may be drawing to an end in a few years. Just as projects like PipeWire stand ready to replace PulseAudio, many of the alternatives listed on Init Freedom already offer alternatives to systemd. If systemd irks you sufficiently, you can easily try several of these alternatives on a live DVD. Otherwise, the solution may be to wait patiently until a more efficient consensus emerges.
Bruce Byfield has been involved in FOSS since 1999. He has published more than 2000 articles, and is the writer of “Designing with LibreOffice,” which is available as a free download here.
I think one thing to keep in mind is that a lot of the people evaluating systemd’s merits and drawbacks are already familiar with SysV init or even highly skilled with it. So they’re not really comparing two things from a neutral perspective, part of their irritation is leaving behind skills and scripts they already have.
I suspect that for most common tasks someone completely new to any init system and unfamiliar with shell scripting would find systemd simpler to grasp.
I’m strongly in favor of init freedom as a philosophy but I have no right to demand that other developers support multiple init systems. That’s as absurd as demanding that they write their programs in second and third programming languages, or port their programs to Haiku and Genode, or add specific features that I want.
My hope is that if the code quality or design decisions of systemd become too troublesome another fork like uselessd (which failed for lack of interest) will be tried.
There are a few more areas where systemd gives a benefit to the users not really needing to interact with it on a day-by-day basis.
* no matter which distro you use, if it is systemd based, you can find a common ground across all those distros to solve your challenge – using the same tools. For average users, this is more streamlining and unifying the toolbox across all distributions.
* Systemd unit files will work on all systemd based distros. For average users, this means higher probability of installed services to work with less maintenance and support work for package maintainers and user communities.
* systemd journal will rotate log files by default, no additional tool needed and package maintainers don’t need to configure and maintain the rotation policies. For the average user, this means no risks of filling up the disk with log data.
* automatic starting of services when needed. Lots of services on many distributions starts and runs in the background, even when not needed. systemd can tackle these use cases without additional tooling, plus once these services have started it monitors them just as it does with the other services already running on the system. Many services can also be extended to automatically shut down when no longer needed, avoiding wasting system resources on running services because you _might_ need it. For the average user this means, things works out-of-the-box and doesn’t hog down on system resources just by installing a new package.
* Since systemd uses D-Bus, there are lots of possibilities to write new types of management software – which will work across all systemd based distributions. One great example is this: https://cockpit-project.org/ … And if you use command line tools and third-party tools managing your system via D-Bus, they can be used in parallel at the same time since systemd is the main source of the management process. For the average user, simpler tools for specific tasks will definitely be helpful; somebody just needs to write them – and it doesn’t need to come from the systemd project. Did I mention cross-distro support basically out-of-the-box?
Meh…systemd’s design was always questionable and the further along it is, the worse it gets.
I guess it’s great if you don’t mind non-deterministic systems. Personally I had enough of that with Windows and no desire to repeat it.
Bruce, thanks for a much more balanced treatment of systemd in this article.
Concerning John Goerzen’s three points. As far as systemd timers avoiding the “thundering herd”, which I guess means everything trying to start at exactly the same time, you can avoid that by using the parallel startup s6 init or even the ancient sequential startup sysvinit. In systemd, the timers just fix the problem systemd introduced in the first place. As far as service monitoring and restart, s6 and runit have always had that. And just like journalctl, s6 and runit can log directly off of stdout and stderr, if that’s what you want. All three of John’s points make sense if the only alternative is sysvinit, but fall apart with the introduction of s6 and runit.
Bruce, you said you tend to ignore claims of systemd being a long term conspiracy to dominate the Linux desktop. First, I’d get rid of the word “desktop”. They pushed just as hard for systemd on the server: Otherwise everyone who cared would have used the sans-system server edition for their desktop, and the animosity never would have happened. Also, I’d suggest that you neither believe offhand nor ignore offhand conspiracy claims. Redhat makes their living with Linux consulting and training: A complex Linux is more profitable for them. One piece of evidence happened long before systemd, when then Redhat CTO Brian Stevens said “Red Hat’s model works because of the complexity of the technology we work with.” You can read the whole paragraph at http://asay.blogspot.com/2006/10/interview-with-red-hat-cto-brian.html , and search for the word “complexity”.
Your article attributes much of the hatred of Poettering to homophobia. That’s the first time I ever heard anything about his sexual preference. I think my reasons for disliking Poettering are much more common: 1) He developed and pushed systemd, 2) He behaves like time marker 23:02 through 26:35 on video https://www.youtube.com/watch?v=ZTdUmlGxVo0 , 3) he openly rejects POSIX and has been known to unfavorably compare Linux with Windows and OS/x: If I wanted something like Windows or OS/x I’d use them, and I’m not alone in this.
Your article mentions that some use cases require more sophistication. Absolutely right. But much of that sophistication can be addressed by technology available before 2014, and whatever sophistication that needed to be created could have been created without encumbering PID 1, and without assembling a monolith glued together by big, complex interfaces.
Mike S. opines that much systemd resistance comes from those familiar with sysvinit and unfamiliar with systemd. Maybe true, but if you have a user comparison test unfamiliar systemd against unfamiliar runit, and have that user deploy a few home-grown daemons, runit will win hands-down. We need to stop treating the situation like a choice between systemd and sysvinit.
Mike S. comes down strongly that developers not be required to support multiple inits. I couldn’t agree more. I’d take one more step and say developers shouldn’t be required to support any init: This should be done by specialists on systemd, sysvinit, s6, runit, OpenRC, Busybox Init, who will be in abundant supply in any medium to large distribution community. If the developer gives the init specialists the interface specifications to their software, the init specialists can do the rest.
Mike S hopes for a workalike like uselessd. That would be jumping from the frying pan into the fire, as uselessd implemented the same monolithicly entangled architecture as systemd. What’s needed is a simple PID1 added to a capable daemon supervisor. I’d recommend the well-maintained s6 init system, with a very simple PID1 but capable of very sophisticated startup. Anything truly lacking in s6 can be done separately, outside of PID1 and probably outside of s6.
Dsommers discusses cross-distro support. That’s a nice goal, but the constellation of systemd software tends to make all distros act like each other. Some folks relish the choice of distro behaviors. I use DIY-heavy Void Linux and would feel constrained by Ubuntu. A Ubuntu user would feel needlessly inconvenienced by Void. Vive la différence.
Mike mentions a dislike of non-deterministic systems. Both sysvinit and Epoch init are deterministic, and s6 can be made deterministic with the addition of s6-rc.
Bruce, it would be great if your next article would enumerate and describe various ways to transition out of systemd.
I never knew that Poettering was gay; it would be the furtherest reason to be unhappy with him. It is /his/ propensity and arrogance to “make Linux as he sees it”, not as we want it. And, as you said, he screwed up Pulseaudio, I still have problems with that today. If systemd remained ONLY an init system ALTERNATIVE, that would have been fine; but it’s got very serious bloatware symptoms and the more of that it has, the more tied in it must be to every other system component that it relies upon, even the kernel. It’s an extreme mess and a huge mistake; most of your arguments above are not close to why systemd is so bad or not supported at all universally and from everything I’ve seen, it has zero to do with Poettering’s own sexuality.
RHEL’s influence with systemd is way understated by your “research”, if you can call it research at all.
Devuan is very much based on Debian, it is by all accounts a tiny project that works hard to improve Debian. You can have Devuan with systemd if you want, but if you wanted that, then you would just use Debian. None of the other init systems are necessarily the work of Devuan — making things work without systemd is an important focus though. Devuan supports a number of init systems and doesn’t force upon it’s users systemd.
With all things, there are usually pros and cons, but as far as I am concerned the cons far outweigh any pro for systemd.
Is there any chance of more Ken Starks articles / posts? They have worth, I’m not sure why Bryan is even here there has been nothing worth reading; I guess the best part of him being here is that we can have easy arguments, but I would much prefer to argue over pov that at least make some sense to begin with.
Steve Litt,
Your argument that Red Hat has a financial interest in making Linux more complicated doesn’t hold for Debian, Arch, or most of the other major distributions that aren’t based on a financial model selling support services. If systemd caused more problems than it solved, it wouldn’t have spread post Red Hat.
I have set up daemons using systemd, and working with .service files is child’s play. You copy a .service file for another service, make the appropriate changes, run as root systemctl daemon-reload, and you’re done. It gets a little trickier if you have multiple services with different dependencies – but that’s trickier under any init system.
The original systemd documentation emphasized modularity. They never formally announced that they were abandoning modularity, at least as far as I know, but the modular build support bitrotted. The uselessd fork took an earlier systemd release with the idea of preserving modularity in all of the components and also stripping some of the modules out entirely. The name was both “useless-d” but more importantly “use-less-d”.
And for what it’s worth, I haven’t had a problem with PulseAudio since 2008, and Linux has been my primary desktop most of the time from then to 2014 and full time since.
I run Ubuntu flavors because I like to run the same versions I can recommend to Linux novices. But I have Void in a VM and I’ve run it on bare metal before. It’s a fine distribution.
@Mike S.
> If systemd caused more problems than it solved, it wouldn’t have spread post Red Hat.
I don’t believe that’s a valid argument. There were a lot of complex reasons for systemd’s adoption by other distros beyond thse that follow Red Hat closely. Less work for package maintainers to sustain more than one init system was a big one, along with camps of vocal systemd proponents embedded in the various distros (even Slackware). Supporting systemd makes it especially difficult to maintain support for anything else because it drags so many dependencies along for the ride.
I agree completely with Steve Litt’s comments. The false dichotomy between “systemd or sysvinit” has always been a very disingenuous way to frame the issues. The fork of systemd: uselessd, being an attempt to slot into systemd’s footprint, was never the solution people wanted who were looking for an escape from systemd, since that footprint is ever changing and ever expanding.
DUH,
Of course I forgot a major and most obvious reason for systemd’s adoption: GNOME.
Combine that with other factors and it’s a strong driving force for systemd adoption.