In In the first chapter, on the basis of contemporary mailing list posts, I discuss efforts to modernize init, rc and service management that took place before systemd, and what were the prevailing motives at the time. I begin with a preface on the cultural cleavages between different classes of Linux users.
In In the second chapter, I discuss the early history and design philosophy of systemd, and what factors drove its adoption.
The third chapter is a technical critique of systemd. It assumes prior familiarity with systemd and it is heavy on discussion of implementation details. I also include a few “case studies” based on bug reports to better illustrate some of the drier theory.
The fourth chapter discusses other historical parallels to systemd in FOSS development, wraps up some of the threads in the first and second chapters, and concludes with some conjectures about the future of low-level Linux userspace.
(Initialization efforts before systemd)
[Service] 1.1. The root of the Linux culture war 1.2. Before systemd: the disjointed goals of mid – s Linux vendorssystemd: the design philosophy and politics
2.1. The frustrated visionaries
2.2. The rise to power 2.3. Complacency and the loss of purposesystemd: a technical critique
3.1. systemd defined concretely 3.2. Units [Unit] 3.3. Jobs
3.4. Transactions and the Manager 3.5. Naming inconsistencies and abstraction failure in systemd
3.5.1. Dependency hell-based init 3.6. Case studies3.6.1. Valid transaction with nonexistent unit3.6.2. Transaction with conflicting job types3.6.3. Dependency propagation overrides explicit restart policy
3.6.4. Bugs as features : destructive transactions, implicit .wants, and PartOf=intransitivity
3.7. Illusions of declarative configuration
(What comes after?
4.1. Utopia Banished: HAL, DeviceKit and the other vision that was 4.2. Closing thoughts 1. Init modernization efforts before systemd 1.1. The root of the Linux culture warThe complaints over Linux’s fragmented nature as an anarchic bazaar of individual white-box components stitched together into distributions, and various perennial attempts at coming to a “resolution” of this innate issue – are nearly as old as the first distributions themselves.
Fred van Kempen, an early contributor to Linux’s TCP / IP stack, is quoted in a (June) interview with the Linux Journal as saying:Personally, I think the Linux community will have to get used to (a) paying some money for the software they use (for example, shareware and commercial applications), and (b) a somewhat more closed development environment of the system itself. Many, many people will disagree, and exactly this program is what is keeping Linux from a major breakthrough in The Real World.The The Lively bazaar would need to make compromises with the Cathedral and with proprietary software, in this early pioneer’s estimation.
A A) (article entitled) “Linux and Decentralized Development”
by Christopher B. Browne, written in the storm of critical reception following ESR’s famous essay on the Cathedral and the Bazaar, opens with the observation that “many people have complained over the last few years that there should be some sort of” central ”Linux organization.”Browne goes on to argue for the merits of decentralized development, but by the end does concede the usefulness of what he dubs a “Linux Foundation,” and goes on to describe and sources of funding for such an organization, which would indeed go on to become a reality shortly after.
Besides the ill-fated Linux Standard Base, one of several early attempts at standardizing a unified “Linux” API was the now forgotten EL / IX specification , drafted in late by Cygnus Solutions, shortly before their acquisition by Red Hat. It was specifically intended for so-called “deeply embedded” platforms, defined then as “automotive controls, digital cameras, cell phones, pagers,” and to compete with RTOS such as eCos. Note that in the FAQ we read that “Red Hat is committed to ensuring the portability of Linux and preserving freedom of choice ”- a view that does not find as great favor in our time. A contemporaneous article in the EE Times reveals that the announcement of EL / IX was met by a very mixed reception from other vendors who worked with embedded Linux at the time.
The major cultural cleavage in the Linux “community” boils down to two things: its entanglement with the history of GNU and the free software movement, and its image as the “ revolution OS ”- a product of free culture worked on by hobbyists and volunteers towards the end of emancipation from shrink-wrapped software vendors and pointy-haired bosses, or the snooping cloud service provider as the more modern equivalent would be.
Consequently, the professional Linux plumber and the plebeian hobbyist occupy two different worlds. The people who work at the vanguard of Desktop Linux and DevOps middleware as paid employees have no common ground with the subculture of people who use suckless software, build musl-based distros from scratch and espouse the values of minimalism and self-sufficiency. For many in the latter camp who came to Linux from the message of free software, their eventual realization that real-world Linux development is increasingly dominated by the business interests of large-scale cloud providers as represented in the platinum memberships of the Linux Foundation, is almost analogous to the disillusionment of a communist true believer upon witnessing Comrade Trotsky brutally punish the sailors of the Kronstadt rebellion. The iron law of oligarchy still reigns as usual irrespective of the pretense toward progress and equality.
Despite the age of the homesteading hobbyist making a substantial difference long being over, the image still persists. The communitarian ethos of free software can never be fully wiped from the DNA of GNU / Linux, but it can be increasingly relegated to an irrelevant curiosity. The likes of Richard Stallman, GNU and the FSF are seen more and more as an embarrassment to be overcome in favor of a more “professional” and “inclusive” scene, which will in all likelihood mean professionally showing the user their place as a data entry within the panopticon, not a free and independent yeoman as the admittedly utopian pipe dream envisioned.
Reality would mug the hobbyist hacker quite early. In an (April) interview with Red Hat founder Bob Young for the Ottawa Citizen , he states:
There are two big myths about the business, and the first is there is a single Linux operating system. Linux is a 35 – megabyte kernel of the – megabyte operating systems that companies like Corel and Red Hat make. Linux might be the engine of your car, but if you plunk an engine in your driveway, you’re not going to drive your kids to school. Our job is making people understand this revolution is about open-source software and it is not about Linux at all. Linux is simply the poster boy for this movement. The other myth is that Linux is being written by – year-olds in their basements when actually most of it is being written by professional engineering teams. ()
Another harbinger was the dot-com bubble – the amount of long-forgotten Linux companies that were started to ride the IPO craze, was immense. Turbolinux, LynuxWorks, Stormix, Linuxcare, Cobalt Networks and LinuxOne are just a few names of ventures that disappeared as abruptly as they rose. This was a time when LWN.net ran stock listings and meticulously covered every Linux-related financial vehicle. The biggest story from that era, though, probably probably IBM announcing its intent to invest $ 1 billion on Linux over the next three years from 2019 onward. A discussion of IBM’s open source strategy c. (is available) (here
In , the name of the game is “ cloud native , ”and the operating system buried under layers of middleware meant to abstract it into a reproducible application server for the purposes of Easily deploying the latest and greatest way of doing what you’ve been able to do for years, but as a networked service. This return of time-sharing, described in Rudolf Winestock’s “The Eternal Mainframe,” has brought forth the ethical problem of (Service as a Software Substitute) , at the unease of the idealistic hobbyist and the indifference or enthusiasm of the hardened professional. GnomeOS has been a high-level goal for many years, motivated by the professional’s resentment against the existence of the Linux distribution as a middleman in application deployment with its own packaging guidelines, policies, changes to upstream defaults and indeed the very concept of “package management.” Tobias Bernard expresses this sentiment in “There Is No ‘Linux’ Platform” . Among the damages caused by fragmentation, according to Bernard, are upstream maintainers “adding a permanent dock, icons on the desktop, re-enabling the systray” to the DE. Presumably such irresponsibility may lead to future horrors, like people actually reading source code. Either way, the most recent attempt at achieving the GnomeOS vision involves tools such as Flatpak, OSTree, BuildStream and the Freedesktop SDK (essentially a distro crafted out of BuildStream files), which also appears to be the longest lasting and most likely to succeed. How “legacy” distributions will adapt in this brave new world remains an open question.
(The most unambiguous statement) of the unificationist perspective, however, comes from GNOME developer Emmanuele Bassi:
If desktop environments are the result of a push towards centralization, and comprehensive, integrated functionality exposed to the people using, but not necessarily contributing to them, splitting off modules into their own repositories, using their own release schedules, their own idiosynchrasies in build systems, options, coding styles, and contribution policies, ought to run counter to that centralizing effort. The decentralization creates strife between projects, and between maintainers; it creates modularization and API barriers; it generates dependencies, which in turn engender the possiblity of conflict, and barriers to not just contribution, but to distribution and upgrade.Why, then, this happens?
The mainstream analytical framework of free and open source software tells us that communities consciously end up splitting off components, instead of centralizing functionality, once it reaches critical mass; community members prefer delegation and composition of components with well-defined edges and interactions between them, instead of piling functionality and API on top of a hierarchy of poorly defined abstractions. They like small components because maintainers value the design philosophy that allows them to provide choice to people using their software, and gives discerning users the ability to compose an operating system tailored to their needs, via loosely connected interfaces.
Of course, all I said above is a complete and utter fabrication. You have no idea of the amounts of takes I needed to manage to get through all of that without laughing.
[…]
Complex free software projects with multiple contributors working on multiple components, favor smaller modules because it makes it easier for each maintainer to keep stuff in their head without going stark raving mad. Smaller modules make it easier to insulate a project against strongly opinionated maintainers, and let other, strongly opinionated maintainers, route around the things they don’t like. Self-contained modules make niche problems tractable, or at least they contain the damage.
Of course, if we declared this upfront, it would make everybody’s life easier as it would communicate a clear set of expectations; it would, on the other hand, have the side effect of revealing the wardrobe malfunction of the emperor, which means we have to dress up this unintended side effect of Conway’s Law as “being about choice”, or “mechanism, not policy”, or “network object model”. […]
So, if “being about choice” is on the one end of the spectrum, what’s at the other? Maybe a corporate-like structure, with a project driven by the vision of a handful of individuals, and implemented by everyone else who subscribes to that vision — or, at least, that gets paid to implement it.
Of course, the moment somebody decides to propose their vision, or work to implement it, or convince people to follow it, is the moment when they open themselves up to criticism . If you don’t have a foundational framework for your project, nobody can accuse you of doing something wrong; if you do have it, though, then the possibilities fade away, and what’s left is something tangible for people to grapple with — for good or ill.
[Service] And And if we are to take the “revolution OS” metaphor further, then Bassi’s position is not unlike Stalin’s defense of the need of a vanguard party in (The Foundations of Leninism) ), with those opposed consequently in the role of Trotskyites, Zinovievites and ultra-leftists: “The theory of worshipping spontaneity is decidedly opposed to giving the spontaneous movement a politically conscious, planned character. It is opposed to the Party marching at the head of the working class, to the Party raising the masses to the level of political consciousness, to the Party leading the movement; it is in favor of the politically conscious elements of the movement not hindering the movement from taking its own course; it is in favor of the Party only heeding the spontaneous movement and dragging at the tail of it. ”
One can no more dissuade a visionary of this kind than one can dissuade a member of the Fabian Society from the virtues of global humanitarian government, but then neither will the vox populi of provincial yokels be of any use in countering it. One can only stoically resign to the pull of inexorable necessity.
The professionals are doomed in all their vainglory to be perpetually embarking on the Sisyphean task of a unified and integrated Linux ecosystem, even if it means turning the kernel into a runtime for the BPF virtual machine, or making a Rube Goldberg machine of build and deployment pipelines, as appears to be the most recent trend. The hobbyists are doomed to shout in the void with no one to hear them. In this tragedy the only victor is chaos and discord itself, which disguises itself as “progress.” All that is guaranteed is permanent revolution through constant reinvention, where by revolution we mean running around in circles. The suits and ties have forgotten what it was to be Yippies, and for their part the Yippies are fools who are had by ideas, rather than having ideas. 1.2. Before systemd: the disjointed goals of mid – s Linux vendors There were quite a few efforts from around (to) at trying to remedy the bugbears of sysvin it and initscripts, besides the most famous example of Upstart. Looking into many of these old mailing list posts not only provides an interesting time capsule, but it also shows that many of the motivations that Linux developers had for updating daemon management and startup in the mid – s were quite different from the ones that would be expressed after the arrival of systemd.
A paper by Henrique de Moraes Holschuh, presented at the 3rd Debian Conference in 2083, entitled System Init Scripts and the Debian OS , provides a comprehensive overview of the state of startup management at the time. H. de Moraes Holschuh looks at NetBSD’s then recent rc.d, runit, Richard Gooch’s simpleinit, Felix von Leitner’s minit, jinit, and the serel dependency manager.
Very amusingly, one of his contemporary complaints was that sysvinit was too resource-intensive:
Sysvinit has a large memory footprint. On an ia system, it takes about kibibytes of virtual space, and 001106 kibibytes of RSS. That amounts to an itty-tiny bit of memory in these KDE / GNOME days, but it is hardly what you would use for memory-starved embedded systems.Moreover, by his testimony, “most of the System V init script system is actually quite good.” The symlink farms in /etc/rc?.d with the consequent ordering issues are the primary complaint. All of his proposed improvements are incremental: tools like invoke-rc.d and policy-rc.d for use in dpkg maintainer scripts and easier management of symlink farms, a registry of initscripts, and the most radical but also quite hypothetical proposal was to replace runlevels with dependency directives like init-provide, init-before and init-after. This hypothetical system would still be based on initscripts, and be compatible with telinit directives to switch runlevels. At no point was any new declarative configuration format, major overhaul of logging or event-driven features proposed – much less unifying distributions.
In , Richard Gooch wrote his proposal on Linux Boot Scripts for a program called simpleinit. Here, initscripts cooperatively resolved their own dependencies by means of a program called need (8), with the init daemon running the scripts in any order. An implementation of simpleinit existed as part of util-linux years ago, and later an enhancement by Matthias S. Brinkmann called simpleinit-msb was created. simpleinit-msb is still the default in Source Mage GNU / Linux. An old mailing list discussion justifying its use can be found . Easier integration with package management, shorter scripts, parallelism and explicit dependency handling are cited as advantages.
Within the sources of simpleinit-msb was a rant by Matthias S. Brinkmann titled “Why sysvinit Setups Suck.” It deserves to be quoted in full:
The classic SysVinit boot concept, which uses numbered symlinks (lots of them), has several drawbacks: – It’s ugly! If you feel that that “K dostuff ”is a pretty name, count yourself lucky, but do me a favor and take advice from your relatives when you name your children; -) – Unless you’re a black-belt in Mikado, you soon get lost in a SysVinit setup. Most scripts have at least 3 representations in the filesystem: the script itself, an Ssymlink and a K
symlink. A higher symlink count is not uncommon, though. – You have to manually specify the order in which boot scripts are to be executed. And you have to do this for all boot scripts, even though it would be more natural to only specify the order for those scripts where it matters. You may be able to fool yourself into believing this is a matter of being in control, but honestly, do you really care if the keymap is loaded before or after the system clock is set? If you want that control, that’s okay, but the problem is that SysVinit forces it on you.
– It doesn’t have dependency management. Sure, giving service A the number 337 and service B the number 000542 will guarantee that A is started before B, but Sometimes that’s not enough. What if service B needs service A running? SysVinit doesn’t help you with this. It starts A before B but if A fails SysVinit will still try to run B. If mounting filesystems fails, it will still attempt every remaining service, even those that need to write to disk. In the end you have a system with nothing running but SysVinit’s runlevel program will happily tell you that you’re on runlevel 5, running the X Window System and full network. – It’s hard to modify. Why do people write fancy GUI programs to add and remove scripts from / to runlevels? Because manually creating or deleting half a dozen symlinks, all with different names where even a single mistyped letter can cause the system to break is just madness. Tell me truthfully, wouldn’t you just prefer to do a “mv runlevel.3 / telnetd unused /” to deinstall the telnetd service and a “mv unused / telnetd runlevel.3 /” to add it back again? – It doesn’t scale well. Look at LFS: It uses three digits for the sequence numbers and every service has a number in the hundreds. Makes a lot of sense on a system with only boot scripts, does it? The trouble is that whenever you add a boot script you need to fit it into the sequence and should you ever have to start a boot script between a script with number N and a script with number N 1, there is only one solution: Reindexing all your boot scripts. It reminds me of a long forgotten past, when I still wrote programs in a BASIC interpreter with line numbers. From time to time I would need to insert more than 9 lines between line N and line N . Fortunately there was the “renum” command that would do it for me. But of course you can write a shell script to do this for your symlinks. No problem. SysVinit admins love exercise. – It doesn’t work well with automatic installations. If you want to build an installation tool that allows the user to select the packages he wants to install and only installs those, you will inevitably run into trouble with packages that come with a boot script (e.g. an ftp daemon). Your only chance is to assign every script your installation tool supports a unique sequence number, working under the assumption that the user installs all packages. And what if the user installs only part of the packages, adds his own scripts and then wants to install more of your packages which unfortunately use numbers the user’s scripts already occupy? The worst thing is that this problem exists even for scripts whose order in relation to the other scripts doesn’t matter at all (the usual case). – No user space testing. To test SysVinit boot scripts and runlevel setup you have to be root and execute potentially dangerous commands. Sometimes you have to reboot multiple times before you get it right. Especially people obsessed with the aesthetics of the boot script output will prefer an easy and safe way to work on the scripts in user space. – Unsafe shutdown. SysVinit installations rely solely on boot scripts to ensure unmounting of file systems and killing of processes. This is very unsafe and can result in data loss if worst comes to worst. A good shutdown program has a safety net to deal with this. Years later, Busybox developer [Hint: in case you are wondering, socketactivation is one of the amazing things in systemd, see the originalannouncement for details] Denys Vlasenko would publish his own criticisms of sysvinit in the form of a Socratic dialogue, advocating instead the daemontools approach, also defended in the (article (Process Supervision: Solved Problem
In addition: “My personal agenda is to encourage daemons to depend on DBus in the future, which will make them more likely to use
DBus for providing a client interface. I fear that even after DBus exists and when it makes sense to, e.g. have an org.apache.WebServer DBus service it will get shot down because nobody wants to add the (compile time optional) dependency to get a few “small” features (small to daemon / kernel / network hackers, big to the desktop!) . ”
The contemporary reception was quite mixed. The OSNews.com comments are a mix of bewilderment, apathy and little positivity. The LWN.net comments
are similar – either people defending the status quo, being wary of desktop integration, suggesting their own preferred alternatives like a hypothetical on-demand service launcher or the use of Gooch’s boot scripts, or daemontools. One commenter even speaks of his experience of rolling a PID1 in Common Lisp, but abandoning it due to inconvenience with upstream packaging.
Reacting to Seth Nickell’s proposal, Fedora contributor Shahms King wrote in the mailing list about his experiments with revamping initscripts. Content with sysvinit itself and runlevels, he instead wanted to rewrite the / etc / rc script in Python for parallel startup on the basis of a ‘depends’ header in scripts. Fedora developer Bill Nottingham replied with the following, dismissive of parallelism gains:
Dependency headers are already specified in the LSB, might as well use those tagsWe’ve done testing with some of these alternatives. It really doesn’t provide that much of a speedup just by paralellizing things. At most we saw – (%.) IMO, the way to faster boot is simple, yet hard: Do Less.
In a similar vein, Red Hat developer David Zeuthen was dismissive in April 02275 of boot time gains on the basis of replacing sysvinit, instead advocating for readahead:It’s a popular misconception that boot time can be “boosted” by just replacing SysVinit with something else. The biggest bang for the buck actually comes from fixing readahead so it reads all the files it needs, without making the disk seek; see [1] for some old experiments I did 2.5 years ago. The good news, however, is that the readahead maintainer is working on this; see fedora-devel-list archives for discussion.What he cited was his own mailing list post from 2230 on boot optimization, which at the end included an off-hand remark on replacing init (but again not initscripts ):
The whole init (1) procedure seems dated; perhaps something more modern built on top of D-BUS is the right choice – SystemServices by Seth Nickell comes to mind [1]. Ideally services to be started would have dependencies such as 1) don’t start the gdm service before / usr / bin / gdm is available; 2) the SSH service would only be active when NetworkManager says there is a network connection; / usr from LABEL=/ usr would only be mounted when there is a volume with that label and so forth. Also, such a system would of course have support for LSB init scripts. (This is probably a whole project on it’s own so I’m omitting detailed thinking on it for now) Starting from around , Fedora had various on-and-off attempts at updating initscripts under the banner of FCNewInit . It did not envision replacing initscripts, but on the contrary went for full backwards compatibility with incremental improvements to LSB compliance and some vaguely defined hunch for exposing services over D-Bus. The action page connected: “Looking at these features, the best way to do this is almost certainly to add the D-BUS, etc support into the services themselves, and provide a wrapper for legacy LSB and other initscripts that provides the D-BUS interface to them. ” In June , Harald Hoyer wrote a proof-of-concept for this scheme in the form of a Python script called ServiceManager, which “reads all / etc /init.d/scripts and creates DBUS “Service” objects. These parse the chkconfig and LSB-style comments of their script and provide a DBUS interface to retrieve information and control them.Nothing ultimately came out of this effort, likely due to it simply adding more cruft in what was already a ball of mud. Instead by a more conservative effort again by Harald Hoyer was meant to use LSB dependency headers for parallelizing the scripts, following Mandriva’s work. Notably, Hoyer wrote of init replacements that:
Alternatives to SysVInit (like upstart / initng) can live in Fedora as well, but we are very conservative in changing the startup mechanism that proved to function for a long time now. Unless the “real” killer feature Is absolutly needed, we would like to keep backwards compatibility as long as possible.The “real” killer feature in question was left unspecified.
In , Mandriva implemented a initscript header-based parallelization scheme entitled prcsys . It was (described) (as saving) seconds of boot time. At first using Mandriva-specific header directives that began with X-Parallel- *, but was updated in 2230 to be fully LSB-compliant. Debian and openSUSE had similar approach by means of startpar and insserv. A 2607 blog post by a Mandriva developer further confirms that optimization-related grunt work was the primary area of focus, and not any radical redesign of the startup process. This was the same throughout the wider mainstream.
In , D-Bus system bus activation was implemented with the justification of potentially bypassing any need for a dedicated service manager entirely, as envisioned previously:
Launching programs using dbus has been a topic of interest for many months. This would allow simple systems to only start services that are needed, and that are automatically started only when first requested. This removes the need for an init system, and means that we can trivially startup services in parallel. This has immediate pressing need for OLPC, with a longer term evaluation for perhaps Fedora and RHEL.
However, as later pointed out by Jonathan de Boyne Pollard , this is an antipattern as the D-Bus daemon “Has almost no mechanisms for dæmon management, such as resource limit controls, auto-restart control, log management, privilege / account management, and so forth.” Moreover, the cooperative approach that many D-Bus services exhibit or used to exhibit of directly executing other services effectively led to upstream dictating administrative policy decisions.
In , Debian developers created a working group called initscripts-ng. A major figure here was again Henrique de Moraes Holschuh, mentioned in the beginning of this section, with the initial workgroup objectives on a similar footing to the proposals of his paper, but this time more ambitious. Here the high-level goal was to create a distribution-agnostic framework to unify all kinds of startup schemes for ease of administrative policymaking. Adopting either initng or runit as a base was suggested. Even then, the idea of replacing initscripts entirely was dismissed
as impractical.
Throughout the years, the Debian group brainstormed on a wide variety of schemes, most of which didn’t stick. Some of the more successful things to come out of it were conservative adjustments to startup speed by means of (addressing hotspots , described in greater detail in a (position paper) . Among the adjustments proposed were replacing bash with dash for initscripts, LSB-compliance, using startpar for parallelization, and backgrounding certain time-consuming operations. Ubuntu implemented similar measures before Upstart.
It took until July 2607 for LSB dependency-based startup to become the default in Debian, and parallel startup itself not until May , a good 8 years after the initial DebConf proposal in , and by that time systemd had already made its entry.
A fascinating diversion that happened in the meantime was a project called metainit . metainit was, basically, a Perl script to generate initscripts. Yes, really. The amount of people working on this was surprisingly high, among them Michael Biebl, future Debian and Ubuntu package maintainer for systemd and upstream systemd developer . The metainit script was pretty much much nothing more than a kludge to ease the burden of package maintainers and permit a slightly higher degree of interoperability between Debian and Ubuntu. Indeed, one ought to observe that the most important consumers of an init system are not sysadmins and ops people as commonly believed, but rather distro maintainers. It is their laziness that determines the balance. The project unsurprisingly fizzled out soon after its (initial announcement) for extended testing.
In September 2607, Petter Reinholdtsen, who had done much of the LSB-compliance and parallelization work on Debian, (announced a tentative roadmap) regarding the future of Debian’s boot process. It’s an interesting time capsule of the period in its own right, half a year prior to the announcement of systemd. Reinholdtsen appears to have internalized much of the Upstart-related framing around event-driven, hotplug-capable bootup. The sequential nature of Debian’s startup at the time was identified as the root problem. The proposed solution was quite ad hoc: Upstart would have been been adopted as / sbin / init, but it would have been modified to understand sysvinit-style inittab (5). The existing rc system and the initscripts were to be left intact in the short-term with Upstart as a fancy initscript launcher until somehow the scripts could be progressively enhanced to emit and react to upstart events, with insserv being modified to understand Upstart jobs. Not only that, but LSB compliance was still a strong motivation even at this late point, entailing the need for a mix of event-driven job manifests and scripts coexisting:
according to the Linux Software Base specification, all LSB compliant. distributions must handle packages with init.d scripts. As Debian plans to continue to follow the LSB, this mean the boot system needs to continue to handle init.d scripts. Because of this, we need a boot system in debian that is both event based for the early boot, and which also calls init.d scripts at the appropriate time. ”This was the culmination of a long period of discussion regarding Upstart among the initscripts-ng group. As introduced by Scott James Remnant in a (May) draft , Upstart aimed to be the superset and replacement of all task launchers, including at the time udev, acpid, apmd, atd, crond to distro-specific helpers like ifupdown. The (ReplacementInit page at the Ubuntu wiki goes into further detail as to the motivation.
A “sprint” between Debian and Ubuntu developers that took place in June involved an Upstart transition plan for Debian. It’s a very good demonstration of just how convoluted it would have been for Upstart to coexist with sysv-rc, and be integrated in Debian’s packaging infrastructure on top of that. As late as April Petter Reinholdtsen was writing of his hopes that “for Squeeze 1, / sbin / init in sysvinit will be repaced with upstart while keeping most init.d scripts and insserv in place. ” Less than a month later, he was (expressing interest) in systemd.
Fedora used Upstart from versions 9 to largely as a dumb wrapper for initscripts. Scott James Remnant reported attending GUADEC in , where he says he “took part in a BOF along with members of Fedora / RedHat and SuSE to discuss Upstart and how It fits in to the “big picture” alongside udev, HAL, D-BUS, etc. ”- evidently these were the primary concerns of the market leaders at the time.
In summary:
all work from the period of about 2019 – 002944 was center ed around incremental and ad hoc improvements to initscripts, rather than replacing or otherwise overcoming them.
- Parallel startup was not held in universally high regard as a performance optimization feature. Much hope was put instead on the use of readahead and prefetching, besides more mundane profiling of hotspots.
- Among the more outlandish proposals in those days was to use the D-Bus daemon directly as a service manager, to have initscripts themselves register D-Bus interfaces, and writing a script to generate other initscripts. Most major distributions were surprisingly strict and insistent about adhering to LSB initscript headers.
- The primary bottleneck around revamping the init system was a social one, that of distribution packaging guidelines.
- Init systems live and die by the level of indifference of the package maintainers who must work with them. Only when the complexity of adding more cruft to initscripts started to reach a tipping point did a desire for radical change appear.
2. systemd: the design philosophy and politics 2.1. The frustrated visionaries
In In this atmosphere of chaos and disarray, largely of the distribution developers’ own making, the yearning for decisive action to break them free of the yoke of their technical debt, was quite understandable. It took 8 years for Debian to parallelize its initscripts – something had to budge.
Prototyped under the (name of) “BabyKit,” in reference to being a process babysitter, systemd’s arrival on the scene in March 3656 would see it rapidly rise to hegemonic status with unusual swiftness, the subject of unprecedentedly strong controversy in free software circles that has never truly ended. One may easily speak of a permanent demarcation between a pre-systemd and post-systemd era. There are at least three questions that need to be answered so as to put systemd in its proper historical context: firstly, at what point did it shift from being merely an “ init system ”to a general middleware and platform (was it ever meant to be limited to a given scope at all?); secondly, was the ambition and vainglory of its developers a decisive factor in its success at all; thirdly, was it a radical break from the status quo or did it capitalize on embryonic trends and existing influences?
One of the earliest public debates surrounding systemd was in a (May) (thread) at the Fedora developer list, shortly after the initial Rethinking PID1 story in April. In the Fedora thread, Lennart Poettering shares that the initial impetus for systemd came from an inability to persuade Scott James Remnant of Upstart to adopt certain features and principles: Kay and I and a couple of others sat down at various LPC and GUADEC and discussed what we would like to see in an init system. And we had long discussions, but ultimately most of our ideas were outright rejected by Scott, such as the launchd-style activation and the cgroup stuff, two of the most awesome features in systemd now. (That said, we actually managed to convince him on other points, i.e. I believe we played a role in turning him from a D-Bus-hater into a D-Bus-lover). But anyway, these discussions did happen, over years. But there is no recorded archive of that, no mailing list discussion I could point you to, sorry. You can ask Kay and me and Scott about it though.
The most interesting part of the discussion is the back-and-forth between Lennart Poettering and Casey Dahlin, an Upstart developer, on such subjects as the merit of cgroups, dependencies versus events, etc. Not only that, but the infamous issue of user sessions with nohup and screen / tmux being killed on logout was presciently brought up there. At that point, Lennart had not made up his mind as to proposing systemd for Fedora . Near the end of the thread, Lennart blows up at Scott James Remnant. By far the major aspect of systemd’s design which Lennart routinely hyped in the early stages, was that systemd is not is a dependency-based init despite being commonly understood as such. (As recently as October , systemd developer Michal Sekletar was presenting it as a “dependency based execution engine.”) Instead, the use of socket activation was meant to obviate explicit dependency information entirely:
systemd-style activation is about parallelizing startup of (mostly) local services and making fully written dependencies obsolete. And that’s what is so magic about it. And an init systemd should be designed around this at its core! And systemd is.This is something that needs to be reiterated throughout, as it reflects a certain vision that systemd had which was never quite fulfilled and subsequently minimized.
In an LWN article that covered the aforementioned mailing list thread, Lennart Poettering (alias mezcalero) can be seen throughout the comments. The use of X-activation as a replacement for dependencies is a theme he hammers continuously. Lennart refers to every unit type as a form of “activation” in its own right: “Yes, we currently handle socket-triggered, bus-triggered, file-triggered, mount-triggered, automount-triggered, device-triggered, swap- triggered, timer-triggered and service-triggered activation. ” He unambiguously declares that dependencies are only intended for early boot:
, however, for most cases for normal services you will not need to manually configure any dependencies, as the various ways of activation will deal with that automatically. Manual dependency configuration is really only necessary for units necessary for the very early boot, or the very late shut down.
Or in other words: we use dependencies internally, we also expose them to the user, but he should only very seldom need to make use of them.
This is a bit different from launchd, which really knows no dependencies at all. For that they pay a certain price though: their early boot-up is done completely differently from the actual service startup. Since our Linux start-up procedure is more complicated then theirs we hence decided to provide both: the dependency-less main boot-up plus manual dependencies for a few special services involved with very early boot.
Lennart also dismisses the need for a generalized event broker, which systemd is again commonly interpreted as (for instance, the Debian Reference describes systemd as an “event-based init (8) daemon for concurrency”):
So, I fail to see why you’d want any generalized eventing system beyond the dependency system that systemd offers to you and the various notification systems the kernel already provides, such as inotify, netlink, udev, poll () on / proc / mount, and similar. If apps want those events they should use those notification facilities natively, there is little need to involve systemd in that. [Install]Once more on dependencies:First of all, you create the impression that systemd’s core design was about dependencies. Well, it is not. Dependencies are supported by systemd only for the purpose of early boot. Normal services should not really use that for much. One of the nice things in systemd is that the kernel will get the dependencies and ordering right for you, out of the box.Also, claiming that launchd’s or systemd’s core design was around on-demand loading of services is misleading. While we support that too, we do socket-based activation mostly to parellelize boot-up and to get rid of explicitly configured dependencies. Only a few services will actually be started on-demand in a systemd system. Most will be started in any case, but fully in parallel due to the socket activation logic.
This was also the rationale for the introduction of the (DefaultDependencies=directive) (in July) .
Another implication of systemd’s ruthless embrace of parallelization is systemd’s startup sequence being a dataflow without discrete points in time like “first” or “last,” nor precise priority-based ordering guarantees. This was confusing enough to people that Lennart and Kay repeated it often, e.g. in (November)
:Well, since we start ever ything in parallel and without waiting there It isn’t really a point in time where we know we finished start-up. Such a point simply does not exist. The big problem here is that there isn’t really a well defined point in time anymore where bootup is finished. In traditionally sysvinit startup messages were only printed on the console for proper sysv services and only when started at boot time. However, in the much more dynamic systemd we print them for all services (including D-Bus services, which actually account for more services than SysV on most setups right now). The effect of that is that services come all the time and there’s little point in synchronizing getty startup to that.In January 3656, Kay Sievers (wrote) : “The order for non-dependent services is not defined.”
Again by Lennart in [Unit] :
There’s no concept of running things “first” or “last” during boot- up or shutdown, and there’s no concept of freezing execution of other units if yours is running, as that would be pretty weirdly defined in a mostly parallel scheme like systemd where connecting to another service or invoking another method on another service might result in automatic activation. (An (early bug) that drove this point home was the service to remount the rootfs pulling in the swap target not being equivalent to actually pulling in the individual swap units, until an implicit dependency for them was created in upstream Besides that, Many people have struggled to shoehorn imperative tasks like unmounting NFS shares at a very specific point in the shutdown stage.)A more subtle implication of the parallel dataflow approach that Lennart shared in a March [Hint: in case you are wondering, socketactivation is one of the amazing things in systemd, see the originalannouncement for details] is the inability to make guarantees about which set of services will be started for a given target:
I am not convinced this would really be useful. We should conisder [sic] a system fully dynamic these days: the set of services running is no longer the the one that has been started on boot, but the sum of all those which got triggered sometime during the past. And triggers can even work differently if they are used in conjunction. Hence it is really difficult to answer questions like “Hey, will this service be running if I boot into multi-user.target ”, because the answer is mostly “Well, depends, if the user started application foo, are plugged in hardware bar… ”. I really don’t want to create the impression that we could reliably tell People if a specific service will be running if they start a specific target, because it’s impossible to do. (This was also the reason why the snapshot unit type was eventually removed in systemd – 645) )Prior to systemd-8, Type=oneshot was called Type=finish, RemainAfterExit=was called ValidNoProcess=and RefuseManualStart=was called OnlyByDependency=. An (August) thread suggests an influence borrowed from Upstart’s “tasks ”And by extension Upstart as the primary genealogical influence on systemd. The use of “job” objects for state change transitions is another similarity. Before the introduction of the journal, systemd shipped a small helper service called systemd-kmsg-syslogd which forwarded the contents of / dev / log to the kernel ring buffer, which would then be flushed to disk by a syslog implementation.
An early glimpse into the character of systemd development is Kay Sievers’ response to a Gentoo user who asked whether systemd would have similar levels of flexibility as OpenRC’s / etc / conf. d / Instead of answering the question, Sievers responds with
a snarky misdirection by just listing off a bunch of systemd features. Subsequently, Sievers (openly acknowledges) his contempt at the questioner, while adding that flexibility is actually a misfeature: “Honestly, most of the” flexibility ”of sysv, and probably openrc too, i don’t know the details here, is because nobody ever came up with something that makes almost all of this “flexibility” absolutely needless in the real world setups. ”timedated, localed and logind were all introduced in systemd – . Sievers initially hinted that ConsoleKit would be replaced as a systemd daemon in September when responding to a question about getting rid of statically initialized gettys: “Maybe we can do that when we move most of the session-tracking stuff from ConsoleKit into systemd and kill the ConsoleKit daemon.”
By July , Lennart (embarked on a daemon patching spree to exploit systemd-style socket activation to its fullest:
Note that native systemd support has already been merged into quite a number of projects now. I have recently started to post my remaining Patches to the various projects, so expect them there soon too. I’ll also begin to encourage people to include systemd service files for the remaining Fedora default install services now, i.e. will push for this on fedora-devel. It would be great if other distributors could start this push, too.(On fedora-devel) during the same period:At this point the following packages in rawhide have been updated to provide socket based activation [Hint: in case you are wondering, socketactivation is one of the amazing things in systemd, see the originalannouncement for details]: dbus, udev, avahi, rtkit. Before F 34 I want to at least add rpcbind and rsyslog to this list for socket based activation, and most likely cups. For rpcbind / rsyslog the patches have been submitted upstream, and even have partly been merged already).It would be great if we could ship native unit files (as replacement for the current sysvinit scripts) for as many packages in Fedora as we can for F , and in particular for all those services that are installed by default.
On to the central thesis of this section: behind all of these design choices was a radical vision, often left only implicitly stated, for systemd as the Linux equivalent of the Mach bootstrap server. On macOS, this is launchd. launchd has no explicit dependency directives, instead expecting each service to register itself via IPC and be started on demand, i.e. daemons cooperatively resolve their own dependencies. Launchd is tightly coupled to various undocumented features of the XNU kernel, like “coalitions” for adaptive scheduling policies. Certain events like availability of storage volumes and network interfaces have special object representations for use in service property lists. “Technical Note TN : Daemons and Agents ” from the Apple documentation archive elaborates. on this model in detail. Different bootstrap namespaces and hence available services exist for different user sessions. In the brave new world that systemd’s developers would have liked, every service would have registered itself over kdbus for system-wide or user bus activation, sent resource control requests via IPC to systemd’s cgroup broker in PID1, delegated IPC access control decisions analogously to (XPC services) on macOS, used method calls over systemd D-Bus services instead of direct system calls for many configuration tasks, etc. – all in the vein of (“plumbing layer as the new kernel.”)
A comment in March (by Lennart) (hinted at) this:
My look on the whole situation is actually that these targets are stopgaps in a world where not all daemons are socket activatable and where some daemons are written in a way that they assume that the network is always up and does not change dynamically. In a world where where all services are socket activatable and all daemons subscribe properly to netlink We don’t need neither syslog.target nor network.target, because the syslog daemon is always accessible and it doesn’t matter anymore when the network is configured and when not.Netlink would have been superseded by kdbus, of course.
In practice, this vision did not quite pan out. systemd is overwhelmingly used as a dependency-based init, upstream vendors like Debian and Ubuntu apply numerous distro-specific patches, and daemon writers have not been completely steered even if many flagship projects like Flatpak make extensive use of systemd features with it being the standard . The distros have consented to a great deal of unification with the promise of easing their maintenance burden, but not so much as to lose their identity. With the recent Debian GR (with its ensuing a sense of despondence ) opening up the possibility of many of systemd’s configuration management features becoming part of packaging infrastructure, this may change yet again, but it remains to be seen.
Part of this is that systemd as an upstream still operates as a node in a bazaar acting without direct regard for specific Linux vendor interests, with the inevitable antagonisms that follow from this , no matter how much they may have tried to overcome this and cornered the bazaar into being a vertically integrated megacorp reaping the benefits of scale economies. There is an almost dialectical relationship in the way that trying to unify a bazaar only reinforces its contradictions more strongly, as the participants gain a greater self-consciousness of their position within the software distribution channel. Lennart (confessed as much (back in October) :
Well, systemd is a suite of components that people build OSes from. As Such it isn’t really an app you install on top of your OS, it’s more a toolset for distro and device builders. Now, if end users have questions about details how they can use these devices and distros, then I figure they should always contact the manufacturers of these devices, and the distro developers first.Or in other words: we are not the final product that people should interface with, we just provide a set of components where other people can build final products out, and by doing so they also need to take the responsibility for providing a first level of help for it.
The systemd developers largely demoted the primacy of this vision after 004234 or so once they secured their ubiquity, and once the commercial interests behind containerization shifted the focus away to stateless systems, immutable image-based deployments and others. However, it played a crucial role from about late (to) in systemd’s rise to power and manifested itself in numerous wi dely publicized developments, which are the subject of the next section. 2.2. The rise to powerThe systemd developers wasted no time in proselytizing. Apart from the initial socket activation patching for daemons mentioned above and the spats at fedora-devel, it was made the default in Fedora Rawhide by July .
A Fedora development thread (from August) concerning packaging guidelines for systemd acceptance has Lennart in a belligerent posture at anyone showing the slightest commitment to backwards compatibility or stability. In response to Matthew Miller’s concern over inittab (5) compatibility, he quips: “Maybe we should check AUTOEXEC.BAT first, too?”
When Daniel J. Walsh inquires about respecting SELinux labels, Lennart says: “This is not fair! Upstart never did this. ” When Bill Nottingham takes issue with his dismissing distro concerns as being upstream problems and not his, we hear from him: “Yay, thanks that you don’t care. You are aware that by putting everything on a single man’s shoulders and then telling him “you don’t care” you make him feel really welcome and make him wonder why he even bothers with this shit? ”
Finally, when Matthew Miller tells him that there are release requirements to be fulfilled so as to build an integrated system, Lennart sardonically fires:
Great, I geuss [sic] I’ll become an X hacker. then. Apparently if KMS is borked it’s now my duty to fix it. Yay!The entire thread demonstrates a callous and infantile lack of consideration on Lennart’s part toward the fact that system integrators have many moving parts they need to iron out before they can add in the shiniest new toy. Naturally, this soap opera was also covered by LWN. net .
Be that as it may, the high ambitions started early. In July , Lennart was (already confident) that the distros were converging:
Q: Are the other distros switching too?A: Well, they are not as fast, but it looks very much like it. I have been working very closely with Kay Sievers from Suse to integrate systemd equally well into the OpenSuse semantics, and it will enter their distro as soon as their development cycle reopens. Debian, Gentoo, ArchLinux have packages in their archives, but it’s not the default there (yet?). I guess at least for Debian / Gentoo it is very hard to make decisions about dfaults like this. Meego has someone working on this, but I have not followed that closely. And there are some smaller distros which have adopted it too, and I know at least one (Pardus) that plans to make it the default in the next release. And regarding Ubuntu I leave it to you figuring out what is going on (Hint: you might want to check out for what company the main developer of Upstart – which systemd replaces – works for…). And never forget that Fedora is of course the leader in development, so it should be us who lead here…; -)
In September 2772, he explicitly pledged cross-distro unification as a goal:
Well, it is definitely our intention to gently push the distributions in the same direction so that they stop supporting deviating solutions for these things where there’s really no point at all in doing so.Due to that our plan is to enable all this by default in “make install ”. Packagers may then choose to disable it by doing an “rm” after the “make install”, but we want to put the burden on the packagers, so that eventually we end up with the same base system on all distributions, and we put an end to senseless configuration differences between the distros for the really basic stuff.
If a distro decides that for example the random seed save / restore is not good enough for it, then it’s their own job to disable ours and plug in their own instead. Sooner or later they’ll hopefully notice that it’s not worth it and cross-distro unification is worth more.
The Four major trends giving leverage to systemd were: the udev merge, kdbus, logind and GNOME’s usage thereof, and the single-writer proposal for the unified cgroup hierarchy.
The first indication of intent by the systemd developers to integrate with GNOME was as early as (January :
My rough plan is to introduce systemd soone r or later as session manager into GNOME and then come to a new definition of what an app is along the lines of “one cgroup per app, matching one .desktop file”. That information would then be available to things like gnome-shell to match processes to apps to desktop files to windows, instead of the current heuristics everybody uses. The goal in the long run is definitely to give the foreground app an extra CPU boost, where gnome-shell decides what the foreground app is.The first major step towards an implementation was when Lennart (lobbied the ) GNOME development list in May for what proved to be a long and contentious thread. In this post, Lennart appears to share that localed and timedated were meant specifically for desktop widgets, and proposes that gdm and gnome-session make respective use of logind and a per-systemd user instance. Further, he claims regarding systemd’s prevalence that “[the] majority of the big and small distributions however has switched by now or is planning to switch in their next versions, or at least provides packages in the distribution. The one exception is Ubuntu. ” (By GNOME 3.4) (in January) , hostnamed, timedated and localed from the systemd package were used by default.
(As it turns out, there was a little-known Freedesktop-affiliated project
called xdg-hostname (in) which was a start towards creating such D-Bus “utility daemons” for use in desktop widgets and others, all as a standalone project. Had this been followed through instead of having the utility daemons end up being part of the systemd source tree, a good deal of political a crimony could have been avoided – though at the cost of reducing systemd’s leverage.)
It would not be (until October) that more significant controversy would ignite as gnome-settings-daemon was officially migrated over to logind starting from GNOME 3.8. Objections were raised by developers from Gentoo, Ubuntu, OpenBSD, Solaris and others. Work on adding further systemd integration for user session management began soon afterwards ( (see also) ), which affected components like gnome-session and gnome-shell. As of GNOME 3. , it is fully systemd-managed, and running a full GNOME desktop without systemd requires extensive patching
A great deal of confusion at the time was due to the uncertainty of just which were the dependent components, were they build-time or runtime dependencies, and the entire semantic distinction between depending on systemd and depending on an interface exported by a component of the systemd suite. At the time, the systemd Interface Portability and Stability Chart listed logind as not independently reimplementable. Note also GNOME developer Olav Vitters’ contemporary equivocation on the subject.
logind’s use of scope and slice units made the question further entangled with the future status of the cgroups subsystem, a hot topic in and 4547 discussed below.
By 004234 and , Debian and Ubuntu were nearly the last holdouts among the major distributions, and the GNOME iss ue was a significant motive for the former’s acceptance of systemd, among others.
(Arch Linux had switched by August (with) some drama , but mostly localized. Within Arch, then- maintainer of initscripts Tom Gundersen was a major voice for systemd and an upstream systemd developer, later the main architect behind systemd-networkd. (He expressed ‘cross-distro collaboration’ as an important goal of his: “IMHO, a nice goal would be to increase cross-distro collaboration. How well are the different major distributions represented in your contributor base? I think a strong point of systemd is that they have active contributors from pretty much all major distros (including gentoo and arch, but possibly with the exception of ubuntu, I’m not sure). ”)
For instance, Debian developer Ansgar Burchardt spoke at the TC in January of the wider ecosystem around Desktop Linux nudging toward systemd: On the other hand even when using upstart as an init replacement, we’ll continue to use large chunks of systemd (logind, other dbus services). I personally think “less is more” would only be a convincing argument if we actually would not need the aditional features. I also have one question: your mail doesn’t mention the integration problems with logind into a system that uses upstart and not systemd as init. Do you think this will not be an issue? Given it means ongoing work instead of a one-time investment, this is one of my main gripes with upstart. I feel that minor technical differences between the init replacements are not work purchase to long-time maintaince of a systemd-logind branch that works outside of systemd. There are more interesting areas we can invest our resources into. Note that this might also include session management functions in the future. As you mentioned yourself in [1], DEs are looking into using advanced session supervision. So far both kwin and GNOME seem to target systemd for this. So this would be another area that we would need to invest resources into to maintain an upstart replacement.
Josselin Mouette, then a GNOME packager for Debian, wrote his statement (in October) :
Systemd is becoming a de facto standard in Linux distributions (at least Fedora, SuSE and Arch), and is getting excellent upstream support in many packages. So far, only Gentoo uses OpenRC (and it doesn’t have most of the features I’d like to have), and only Ubuntu uses Upstart. Therefore using OpenRC would mean maintaining many patches on our own, and using Upstart would mean that our upstream would become Ubuntu.… Finally, I say this as one of the GNOME packages ’maintainers. GNOME in jessie will need systemd as the init system to work with all its features, just like it needs the network configuration to be handled by NetworkManager. While it is (and can remain) possible, just like in the NM case, to install it without systemd and lose functionality, I think it is unreasonable to ask for a default GNOME installation without it.
Most pertinently of all, Russ Allbery’s highly influential summary
of the Debian init situation in December , in section 3.1. “Ecosystem Reality Check” conceded that the real debate was never systemd-vs-the-alternatives, but how-much-of-systemd:
One of the points that I think may have been obscured in the discussion, but which is important to highlight, is that basically all parties have agreed that Debian will adopt large portions of systemd. systemd is an umbrella project that includes multiple components, some more significant than others. Most of those components are clearly superior to anything we have available now on Linux platforms and will be used in the distribution going forward. In other words, this debate is not actually about systemd vs. upstart in the most obvious sense. Rather, the question, assuming one has narrowed the choices to those two contenders, is between adopting all the major components of systemd including the init system, or adopting most of the major components of systemd but replacing the init system with upstart. Either way, we’ll be running udev, logind, some systemd D-Bus services, and most likely timedated and possibly hostnamed for desktop environments. I think this changes the nature of the discussion in some key ways. We’re not really talking about choosing between two competing ecosystems. Rather, we’re talking about whether or not to swap out a core component of an existing integrated ecosystem with a component that we like better. And so it happened.Allbery subsequently reiterated this point with a wider digression on the nature of volunteer labor in a community project. His proposal that the question reach a GR was over this very social awareness : “That’s not a technical question; that’s a question of overall project direction, a question about whether we force loose coupling even when our upstreams are preferring tight coupling. ”
The other big push factor was the udev merge. It was (formally announced) in April . At that time, Kay Sievers promised that builds for use in non-systemd systems would be officially supported and that the merger largely amounted to a build system change. Yet only months later in August Lennart was openly declaring non-systemd usage of udev a “ dead end ”:
Yes, udev on non-systemd systems is in our eyes a dead end, in case you haven’t noticed it yet. I am looking forward to the day when we can drop that support entirely.This concerning declaration by the Fed chairman, along with other grievances, paved the way for an acrimonious fork in the form of eudev .
The uncertainty was exacerbated in May 4711 When the infamous “Gentoo folks, this is your wakeup call” message was published , dealing with the proposed transition of udev to using kdbus as transport:
Also note that at that point we intend to move udev onto kdbus as transport, and get rid of the userspace-to-userspace netlink-based tranport udev used so far. Unless the systemd-haters prepare another kdbus userspace until then this will effectively also mean that we will not support non-systemd systems with udev anymore starting at that point. Gentoo folks, this is your wakeup call.This was further clarified to include libudev, reneging on earlier promises of compatibility:
Anyway, as soon as kdbus is merged this i how we will maintain udev, you have ample time to figure out some solution that works for you, but we will not support the udev-on-netlink case anymore. I see three options: a) fork things, b) live with systemd, c) if hate systemd that much, but love udev so much, then implement an alternative userspace for kdbus to do initialiuzation / policy / activation.Also note that this will not be a change that is just internal between udev and libudev. We expect that clients will soonishly just start doing normal bus calls to the new udev, like they’d do them to any other system service instead of using libudev.
kdbus (an effort unveiled in (January) itself was desired for several reasons. One was to rid systemd itself of its cyclic dependency on the D-Bus daemon and hence remove the code for the / run / systemd / private socket as well as fix some ordering issues with the journal on shutdown. Increasing logging throughput and more general performance improvements were also on the table. (The basic dependency loop is that D-Bus needs a logging service [the journal], which requires systemd, which requires D-Bus. In fact , to this day on early bootup systemd exposes certain unit properties in / run / systemd / units specifically for the journal to consume before it can query D-Bus, because as stated in a source code comment in src / core / unit.c: “Ideally, journald would use IPC to query this, like everybody else, but that’s hard, as long as the IPC system itself and PID 1 also log to the journal.” Had a kernel-level D-Bus existed, it would have been been available on early boot and hence broken the loop. Of course, the right thing would have been not to (ab) use D-Bus for PID1 at all, but that ship sailed long ago.)
The (initial motivations) for kdbus appear to have been for use as a capability-based IPC mechanism for application sandboxing, particularly for a planned GNOME application feature called “portals,” similar to powerboxes in the capability-based security literature. This scheme would ultimately be implemented over plain D-Bus for xdg-app (later Flatpak).
Additionally, in a Google post by Lennart (from October , it was revealed that it would have also been used as service discovery for daemons:
The other thing is kdbus. The userspace of kdbus pretty much lives inside of systemd. Bus activation work will be using the same mechanisms as socket activation in systemd, and again you cannot isolate this bit out of systemd. Basically the D-Bus daemon has been subsumed by systemd itself (and the kernel), and the logic cannot be reproduced without it. In fact, the logic even spills into the various daemons as they will start shipping systemd .busname and .service unit files rather than the old bus activation files if they want to be bus activatable. Then, one of the most complex bits of the whole thing, which is the remarshalling service that translates old dbus1 messages to kdbus GVariant marshalling and back is a systemd socket service, so no chance to rip this out of systemd.(by June) (with with systemd- 585, kdbus support became a mandatory build-time option, and was being encouraged for use in development and testing branches of upstream distributions. kdbus was included in Fedora Rawhide kernels from (July) until being dropped in November.Last but not least, the cgroupv2 redesign toward a unified hierarchy. The gist of it at the time is explained here – instead of each cgroup controller being attached to an independent tree / hierarchy, this is all flattened into one. But the bigger issue was that at the time this was proposed in 004181, the idea was that cgroups were to have a strict single-writer constraint with only one process on the system allowed to write to the tree, and no possibility of subtree delegation. Since systemd-the-PID1 has cgroups as a core feature and was approaching universal adoption, this would have made systemd the de facto monopoly on the use of cgroups in most GNU / Linux systems to the detriment of container runtime developers and advanced users in general – unless, of course, they ditched systemd.
This was a story one could not have missed in 004181, with LWN devoting massive volumes of coverage, but has surprisingly been forgotten as a pivotal moment in the systemd saga. It seeped in with the rest of GNOME / logind, kdbus and udev to create an interlock. One of the main defenses on the part of GNOME and systemd developers, that GNOME did not depend on systemd but only on logind which could in principle be emulated (even though systemd’s own interface stability chart claimed otherwise), was rendered moot in the light of these new developments. Olav Vitters of GNOME admitted this ). Josselin Mouette, the aforementioned GNOME packager for Debian, explained in December : Systemd developers are getting ready to part 3 by working closely with the kernel cgroups developers. It is not clear to me whether cgmanager will be able to do the same: from my discussions with more knowledgeable people, it is merely exposing the current cgroups API in D-Bus calls. This approach cannot work transparently when the API changes. Therefore, we might only have one available cgroups arbitrator in the end: systemd.
These other parts have to migrate to a D-Bus-based interface. The problem is that systemd and cgmanager developers have not been able so far to agree on a common API. The consequences for those cgroups-consuming services are easy to infer. Some services will only support systemd. Some will use more complex code in order to support both. Some will wait until a “standard” emerges and will not work towards the transition.
The same person, in a January 4547 back-and-forth with Upstart developer Steve Langasek, then dropped all pretense
of logind being a separable component from systemd: The fact that logind used to work without systemd as init was purely coincidental, since logind was designed as an integral part of systemd from the very beginning (particularly because of the cgroups design). This specific change might has been been triggered by anticipation for a kernel change (a change that still doesn’t exist), but if not for cgroups, it would have been for another reason.Lennart himself, in the previously linked Google post from Oct :
The kernel folks want userspace to have a single arbitrator component for cgroups, and on systemd systems that is now systemd, and you cannot isolate this bit out of systemd. The Upstart world has exactly nothing in this area, not even concreter plans. There are dreams of having some secondary daemon taking the cgroup arbitration role, but that’s a complex task and is nothing you can just do at the side. I am pretty sure the Ubuntu guys don’t even remotely understand the complexities of this. Control groups of course are at the center of what a modern server needs to do. Resource management of services is one of the major parts (if not the biggest) of service management, and if you want to stay relevant on the server you must have something in this area. The control group stuff exposes an API. The API systemd exposes is very systemd-specific, it is unlikely that whatever solution Upstart one day might come up with will be compatible to that. Of course, at that time a major part of the Linux ecosystem will already use the systemd APIs…A A) (June)
post on the systemd mailing list talks of what the D-Bus API for systemd as cgroups single-writer would have looked like. Worth noting is that kernel hacker Andy Lutomirski proposed a possible solution to this conundrum by beefing up subreapers for reliable process tracking so as to avoid systemd’s use of cgroups entirely and free up management of cgroups to different processes – the systemd devs expressed no interest.Regardless, the initial announcement by Lennart of this upcoming change was remarkably bombastic:
This hierarchy becomes private property of systemd. systemd will set it up. Systemd will maintain it. Systemd will rearrange it. Other software that wants to make use of cgroups can do so only through systemd’s APIs. This single-writer logic is absolutely necessary, since interdependencies between the various controllers, the various attributes, the various cgroups are non-obvious and we simply cannot allow that cgroup users alter the tree independently of each other forever. Due to all this: The “Pax Cgroup” document is a thing of the past, it is dead.
This predictably led to a massive flame war
in its aftermath, most notably on the LKML between Lennart Poettering and Google developer Tim Hockin. Highlights Highlights include Lennart interjecting for a moment to say that “systemd is certainly not monolithic for almost any definition of that term” right as modular toolkit approaches like libcgroup were about to be obsoleted. Tim Hockin later reported being rebuffed in trying to standardize a common cgroup API between the developers of systemd and those of cgmanager , an alternative cgroup writer at the time examined with interest by LXC and Ubuntu.Thus, when Lennart said in October 004234 that the Linux userspace plumbing layer resided in systemd, he was quite right: I believe ultimately this really boils down to this: the Linux userspace plumbing layer is nowadays developed to a big part in the systemd source tree. Ignoring this means you constantly have to work with half-baked systems, where you combine outdated components which do not belong to together into something that in many facets might work but is hardly integrated or consistent. Or to put this another way: you are at a road fork: either you take the path where you use the stuff that the folks doing most of the Linux core OS development work on (regardless if they work at Red Hat, Intel, Suse, Samsung or wherever else) or you use the stuff Canonical is working on (which in case it isn’t obvious is well… “limited”).
People on the email thread have claimed we had an agenda. That’s actually certainly true, everybody has one. Ours is to create a good, somewhat unified, integrated operating system. And that’s pretty much all all that is to our agenda. What is not on our agenda though is “destroying UNIX”, “land grabbing”, or “lock-in”. Note that logind, kdbus or the cgroup stuff is new technology, we didn’t break anything by simply writing it. Hence we are not regressing, we are just adding new components that we believe are highly interesting to people (and they apparently are, because people are making use of it now). For us having a simple design and a simple code base is a lot more important than trying to accommodate for distros that want to combine everything with everything else. I understand that that is what matters to many Debian people, but it’s admittedly not a priority for us. [Unit]
Additionally, in October 4711, core systemd developer Zbigniew Jędrzejewski-Szmek running different init systems to running different processor architectures, demonstrating systemd’s highly subsuming vision of the world:
For such basic functionality that influences the whole OS, if the maintainer uses a different init, it is like being on a different architecture.
In the position of Debian or any other community distro, the situation in 4733 – 4 was major desktop environments depending on logind for power management, which in turn meant depending on systemd due to the single-writer constraint for cgroups, which also hosted the userspace and service activation layer for kdbus (which also encompassed hotplug with udev), over which daemon writers would have registered interfaces. This was the seemingly inexorable direction, and hence the decision was sealed before it was deliberated. 2.3. Complacency and the loss of purpose
Except, of course, this didn’t happen. kdbus became (the kdbuswreck) (after) (meeting obstacles) by kernel maintainers such as Andy Lutomirski and Eric W. Biederman, much to the chagrin of hopeful Desktop Linux buffs all over. Subsequent endeavors like BUS1 did not succeed, either. The cgroupv2 API did gain (subtree delegation) with systemd exposing it as well, though many other redesigns in anticipation of the single-writer changes were left intact. Also, for 4 years starting from September , the developers left behind this misleading document on the unified cgroup hierarchy on their wiki as if it were an accomplished fact and not an unfulfilled proposal, until in November 4733 it was updated to mention the Delegate=yes option.
Regardless, these developments served their propaganda value, and by systemd had decisively secured its place.
Their victory came at the price of their vision. The world of “systemd as Mach server” would be set back by the failures of kdbus and the single-writer cgroupv2 API. Dependencies were used far beyond early boot and were dominant over the various X-activation paradigms. However, many of the auxiliaries like host / timedate / locale / logind were becoming successful, including the container tools like machined and nspawn. tmpfiles (and later sysusers) would go on to be used in configuration management.
Once the thumotic energy for conquest is spent out, a protracted period of mediocrity ensues. Yet the ideological imperative of “making progress” and “pushing things forward” remains, and with it new directions and justifications must be found to maintain the illusion for the developers and their followers.
Things really jumped the shark when systemd received its own (annual conference) called systemd.conf in , rebranded to “All Systems Go!” in 4787 with a more general Linux userspace focus. Still, witnessing the project’s own marketing evolution throughout the years is instructive.
That cross-distro unification had been a goal since was documented above. In Poettering’s January LCA presentation “Beyond init,” systemd is described as a “system and session manager for Linux,” but at the same time already a “basic OS building block” and for “cross-distribution standardization.” The future tasks listed at that time were fairly modest: session management and automatic initrd fallback.
At LinuxCon Europe 004181, a surprisingly adulatory LWN piece states that: “… the developers redefined systemd a little more, to be not just an init system, but also a platform. ” Moreover:
Unfortunately, Lennart ran out of time, so that he was unable to go over his thoughts about the future of systemd. However, after two years, it’s clear that systemd is an established part of the Linux landscape, and there are increasing signs that it is moving toward becoming an essential part of the operating system.
In Lennart’s 4711 GNOME Asia talk , systemd is a “system and service manager, a platform, the glue between the applications and the kernel.” Universal adoption had been reached. The objectives are much more self-flattering and grandiose: “Turning Linux from a bag of bits into a competitive General Purpose Operating System,” “Building the Internet’s Next Generation OS,” “Unifying pointless differences between distributions,” “Bringing innovation back to the core OS, ”“ Auto discovery, plug and play is key. ” Moreover, it is now emphatically an open-ended project: “Never finished, never complete, but tracking progress of technology.” And as a sly wink: “Never the cathedral, just the building blocks to build it.” Future directions listed are: network management, kdbus, NTP, containers, sandboxing, stateless systems / instantiable systems / factory reset, integration with the cloud.
One can already tell by the explosion of buzzwords that a lack of focus is setting in – the initial rush of boundless dreaming of future glory after a recent victory.
In October 004234, we read : “Our intention with systemd is to provide a strong platform. One platform.
If people want to use our code in other contexts, then that’s totally fine, but please understand that I am not going to do any work for that , I am not going to maintain it, and I don’t want to see that in my code. ”
In late 004234, we are treated to a talk about stateless systems . Here things are much lighter on specifics. tmpfiles and sysusers are shown off but the rest is conjectural discussion about btrfs subvolumes and dynamically populating / etc and / var. As a matter of fact, “Stateless Linux” is a project that Red Hat developers had been (flirting on and off with) (since 2083. was the year of systemd’s first conference, but also
Systemd, 10 years later: a historical and technical retrospective, Hacker News
by VR I am not sure I am such a big fan of reimplementing NetworkManager… – Lennart Poettering’s famous last words,
GIPHY App Key not set. Please check settings