Torvalds at LinuxCon Part I: The Highlights and the Lowlights

Much of what Linus Torvalds talks about in this excerpt from his onstage interview at LinuxCon 2016 will be recent memories for many of our readers. For others, it’ll be ancient history.

On Wednesday, when Linus Torvalds was interviewed as the opening keynote of the day at LinuxCon 2016, Linux was a day short of its 25th birthday. Interviewer Dirk Hohndel of VMware pointed out that in the famous announcement of the operating system posted by Torvalds 25 years earlier, he had said that the OS “wasn’t portable,” yet today it supports more hardware architectures than any other operating system. Torvalds also wrote, “it probably never will support anything other than AT-harddisks.”

“Because it is what I had,” Torvalds quickly quipped back, which was almost an exact quote of what he had written back in 1991. The line about AT hard disks had ended with the clause, “as that’s all I have.”

“You have to realize the background is that it was a completely personal project,” Torvalds explained. “I expected other people to be interested from a theoretical standpoint. Students of operating systems might want to look at ‘here’s another operating system we can look at.’ That was my expectation, which meant that the kind of hardware I had was the only hardware that it ran on.”

He’s always a good interview and gives the interviewer and the audience what they want. Although he limits his number of speaking engagements, it’s obvious that he likes sitting down for these Q&A sessions, which he often does with Hohndel. He’s relaxed and works from his head instead of from any prepared notes — thinking on his feet and speaking off-the-cuff, often speaking in run-on sentences or changing his approach mid-sentence, which requires a bit of editing to convert into written text.

“For the first two or three releases you not only had to have an AT hard disk,” he continued, “you had to have the same AT hard disk as I had because the geometry of the hard disk was hard coded in the source code. You could have a different AT hard disk, but you had to change the numbers to match how many sectors you had. So, the first releases were a bit unpolished.”

Hohndel’s interviewing style was mostly to prompt and then stay out of the way. “So what are the highlights, or the lowlights, of these 25 years?”

“The highlights are really hard to even mention because mostly it’s been really, really good,” was Torvald’s short answer.

“The moments where I went ‘this is awesome’ that I remember clearly have been when I’ve been struggling with something that just took forever, like I was struggling on the same thing for weeks and nothing worked. Most of those were actually very early on.

“The reason nobody sane does operating systems from scratch is that when nothing works it is really, really frustrating. But then even the slightest sign of life makes you go ‘wow, I really mastered this machine.’ When the first character shows up on the screen, and the system does nothing else, you’re really pumped because you actually brought a character on the screen. Completely useless, but those early times were actually some of the most memorable for me, when nothing really worked and you go from that nothing to something.

“Later on, and these days, I’m actually most happy when the process works. I don’t write code any more, so I don’t have any code I can be proud of because I send off emails to random people. I say, ‘Hey, this guy reports a bug. You’re the expert in this area, can you fix it?’ So I don’t have code I can be proud of for making things work. I can be proud of when the release process really works and people get things done and we don’t have a lot of issues. That makes me happy these days.”

“Lowlights?” Hohndel prompted.

“We’ve had some,” Torvalds answered. “Almost none of them have been about technology. We had difficult times when things really didn’t make progress.”

As might be expected, Torvald’s “lowlights” might not have been a direct result of technology, but they certainly revolved around technology.

“The developers in the audience may remember, if you’ve been around for long enough, power management was such a bummer for so many years,” he said. “We really struggled with that. It took years and years to get it to the point where you could just take a random laptop and suspend and resume it and just assume that it works. That was very frustrating. But on the whole, lowlights from the technical side have not been a big issue. You have frustrating issues that come up and are hard to debug, but most of the low lights have been really on the social, and especially, the development side when the process doesn’t work.

“If you go back fifteen years, we had huge process issues. During the 2.4 series we…I’ll take blame, I screwed up the memory management and we had this situation where during a few releases we completely switched around which memory manager…. I think Andrea might be here somewhere, he ended up fixing a lot of things and it was very, very, very painful. I still remember it as a lowlight and as ‘this is not how we’re supposed to do it.'”

Then there was BitKeeper, the then proprietary source code manager used by Linux beginning in 2002. Because it was proprietary, its use was always controversial, drawing criticism from Linux developers as well as from Richard Stallman. Things came to a head in 2005 after BitKeeper withdrew free commercial use of the project to key Linux developers, including Torvalds and Andrew Morton, which prompted Torvalds to write Git.

“Everybody has heard about how BitKeeper was a big failure and how that brouhaha went when I had to go off and make Git work,” Torvalds said. “But BitKeeper actually was a big savior for me personally because the situation we were in from a process standpoint before BitKeeper was such a disaster.

“People may not realize, because a lot of Linux people have come in fairly lately, but fifteen years ago there was a fair amount of commercial interest, but it was still pretty early and our processes at the time were geared towards the precommercial interest models that we had when we had a lot of individuals at universities and our kernel was much smaller. We had maybe ten to 50 developers, then all these big commercial companies come in and now we have hundreds of developers and lots of new code, lots of problems we didn’t have to face before, and our process was just a mess. It was very painful.”

Those years during the first decade of the new century were evidently much more stressful on those doing the work of developing and maintaining Linux than any of us on the outside could imagine.

“That was probably the only time in the history of Linux where I was like ‘this is not working’ and I felt like…. In retrospect I look back and say that might have been the moment where I just gave up. But that was fifteen years ago. We had problems after that, but for the last few years the release schedule and everything has been so smooth and we have thousands of people involved now and it’s working.”

Read Part II of our coverage of Linus Torvalds’ keynote interview at LinuxCon 2016.