“How Linux Became My Job”: Extended Cut, Geeks Edition

Recently the opensource.com editors made an open call for people to submit their own “open source story.”

I thought it would be a fun trip down memory lane, so I put a draft together and submitted it. Thanks to the great editors at opensource.com, that story is now published and live: “How Linux Became My Job.” To keep my story from being a rambling mess of overly technical details, I had to leave out a lot of the intricacies, and thanks to the awesome editing from the opensource.com team, I was very happy with the clean and readable finished article.

But sometimes these extra details are the fun tidbits for us geeks to devour, and sometimes we are OK drowning in the minor (and likely unimportant!) details. Given the main story is now published, I thought it might be fun to add this “geeks extended cut” here on my own blog with a few of these details.

OS/2, the PS/2 and Boca Raton, FL

I started my IBM career in Boca Raton, Florida, which happens to be the birthplace of the IBM PC. While manufacturing had long moved away from Boca, the old manufacturing buildings had been repurposed into office space, and it was in that set of buildings that I worked with hundreds of other IBMers (and a few Microsoft contractors, if you can believe it) on OS/2 for my first few years at IBM. We were testing our operating system on a myriad of PS/2 models filling large raised floor test labs. 20MB hard drives and the new 486 processor were the exciting new developments in those days!

This was before laptops, so if you wanted a “home terminal” for checking email or doing work at home, IBM had the P70 with an amber screen, diskette drive, and 386 processor that I had in my apartment for awhile, dialing up via modem to IBM! Original list price for the P70? Just under $5000 dollars!

I ended up not working directly on OS/2 for very long, as a new team was formed to work with Apple on OpenDoc, an OLE competitor that married some Apple technology with IBM’s object oriented library, SOM (System Object Model), and it’s OO RPC (CORBA-based) associate, DSOM. It was my time on OpenDoc (though short-lived given Steve Jobs killed OpenDoc on his return to Apple) where I really started learning what it meant to be a developer. I started to really enjoy debugging, and I was asked to spend a considerable amount of time debugging and fixing problems in the storage layer (Bento) of OpenDoc that was extremely buggy at the time. Let’s just say I found out all too well how painful manual reference counting that controls memory allocation/deallocation can be!

I also started to experiment with small software projects “just for fun”, and decided it would be interesting to write a “chat” tool on OS/2 that would use DSOM to share the messages (as objects) with any participating client that would join the DSOM server. It was horribly buggy because DSOM was unstable and buggy, but I had a father-in-law, brother-in-law, and my wife all at IBM sites at the time and we were able to use it for simple communication throughout the day. In hindsight that was my opportunity to beat AIM to the punch and create my own startup! Given this was Florida and I had an interest in weather, I also wrote a hurricane tracking map application using publicly available NOAA weather data.

IBM’s Java Technology Centre (JTC)

IBM joined up as a full Sun Java source code licensee in the JDK1.0 timeframe. A “JTC” organization was created, and a small group of us, now located in Austin, Texas, were asked to be an extension of the JTC (located in Hursley, England) to handle the porting of the JVM’s GUI implementation to OS/2 presentation manager. This work would enable the AWT (“advanced windowing toolkit”) classes in Java for OS/2, using the official Java sources Windows GUI implementation as a starting point.

At this point I had became a Linux user and hoarded enough hardware to have a Linux box in my office at IBM. It was an “exciting” time to use Linux inside IBM, as the network technology was still token ring based, not Ethernet. The token ring driver for Linux was under constant development to keep up with specific PCI card revisions, models, and ring implementations. Any distribution updates usually meant making sure the ibmtr or olympic drivers were available for a new kernel revision, and you had to build it manually and, if necessary, fix any issues so you could be on the network!

During these JVM porting days, my manager and I convinced IBM that I needed real Sun hardware to do “testing” on the official Java platform. So, for several years I had a top of the line Sun UltraSparc system with a Sun Creator 3D graphics card and beautiful display. This system was running Solaris of course. When I started the porting effort for Java for Linux, IBM also provided me with an IBM Power workstation (I think a 43P model) running AIX, as we had already ported the JVM to AIX as a “reference” for my porting work. My office was effectively the United Nations of UNIX and Linux systems!

One of my first forays into dealing with open source, licensing, and distribution happened during the porting of the JVM to Linux. On AIX and Solaris, the Motif library was used in the AWT graphical implementation, and for these licensed proprietary operating systems, the distributor of the OS had a license from the Open Group to ship Motif libraries. Linux distributions had openmotif but there were questions about whether it was properly licensed and distributable. We had to work with The Open Group and IBM legal to understand if we were approved as a licensee to distribute a shared Linux library of Motif inside our Linux JDK installation package. Thankfully people smarter than me worked out the details with the Open Group, and many years later, Motif was finally released under the LGPL.

Embedded Linux & Linux Distros

My time in IBM’s Linux Technology Center was focused on distribution partnerships (SUSE and Red Hat in those days; expanding to include Canonical/Ubuntu years later) including enabling Linux for all our hardware platforms. This included some interesting devices that were not standard x86 architectures. One of the key devices needing proper Linux support were our service processors for IBM POWER and z Systems high end servers. There were also other embedded devices which had been using proprietary OS components for many years prior to this Linux initiative.

The fun part of this job is that in the early 2000s many upstream Linux projects hadn’t enabled build support for embedded/non-x86 processors. At the time we were supporting MIPS, ppc4xx, and ARM as well as POWER and z (s390x) systems. Even if a project’s Makefile build system or the autoconf or autotools support was correct, sometimes the provided RPM spec file didn’t properly handle other architectures by default. So, my job was to review (and fix as necessary) hundreds of Linux distribution packages: enabling cross-compiler tool support, helping the autotools setup “do the right thing” regarding “guesses” for a cross-build environment, and anything else that needed proper enabling for non-x86 builds. Today much of this multi-platform enablement is cleaner in upstream open source projects, but at that point it was brand new and support was very spotty. Along with this manual work of package by package fixups, I also became an expert in building gcc for various build/host/target cross-tools combinations. While it wasn’t rocket science, it was fun to have that knowledge of something that had seemed like a black art at the time! These were enjoyable years working closely on these tasks with a great team of people, including Josh Boyer, who was the ppc4xx maintainer in the Linux kernel at the time, and later when on to be an active leader in the Fedora project, working at Red Hat.

Containers, Containers, Containers

Fast forward to my current work, and clearly I have been focused on Docker and the Moby project, the Open Container Initiative (OCI), and the CNCF containerd project. But when I stepped into that work back in August 2014 I had no idea how much it would become a part of my future work and career path.

I had some fun interactions with all kinds of developers in those early days of #docker-dev on IRC. I was a total newbie and was focused on trying to learn this new area of container technology. Looking back, it was an amazing cast of characters from Jess Frazelle, to Vincent Batts, Alexandr Morozov, Michael Crosby, and many others. But one interaction stands out above the others. At the time I had no idea who this “kelseyhightower” was who chatted in the IRC channel now and then. One day he starting asking about IPv6 support in Docker and how to get the ball rolling on figuring out a design and what work would be needed to make it happen. I had seen a patch come in from a contributor who had then disappeared and I was slowly trying to get that code ready for a PR. Here’s a log of the actual discussion that happened that morning in IRC:

So, I had a chat about IPv6 with Kelsey Hightower long before I ever knew who he was and before I had ever met him (and maybe before many other people knew who he was)! But, looking back those were some amazing days of collaboration and camaraderie that are nearly impossible to ever be recreated, but I know many of us have great memories from those times and enjoy reminiscing when we see each other around the conference circuit. They were hectic days; so many releases, hundreds of PRs a week, new issues appearing every few minutes it seemed like, 24/7! It was a huge time of learning for me, but many of my prior experiences with Linux, with software development, and with debugging all came in handy to make it a very successful time for me as well.

Summary

So that’s my “Extended Cut, Geeks Version” of How Linux Became My Job! Hopefully a few of you may connect with one or more of these random details of my time in software development over the past two decades. Feel free to share your most interesting tidbit about your time in tech in the comments below!