AUSTIN (May 24, 2022) — The Adélie Linux distribution is preparing to publish its third release candidate, RC3. This long-awaited milestone is a major turning point in the project’s trajectory. Adélie is on track to target supercomputers after the 1.0 general release.
We’ve reached several important technical milestones including significant software updates, security fixes, and infrastructure-related matters. These changes will make it possible to reach 1.0 with confidence, the focus of which is stability.
RC3 includes Linux 5.15 and a smoother installation experience. Other highlights include Python 3.10, LLVM 13, Perl 5.34, musl 1.2.3, and updated desktop environments.
Rust and Java will be fully up-to-date, and we’re working to improve their bootstrap processes. Not only that, but software updates and fixes will be rolled out regularly again.
RC3 will be officially announced in the coming weeks. This means in-place upgrades for our existing users, and new installation media for new users.
But wait, there’s more.
Did you know that we now include Spack, a package manager for supercomputers, giving you access to thousands of scientific software packages and bringing us closer to our goal of making Adélie into a musl-based platform for high-performance computing (HPC)?
Adélie will begin maintaining Spack patches for musl and big-endian POWER hardware. We encourage other projects to contribute to our fork as we intend to upstream relevant patches to Spack or the original upstream. Spack is not ready for production use on musl.
There’s only one small problem. We don’t have tens of millions of dollars or the expertise to set up such an environment, nor scientists who can test their codes on our platform.
So we’re excited to announce a new collaboration with the GW4 Alliance (a partnership between U.K.-based universities, Cray Inc., and the U.K. Met Office) for access to an ARM-based Cray XC50 production supercomputer called Isambard, clocking in at 20,992 cores.
This supercomputer includes Fujitsu A64FX, Marvell ThunderX2, and IBM POWER9 CPUs, in addition to NVIDIA V100 GPUs on top of a 100 Gbps Infiniband backbone.
Thank you to Prof. Simon McIntosh-Smith, leader of the Isambard project and Professor of High Performance Computing at the University of Bristol, for making this happen!
Behind the scenes, we’re in talks with multiple leading cloud and/or chip companies to explore other partnership opportunities for both hardware and technical support.
We hope that these will lead to interesting research opportunities, including opportunities to publish, receive grant money, and serve scientific communities around the world.
We have some financial news to share with you, too. We are in the process of consolidating some of our assets held at various donation platforms. Our financial standing with SPI is now public information with a 1-2 month delay. SPI better aligns with our needs and allows our developers to separate our personal lives from key areas of the project.
SolidRun donated a 16-core ARM board to our project for use as our primary ARM build server. This board has been installed in a 1U dual Mini-ITX server chassis alongside a SiFive HiFive Unmatched, on loan from Zach, for riscv64 porting efforts.
The riscv64 server has been patched, a year later, to finally enable remote reboots. It is now humming along at the data center building packages for our next Adélie port!
This means Adélie now fully owns, hosts, and does not rely on any third party for its build and service infrastructure (except for email, DNS, and ISPs).
To better serve our European users, we’ve added a 10 Gbit mirror in Prague, Czechia. It is synchronized every 6 hours. Thank you to NIC.CZ for sponsoring it.
We would like to thank our sponsors and supporters; without you, this project would not be possible. For example, a small donation might afford a box of USB sticks that we can distribute at local computer shops. Larger donations afford much-needed hardware.
As a side note, we firmly believe in reciprocity. So we’ve stepped up to host the first public Linux host running the Apple Silicon M1 chip. This aarch64 system (32-bit not supported) can be used for development, testing, and benchmarks. Kernel patches by Asahi Linux.
This, of course, foreshadows our own effort to bring Adélie to Apple Silicon computers.
For some background on how we got here, RC2 was published in October 2020. Additional updates were published between November 2020 and January 2021, but no packages had been built or published since then until January 2022, nearly one year later.
It was during this 2020-2021 period that A. Wilcox, the founder and former project lead, had stepped down to focus on financial stability and family matters. With it, we lost access to our POWER build server and main driving force behind our distribution.
Meanwhile, security vulnerabilities and technical debt continued to accrue. Our aarch64 build server, sponsored by (then) Packet.net, abruptly became unreachable soon after the RC2 release. Support inquiries went unanswered.
In May 2021, we reached out again for support. The next day, they informed us that our project was not “server” enough and that the box had been repurposed without warning, ability to recover data, or possibility of appeal.
Further discussion involved bad-faith arguments and went in circles. It became clear that only corporate-benefiting projects “where there are clear signals of customer demand” would be supported. Other FOSS projects seem to have been treated similarly.
For a company (now Equinix) of that stature, to offer top-notch hardware resources — to vetted FOSS projects — and then turn an about-face was a disappointing maneuver. The relationship is between the hardware donors and Packet.net, not Packet.net and projects.
Our aarch64 signing keys went the way of the dodo. How the armv7 keys were recovered won’t be detailed, suffice to say it involved scanning for and testing all key-like material in an old backup and processing the data with a chain of kludgy scripts.
These are potent lessons in managing risk and avoiding singular points of failure. Of comical coincidence, our x86_64 build server then crashed and could not be recovered.
We’re moving forward, though, and overcoming many challenges along the way.
Digital data and physical hardware are not the only assets that need to be transferred when a project changes hands. A third, and arguably the most important, is knowledge transfer.
While some knowledge transfer had taken place, three of our core contributors also moved on from the project. Asked how “distfiles” (our collective repository of package binaries and their sources) could be reproduced, the answer was “LOL”. Knowledge includes a project’s history, of utmost relevance to understanding intent, which guides decisions.
How does one set up a build server? Crickets. How do we keep up project morale? Blank stares. How to manage our existing services and infrastructure? It’s tribal knowledge and the person who knows is unavailable. Do we have any other build servers? Two virtual machines, maybe? Can we bootstrap? Not since years ago, and that was not documented.
To be fair to the team, maybe these are trivial matters. But it is not trivial to get everyone into a room to discuss it at once, nor is it trivial to figure out what one doesn’t yet know.
That is where we left off mid-2021. I cannot overstate how grateful I am for the work that our team has put in since our December 2021 update, because a new website and hardware won’t automatically build, test, fix, and publish software updates to our users.
This happens in our package repository. It’s what contributors want to, well, contribute to. My job as project lead is to grease the gears and make it as effortless as possible for contributors, old and new, to engage with the project — to donate knowledge and time!
We currently have two package repositories. “System” includes all software required to build everything in “User”, itself comprising the rest of our ecosystem (meaning desktop environments and everything else a user could want to install).
Until now, packages were not built in a deterministic order. They had been bumped, added, and/or removed as needed based on whatever the current state of the build server was in at the time. Which almost certainly differed between architectures.
We have developed tooling to ensure that, at any given commit in our package repository, packages are built in (topological) order of their dependencies. This tooling parses APKBUILD files and is able to handle subpackages reliably as well. It detects errors such as a “System” package depending on a “User” package, as well as cycles and more.
An experimental variant of this tooling enables distributed Makefile-based builds, significantly reducing the time it takes for us to completely rebuild the world.
There are still a few bugs to iron out, but this is how we learn and grow.
Additional new tooling sets up a complete package build and/or test environment, perfectly reproducible on any Linux-based machine. This environment is isolated from a host system and builds the entire world from source without intervention, finally reproducing “distfiles”.
The only visible difference between your computer and our build infrastructure is which key(s) you sign packages with. If a key is not provided, it will be generated locally.
In the way of security, we’re formalizing procedures for the management and handling of sensitive key materials. We’re implementing automatic and semi-automatic code and vulnerability scanning tools.
With package building infrastructure mostly out of the way, we can layer on nice-to-have features and services. As mentioned previously, we plan to offer a man(1) webpage.
In other news…
Our email is now hosted on an excellent third-party platform, Migadu, and it is no longer an opaque self-hosted system. Migadu receives high marks from us for their fast customer support, knowledgeable staff, robust service, and administrative features.
Our GitLab IRC bot is functioning again; she informs the developer channel of changes to our repositories, providing some positive reinforcement.
Our package listing is undergoing a major overhaul so that it is easier to maintain and use.
Looking forward, I’d like to extend a personal assurance that post-1.0 development will include a strong focus on accessibility and improved user experience. Síle Ekaterin Liszka will coordinate and lead this effort.
I’d like to thank A. Wilcox, who, despite officially logging out and working through ongoing personal matters, has been instrumental in unblocking tasks around the Adélie household.
Our future is bright. Thank you for being part of it.