dnf, where art thou?

Published July 10 2019 by Alex Blackie · Updated November 4 2019

For a while now I have done all my dayjob work within a VM over SSH and Samba. This has worked quite well, and meant I could keep my development environment a bit more slow-moving (generally, the latest Ubuntu LTS) while running a faster-moving desktop distribution on the hardware itself. It also means when I travel I can rsync the VM to my personal laptop and not have to lug two laptops with me everywhere just to keep that work/personal separation.

So. I tried to install updates. About halfway through installing updates, something I do a couple times a week usually, the VM froze. The SSH connection was responsive but I could tell all disk I/O was totally dead, and basically nothing worked -- not even closing a shell session. I opened the VM's console display and sure enough there was a screen full of watchdog warnings saying the Kernel had hung. I gave it a couple minutes but it seemed clear that something had gone horribly wrong.

As it turns out, the virtual disk image I had copied from another computer was in VirtualBox format, and something in the driver in QEMU/KVM would cause it to completely stall periodically. Of course, this failure occurred at the worst time -- in the middle of installing updates -- and so this is my tale of trying to repair the VM once I rebooted it.


~ % sudo apt update
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

As always, make sure to glare disapprovingly at the http:// plastered across your screen, knowing that even if you added an s it would not work because no one seems to provide secure Debian package mirrors, and unless you installed some extension package (over plain HTTP of course) apt doesn't even support secure connections.

All packages are up to date. Um, well, that's entirely false. As soon as I saw this I knew I was in for a wild ride, and started thinking through how much work it would be to just rebuild the VM from scratch.

I lost my shell history for this part but I had to eventually find out to run some dpkg command to "resume" the in-flight upgrade:

dpkg --configure --some-other-flags-i-think

That did a bunch of stuff, seemingly finished the upgrade successfully.

“Well, let's just try again!”

~ % sudo apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: The package libgl1-mesa-dri needs to be reinstalled, but I can't find an archive for it.

At least we're getting an error again.

I first tried apt reinstall which I always forget does not exist. I moved on to just trying to remove and install it myself…

~ % sudo apt remove libgl1-mesa-dri
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: The package libgl1-mesa-dri needs to be reinstalled, but I can't find an archive for it.

Yes… Thanks, I know… “Sorry, we can't remove this package because we can't find a locally cached tarball to upgrade it, and apparently can't just redownload it from the repo”…

With my hope dwindling, I finally tried the old install -f, which works when you install a deb manually, which often requires you to temporarily break your system until you install -f. Installing a deb is an unfortunate process (that is slowly getting better with apt replacing apt-get) but that's not the topic for today's discussion.

~ % sudo apt install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: The package libgl1-mesa-dri needs to be reinstalled, but I can't find an archive for it.

Nope.

My patience growing thin I finally resorted to copying and pasting the error into a search engine. StackOverflow, of course, delivered.

~ % sudo dpkg --remove --force-all libgl1-mesa-dri
 dpkg: libgl1-mesa-dri:amd64: dependency problems, but removing anyway as you requested:
 libglx-mesa0:amd64 depends on libgl1-mesa-dri.

 dpkg: warning: overriding problem because --force enabled:
 dpkg: warning: package is in a very bad inconsistent state; you should
  reinstall it before attempting a removal
  (Reading database ... 166745 files and directories currently installed.)
  Removing libgl1-mesa-dri:amd64 (18.2.8-0ubuntu0~18.04.2) ...

dpkg, of course, complains, but does do as asked, and removes the broken package from the system. Finally.

Now I can finally just run apt install --fix-broken which installed some various packages, their importance at this point not top of mind. At this point I am so tired of trying to fix the system I do not care.

I can now run apt again, and running a subsequent apt upgrade was successful.


After all of this, I just cannot help but think about dnf. I have experienced failed upgrades in Fedora! It has happened! But because dnf gracefully knows a previous transaction was aborted it has facilities to resolve the situation, either through rolling back, retrying, or at the very least just letting you see up front what the state of your system is, without having to search for some arcane flags to a piece of software you never interact with (dpkg).

The split-brain between apt and dpkg makes for a bit of a confusing mess, especially when the state gets out of sync between them, as was just chronicled in this post.

The focus on this in recent years with apt is a glimmer of hope that this situation will improve in the Debian ecosystem. I work with Ubuntu systems regularly, and I always pine for dnf or yum most of the time. I sincerely hope Debian can catch up to where dnf has pulled ahead, and help one of the most widely-deployed operating systems in the world a little more stable and trustworthy.