apt(8) is dangerous

Published July 10 2019 by Alex Blackie

Alternatively, “How is dnf the only package manager that actually works??

For a while now I have done all my dayjob work within a VM over SSH and Samba. This has worked quite well, and meant I could keep my development environment a bit more slow-moving (generally, the latest Ubuntu LTS) while running a faster-moving desktop distribution on the hardware itself. It also means when I travel I can rsync the VM to my personal laptop and not have to lug two laptops with me everywhere just to keep that work/personal separation.

So. I tried to install updates. About halfway through installing updates, something I do a couple times a week usually, the VM froze. The SSH connection was responsive but I could tell all disk I/O was totally dead, and basically nothing worked -- not even closing a shell session. I opened the VM's console display and sure enough there was a screen full of watchdog warnings saying the Kernel had hung. I gave it a couple minutes but it seemed clear that something had gone horribly wrong.

I forced off the VM and rebooted, which thankfully worked fine. So then I tried to resume the upgrade.

apt, of course, was no help.

~ % sudo apt update
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic-security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.

As always, make sure to notice and laugh out loud when you see the proud http:// plastered across your screen, knowing that even if you added an s it would not work because Canonical do not provide secure package mirrors, and unless you installed some extension package (over plain HTTP of course) apt doesn't even support secure connections.

All packages are up to date. Um, well, that's entirely false. As soon as I saw this I knew I was in for a wild ride, and started thinking through how much work it would be to just rebuild the VM from scratch.

I lost my shell history for this part but I had to eventually find out to run some dpkg command to "resume" the in-flight upgrade:

dpkg --configure --some-other-flags-i-think

That did a bunch of stuff, seemingly finished the upgrade successfully.

“Well, let's just try again!”

~ % sudo apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: The package libgl1-mesa-dri needs to be reinstalled, but I can't find an archive for it.

There it is - upgrading this package is what hosed the system to begin with. Good to see there is at least a problem being logged somewhere.

I first tried apt reinstall which I always forget does not exist. I moved on to just trying to remove and install it myself…

~ % sudo apt remove libgl1-mesa-dri
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: The package libgl1-mesa-dri needs to be reinstalled, but I can't find an archive for it.

Yes… Thanks, I know… “Sorry, we can't remove this package because we can't find a locally cached tarball to upgrade it, and apparently aren't smart enough to figure out how to just redownload it from the repo"…

With my hope dwindling, I finally tried the old install -f, which works when you install a deb manually, which often requires you to temporarily break your system until you install -f. Installing a deb is a hilariously shitty process but that's not the topic for today's discussion.

~ % sudo apt install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: The package libgl1-mesa-dri needs to be reinstalled, but I can't find an archive for it.

It did not work.

My patience growing thin I finally resorted to copying and pasting the error into a search engine, which so far is the only benefit to running Ubuntu I can come up with.

StackOverflow, of course, delivered.

~ % sudo dpkg --remove --force-all libgl1-mesa-dri
 dpkg: libgl1-mesa-dri:amd64: dependency problems, but removing anyway as you requested:
 libglx-mesa0:amd64 depends on libgl1-mesa-dri.

 dpkg: warning: overriding problem because --force enabled:
 dpkg: warning: package is in a very bad inconsistent state; you should
  reinstall it before attempting a removal
  (Reading database ... 166745 files and directories currently installed.)
  Removing libgl1-mesa-dri:amd64 (18.2.8-0ubuntu0~18.04.2) ...

dpkg, of course, complains and moans about doing the literal one and only solution to fixing the problem that it caused in the first place. But does do as asked, and removes the broken package from the system.

Now I can finally just run apt install --fix-broken which installed some random other packages I guess that were required. At this point I am so tired of debugging this problem I do not care.

I can now run apt again, and running a subsequent apt upgrade was successful.

After all of this, I just cannot help but think about dnf and how none of this would have happened if the package manager in Debian was built well and fully integrated. This split-brain between apt and dpkg, with the user having to juggle a bunch of non-memorable flags and commands for both just to figure out what the hell is going on, let alone to repair a system. This also leads into one tool not understanding the state of the other, leading to the situation of this entire post.

I have experienced failed upgrades in Fedora! It has happened! But because dnf is good software, it gracefully knows a previous transaction was aborted and offers to resume it, or lets you roll it back safely, or at least lets you see what is going on with your system and what problems you might run into because of the failure.

But don't worry:

~ % apt -h
[...]

This APT has Super Cow Powers.

However, apparently "super cow powers" are not enough to safely install package updates.