How to fix a broken Mageia kernel upgrade

YESTERDAY I GOT HIT by the sort of inexcusable blunder that only very rarely afflicts Linux users. A maintenance upgrade to Mageia killed my desktop system.

By "very rarely" I mean that this is the first time I can recall something like this happening with a remote maintenance upgrade to a Linux system in over ten years, and I must have run hundreds of software maintenance updates to Linux distributions during that time.

It was a Linux kernel upgrade to Mageia 2, and when it was done my desktop wouldn't reboot, not even in Failsafe single user mode. It just hung with the sort of dire hints in the system log during boot that you never want to see - "runaway modprobe loop" and "kernel panic!" Ouch.

Since Mageia has recently become a very popular Linux distribution for users fleeing Mandriva, presently sitting in second place in page hits at Distrowatch, it has occurred to me that a fair number of other Linux users might have run into the same problem, or might be about to face it. So here I'll explain what happened and outline how to fix it.

I always keep a backup Linux system on my machines out of habit, because I'm not immune to screwing up and having to boot a working system in order to fix whatever I've managed to break. So I rebooted into my previous Linux system and set about figuring out how to fix this. Booting to a Live CD also would have worked just as well.

Since it was a boot problem, I looked at the /boot partition on the target system hard drive where boot was hanging, like this:

Open a command line terminal session window. That's in the Mageia Linux desktop menu under Tools, and it's called "Terminal" under Gnome or "Konsole" under KDE.

su # get into root mode (enter root password)

mount -t ext4 /dev/sda5 /disk1 # mount the /boot partition on the target system

ls -al /disk1 # list the dead system /boot directory

drwxr-xr-x 5 root root 1024 Aug 23 08:59 ./
drwxr-xr-x 28 root root 1024 Aug 23 08:59 ../
lrwxrwxrwx 1 root root 27 Aug 23 08:59 config -> config-3.3.6-desktop-2.mga2
-rw-r--r-- 1 root root 129528 May 17 09:26 config-3.3.6-desktop-2.mga2
-rw-r--r-- 1 root root 129550 Jul 30 14:37 config-3.3.8-desktop-2.mga2
drwxr-xr-x 2 root root 1024 May 20 05:19 dracut/
-rwxr-xr-x 1 root root 527360 Jul 1 10:58 gfxmenu*
drwxr-xr-x 2 root root 1024 Aug 23 08:59 grub/
-rw-r--r-- 1 root root 6754060 Jul 1 19:30 initrd-3.3.6-desktop-2.mga2.img
-rw-r--r-- 1 root root 6753758 Aug 23 08:59 initrd-3.3.8-desktop-2.mga2.img
lrwxrwxrwx 1 root root 31 Aug 23 08:59 initrd-desktop.img -> initrd-3.3.8-desktop-2.mga2.img
lrwxrwxrwx 1 root root 31 Aug 23 08:59 initrd.img -> initrd-3.3.8-desktop-2.mga2.img
drwx------ 2 root root 12288 Jun 18 10:48 lost+found/
-rw-r--r-- 1 root root 164504 Jan 30 2011 memtest.bin
-rw-r--r-- 1 root root 167656 May 17 09:26 symvers-3.3.6-desktop-2.mga2.xz
-rw-r--r-- 1 root root 167408 Jul 30 14:37 symvers-3.3.8-desktop-2.mga2.xz
lrwxrwxrwx 1 root root 31 Aug 23 08:59 System.map -> System.map-3.3.6-desktop-2.mga2
-rw-r--r-- 1 root root 2159033 May 17 09:26 System.map-3.3.6-desktop-2.mga2
-rw-r--r-- 1 root root 2159478 Jul 30 14:37 System.map-3.3.8-desktop-2.mga2
lrwxrwxrwx 1 root root 28 Aug 23 08:59 vmlinuz -> vmlinuz-3.3.8-desktop-2.mga2
-rw-r--r-- 1 root root 2716800 May 17 09:26 vmlinuz-3.3.6-desktop-2.mga2
-rw-r--r-- 1 root root 2725632 Jul 30 14:37 vmlinuz-3.3.8-desktop-2.mga2
lrwxrwxrwx 1 root root 28 Aug 23 08:59 vmlinuz-desktop -> vmlinuz-3.3.8-desktop-2.mga2

Well, duh! No wonder it didn't work. A now very embarrassed Mageia Linux kernel maintainer obviously miscoded the commands to update the config and System.map soft links to the new versions of those files and pointed to the old versions instead.

So all that needs to be done is change those soft links in the target system /boot partition and it should work. Here are the commands:

cd /disk1 # switch to the target directory

rm config # remove the existing soft link

ln -s config-3.3.8-desktop-2.mga2 config # define the correct soft link

rm System.map # remove the existing soft link

ln -s System.map-3.3.8-desktop-2.mga2 System.map # define the correct soft link

And that's it, you're done. I also restored /disk1/grub/menu.lst.old to /disk1/grub/menu.lst but you can fix that later, if needed.

Here's the boot partition as it looks after having been repaired:

drwxr-xr-x 5 root root 1024 Aug 23 10:01 ./
drwxr-xr-x 28 root root 1024 Aug 23 10:02 ../
lrwxrwxrwx 1 root root 27 Aug 23 10:01 config -> config-3.3.8-desktop-2.mga2
-rw-r--r-- 1 root root 129528 May 17 09:26 config-3.3.6-desktop-2.mga2
-rw-r--r-- 1 root root 129550 Jul 30 14:37 config-3.3.8-desktop-2.mga2
drwxr-xr-x 2 root root 1024 May 20 05:19 dracut/
-rwxr-xr-x 1 root root 527360 Jul 1 10:58 gfxmenu*
drwxr-xr-x 2 root root 1024 Aug 23 08:59 grub/
-rw-r--r-- 1 root root 6754060 Jul 1 19:30 initrd-3.3.6-desktop-2.mga2.img
-rw-r--r-- 1 root root 6753758 Aug 23 08:59 initrd-3.3.8-desktop-2.mga2.img
lrwxrwxrwx 1 root root 31 Aug 23 08:59 initrd-desktop.img -> initrd-3.3.8-desktop-2.mga2.img
lrwxrwxrwx 1 root root 31 Aug 23 08:59 initrd.img -> initrd-3.3.8-desktop-2.mga2.img
drwx------ 2 root root 12288 Jun 18 10:48 lost+found/
-rw-r--r-- 1 root root 164504 Jan 30 2011 memtest.bin
-rw-r--r-- 1 root root 167656 May 17 09:26 symvers-3.3.6-desktop-2.mga2.xz
-rw-r--r-- 1 root root 167408 Jul 30 14:37 symvers-3.3.8-desktop-2.mga2.xz
lrwxrwxrwx 1 root root 31 Aug 23 09:58 System.map -> System.map-3.3.8-desktop-2.mga2
-rw-r--r-- 1 root root 2159033 May 17 09:26 System.map-3.3.6-desktop-2.mga2
-rw-r--r-- 1 root root 2159478 Jul 30 14:37 System.map-3.3.8-desktop-2.mga2
lrwxrwxrwx 1 root root 28 Aug 23 08:59 vmlinuz -> vmlinuz-3.3.8-desktop-2.mga2
-rw-r--r-- 1 root root 2716800 May 17 09:26 vmlinuz-3.3.6-desktop-2.mga2
-rw-r--r-- 1 root root 2725632 Jul 30 14:37 vmlinuz-3.3.8-desktop-2.mga2
lrwxrwxrwx 1 root root 28 Aug 23 08:59 vmlinuz-desktop -> vmlinuz-3.3.8-desktop-2.mga2 

Sure enough, my desktop system booted right up after making these changes, rescuing me from possibly having to reinstall Mageia and recustomise it to recover from this maintenance error.

If you are a Linux user and found this helpful, please let us know in the comments below or via email and I might write some more such not too technical Linux How-to articles in the future. µ