Page MenuHomePureOS Tracker

Cryptsetup and initramfs issues after upgrade via Gnome Software
Open, HighPublic

Description

This is a placeholder to compile information regarding several cases with initramfs and cryptsetup that we have seen in a short period of time. Usually after the user does an upgrade with Gnome software.

All 3 cases so far share the same story: Users cannot unlock LUKS after a recent update, and are dropped to recovery shell.

They can only unlock LUKS and login via advanced options in GRUB with an older kernel.

All 3 cases were reported in less than two weeks

Case 1

Machine fails unlock encrypted partition, after failed attempts machine drops to initramfs.
User was asked to try to boot in graphic mode with an older kernel via advanced options in GRUB and run the following commands:

  • sudo dpkg --configure -a
  • sudo apt update && sudo apt full-upgrade
  • sudo update-initramfs

dpkg --configure and full-upgrade revealed no errors

sudo update-initramfs revealed the following error:

update-initramfs: Generating /boot/initrd.img-4.19.0-5-amd64
cryptsetup: WARNING: Resume target luks-1a9af0c3-db46-4947-8b83-5ad57975ad15 
    uses a key file

These are the user crypttab and fstab files

/etc/crypttab: mappings for encrypted partitions.


 <name>               <device>                         <password> <options>
luks-991a8170-d20f-48dc-b858-2ffb86d8dada UUID=991a8170-d20f-48dc-b858-2ffb86d8dada     /crypto_keyfile.bin luks,keyscript=/bin/cat
luks-1a9af0c3-db46-4947-8b83-5ad57975ad15 /dev/disk/by-id/wwn-0x500a0751e1d8a59f-part3  /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256
/etc/fstab: static file system information.

 <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=b64b86f5-1c85-4761-bb38-0007cf00925f /boot          ext4    defaults,noatime,discard 0 2
/dev/mapper/luks-991a8170-d20f-48dc-b858-2ffb86d8dada /              ext4    defaults,noatime,discard 0 1
/dev/mapper/luks-1a9af0c3-db46-4947-8b83-5ad57975ad15 swap           swap    defaults,noatime,discard 0 2
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0 0

Case 2:

Machine fails unlock encrypted partition, after failed attempts machine drops to initramfs
User was asked to try to boot in graphic mode with an older kernel via advanced options in GRUB and run the following commands:

  • sudo dpkg --configure -a
  • sudo apt update && sudo apt full-upgrade
  • sudo update-initramfs

dpkg --configure and `full-upgrade revealed no errors

sudo update-initramfs revealed the following error:

These are the user crypttab and fstab files

/etc/fstab: static file system information.

 <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=7e307419-1fb1-42c2-b2c7-959957a77790 /boot          ext4    defaults,noatime,discard 0 2
/dev/mapper/luks-24cee226-f119-4e37-8b2c-67de537e66fe /              ext4    defaults,noatime,discard 0 1
/dev/mapper/luks-16eaa33b-133f-49be-8624-24bc7d2e767a swap           swap    defaults,noatime,discard 0 2
tmpfs                                     /tmp           tmpfs   defaults,noatime,mode=1777 0 0
# /etc/crypttab: mappings for encrypted partitions.

 <name>               <device>                         <password> <options>
luks-24cee226-f119-4e37-8b2c-67de537e66fe UUID=24cee226-f119-4e37-8b2c-67de537e66fe     /crypto_keyfile.bin luks,keyscript=/bin/cat
luks-16eaa33b-133f-49be-8624-24bc7d2e767a /dev/disk/by-id/wwn-0x5002538e4052606c-part3  /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256

Case 3:

Machine fails unlock encrypted partition, after failed attempts machine drops to initramfs
Status: Still waiting on information from the user


Why create this issue:

Although these problems reported here are probably a bug that needs investigating.

We have some users that are affected by this and waiting for a reply from us regarding this issue. And we need some input about options to recommend to the affected users to restore their systems other than just say: reinstall the system from scratch.


This is our current wiki page regarding fixing broken system upgrade


@jonas.smedegaard already suggested that:

this all looks like an issue with PureOS - particularly with how the installer creates cryptsetup (and possible related to LUKS v2 not working well with GRUB: https://cryptsetup-team.pages.debian.net/cryptsetup/encrypted-boot.html ).

Event Timeline

joao.azevedo updated the task description. (Show Details)Jun 13 2019, 06:43
joao.azevedo added a subscriber: jonas.smedegaard.
jeremiah.foster triaged this task as High priority.Jun 14 2019, 13:05
jeremiah.foster added a subscriber: jeremiah.foster.
joao.azevedo updated the task description. (Show Details)Jun 16 2019, 13:00
joao.azevedo updated the task description. (Show Details)
Wayne added a subscriber: Wayne.Jun 17 2019, 08:15

Can we assume that today's update (6/17) for the cryptsetup packages to 2.1.0-5 does not address this issue? I have marked all those as 'hold' for now.

I don't know for sure. I'm taking a look at them right now and have downloaded and updated here. I'll report my experience with the updates.

In the first case described here, updating grub (sudo update-grub) had no effect. Problem remains

I've not experienced any issues with recent updates, including update to cryptsetup packages.

Can we check to see if

GRUB_ENABLE_CRYPTODISK=y

is enabled in /etc/default/grub/. If that is enabled and not commented out the system asks for a password via the shell.

mladen added a subscriber: mladen.EditedJul 28 2019, 11:32

A user recently upgraded via Software Center and had this issue. He can successfully boot *4.19.0-2* but the newer kernel (4.19.0-5):

@mladen Can we ask the user to do an apt-get update && apt-get upgrade? Otherwise we need to get more information.

mladen added a comment.Jul 29 2019, 15:48

@jeremiah.foster

The user upgraded via sudo apt update && sudo apt full-upgrade and also did sudo update-initramfs -u -k all and here's the output:

sudo update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-4.19.0-5-amd64
cryptsetup: WARNING: Resume target
luks-36631c43-3d73-428b-ac97-b518d1838185
    uses a key file
update-initramfs: Generating /boot/initrd.img-4.19.0-2-amd64
cryptsetup: WARNING: Resume target
luks-36631c43-3d73-428b-ac97-b518d1838185
    uses a key file

Hasn't fixed the issue.

@mladen @jeremiah.foster

cryptsetup: WARNING: Resume target
luks-36631c43-3d73-428b-ac97-b518d1838185
uses a key file

That is the same error that the first users in this thread had, in their case from reading their crypttab and fstab files, it corresponds to the swap partition

@mladen @jeremiah.foster

cryptsetup: WARNING: Resume target
luks-36631c43-3d73-428b-ac97-b518d1838185
uses a key file

That is the same error that the first users in this thread had, in their case from reading their crypttab and fstab files, it corresponds to the swap partition

mladen added a subscriber: mak.Jul 30 2019, 15:10

@jeremiah.foster @mak not that update-initramfs did not fixed issues, but it made it worse: customer now cannot boot even the older kernel version.

We need to determine where the issue is. Can we get a copy of /etc/crypttab? Also, can the customer boot into recovery mode? Is there any other information on the situation like log messages or output from 'journalctl -xe'?

In the cases where the /etc/crypttab is pasted (in the original post in this issue) I can see that both crypttabs have /

 <name>               <device>                         <password> <options>
luks-991a8170-d20f-48dc-b858-2ffb86d8dada UUID=991a8170-d20f-48dc-b858-2ffb86d8dada     /crypto_keyfile.bin luks,keyscript=/bin/cat

Can those users try with;

# <name>               <device>                         <password> <options>
luks-991a8170-d20f-48dc-b858-2ffb86d8dada UUID=991a8170-d20f-48dc-b858-2ffb86d8dada  /etc/cryptsetup-initramfs/cryptkey.gpg luks,keyscript=decrypt_gnupg-sc

or

# <name>               <device>                         <password> <options>
luks-991a8170-d20f-48dc-b858-2ffb86d8dada UUID=991a8170-d20f-48dc-b858-2ffb86d8dada    none luks

Note

What is happening here is that we're changing the options section in the /etc/crypttab file to move away from "keyscript=/bin/cat" to either a decrypt_gnupg-sc script or to none.

jjakob added a subscriber: jjakob.EditedAug 10 2019, 04:25

I think this is relevant: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=930696
IMO a keyscript is unnecessary.
I ran into it when migrating my unencrypted PureOS to a new disk and adding LUKS to it at the same time (I first tried a fresh install but the installer failed for which I already opened a separate bug). I wrote up a quick guide on how to do that on Pureos/Debian, by the way: https://github.com/jjakob/wiki/wiki/Migrating-an-unencrypted-PureOS-Debian-install-to-fully-encrypted
Also getting rid of the unencrypted /boot partition would largely improve the security, I would highly recommend it as the default for the encrypted install. There's no need to enter the passphrase twice if a keyfile is used, and it works flawlessly.

This is my working crypttab:

# <name>               <device>                         <password> <options>
crypto-nvme0n1p1 PARTUUID=de043f60-01 /etc/cryptkeys/nvme0n1p1.key luks

/etc/cryptsetup-initramfs/conf-hook has:
KEYFILE_PATTERN="/etc/cryptkeys/*.key"

initramfs.conf has:
UMASK=0077

/etc/cryptkeys is mode 700, the keyfile is mode 400. Created with dd bs=512 count=4 if=/dev/random of=/etc/cryptkeys/sdX#.key iflag=fullblock.

@joao.azevedo please note that bundling multiple issues makes them all harder to track - e.g. I would be uncomfortable assigning the issue report to myself if knowledgable about only a subset of involved issues, would only flag it as done when all parts were done, and would have a hard time noticing when all parts were done.

I recommend to file one report per issue. Issues are easy to merge if multiple issues turn out to be the same.