| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-09 | |||
| 09:58:16 | stephenfin | I don't know how to do the equivalent in Horizon | |
| 09:58:22 | jrosser | ooooohhhh - well thats explains it! | |
| 09:59:24 | jrosser | perhaps that warrants a docs fix here https://docs.openstack.org/nova/latest/user/rescue.html | |
| 09:59:47 | stephenfin | Yes, good point | |
| 09:59:58 | stephenfin | lyarwood: Maybe if you have time in the next few weeks? ^ | |
| 10:11:14 | jawad_axd | d.Any comments on this?Thanks | |
| 10:11:14 | jawad_axd | Hi folks, how luks password is generated for instance booted from encrypted volume? Barbican is used in environment. I can grab luks password like “virsh secret-get-value SECRET-UUID | —base64 —decode” when instance is running on compute node. How can I generate this password on my own? Use case is: if instance is backed up outside somewhere and needs to boot standalone, thus needs luks passwor | |
| 10:15:06 | jrosser | jawad_axd: there was a thread on the mailing list about this recently http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020374.html | |
| 10:17:29 | jawad_axd | @jrosser I missed that one. I look into it. Thanks | |
| 10:21:53 | lyarwood | stephenfin: sorry was afk, reading | |
| 10:23:30 | lyarwood | huh how did I miss that | |
| 10:32:02 | lyarwood | stephenfin: https://review.opendev.org/c/openstack/nova/+/779479 | |
| 10:32:17 | lyarwood | jawad_axd: hey sorry reading | |
| 10:33:20 | lyarwood | jawad_axd: yeah that post on the ML spells out how to grab the passphrase from the key manager, it's awful I know but thanks to design choices made before my time :/ | |
| 10:34:22 | lyarwood | jawad_axd: http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020421.html specifically lists the steps | |
| 10:42:00 | jawad_axd | @lyarwood Thanks. Apparently, it requires volume to to be mapped. I am wondering if there is an easy way to decrypt/decode it from barbican cli, when encrypted volume exists in environment and is in in AVAILABLE state.a | |
| 10:44:35 | lyarwood | jawad_axd: No, you can't stream/decode an unmapped encrypted volume. | |
| 10:45:03 | lyarwood | jawad_axd: unlike Glance, Cinder doesn't send actual volume data via the API | |
| 10:45:28 | lyarwood | jawad_axd: so you will need to map the encrypted volume to a host before being able to decrypt it | |
| 10:46:27 | lyarwood | jawad_axd: as the owner of the volume you could snapshot it and download the encrypted volume snapshot from glance I guess | |
| 10:46:44 | lyarwood | jawad_axd: that would give you a local file you could then decrypt but it's not the live volume | |
| 10:47:11 | gibi | stephenfin: question in https://review.opendev.org/c/openstack/nova/+/779449/1/setup.cfg#11 | |
| 10:57:00 | jawad_axd | @lyarwood Ok.. if I understand correctly, luks header is attached to 'rbd disk in my case' when we create instance from encrypted volume, is that right ? My understanding is, when we create encrypted volume from image, it attaches luks header to volume_from_image that time, if its true then it should be possible to retrieve luks password when it is not mapped. | |
| 10:59:32 | lyarwood | jawad_axd: the header is always attached but the actual passphrase is stored outside of that in the key manager and doesn't require access to the volume or image | |
| 11:01:00 | lyarwood | jawad_axd: to be clear, fetching the passphrase and decrypting data are two seperate steps | |
| 11:01:14 | lyarwood | jawad_axd: you don't need access to the volume to fetch the passphrase from the key manager | |
| 11:01:37 | lyarwood | jawad_axd: you do need access to the volume or a downloaded image to decrypt the actual data held within them | |
| 11:02:29 | jawad_axd | @lyarwood I am actually concerned about passphrase here. How to get it from key manager.. I | |
| 11:03:03 | lyarwood | jawad_axd: kk then it's this part | |
| 11:03:13 | lyarwood | openstack secret get --payload_content_type 'application/octet-stream' http://192.168.122.208/key-manager/v1/secrets/6fd4f879-005d-4b7d-9e5f-2505f010be7c --file mysecret.key | |
| 11:03:37 | lyarwood | jawad_axd: ^ where the URL is provided by `openstack secret list` and lists your secret UUID | |
| 11:04:25 | lyarwood | ah yeah it does | |
| 11:04:28 | lyarwood | https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=show-a-volume-s-details-detail#show-a-volume-s-details | |
| 11:04:39 | lyarwood | if you do an `openstack volume show $volume` | |
| 11:04:46 | lyarwood | encryption_key_id is the secret uuid | |
| 11:05:27 | lyarwood | jawad_axd: `hexdump -e '16/1 "%02x"' mysecret.key` will then give you the passphrase used to unlock the LUKS header | |
| 11:05:39 | lyarwood | which is awful | |
| 11:05:41 | lyarwood | but here we are | |
| 11:06:45 | lyarwood | jawad_axd: FWIW by default only the owner of the volume should have access to this secret | |
| 11:12:08 | jawad_axd | @lyarwood Yesss... I just tested and got passphrase from "hexdump -e '16/1 "%02x"' mysecret.key" . Very cool. Thanks alot. | |
| 12:50:02 | stephenfin | gibi: Done | |
| 12:52:41 | jrosser | stephenfin: thankyou for the api version tip before, i am now able to put a boot-from-volume instance into rescue | |
| 12:52:52 | stephenfin | hth | |
| 12:53:02 | jrosser | it doesnt quite behave as i expect though :) | |
| 12:53:17 | jrosser | i seem to get the original instance root disk as the root disk in the rescue instance | |
| 12:54:01 | jrosser | this is the xml i get for the disks http://paste.openstack.org/show/803376/ | |
| 13:05:25 | lyarwood | jrosser: yup that's by design, it's using stable device rescue so the original disks are presented first | |
| 13:05:33 | lyarwood | jrosser: with the rescue disk last | |
| 13:06:01 | jrosser | but i would expect it to boot off the USB disk in that case? | |
| 13:06:18 | lyarwood | jrosser: note the boot order element, it boots from the rescue disk | |
| 13:06:35 | gibi | stephenfin: thanks +2 | |
| 13:06:51 | jrosser | yeah, so with hw_rescue_bus=usb i don't see it boot from the usb device | |
| 13:07:06 | lyarwood | jrosser: is it the same image in the volume and rescue image? | |
| 13:07:28 | jrosser | it should be, +/- the snapshots in the ceph backend | |
| 13:07:55 | lyarwood | jrosser: right so are you sure it's booting from the original volume? | |
| 13:08:04 | lyarwood | this caught be out a few times while testing this | |
| 13:08:48 | jrosser | yes, its confusing, one moment | |
| 13:09:10 | lyarwood | hmm actually I wonder if this is a valid bug when using a different bus | |
| 13:09:20 | jrosser | ok before i check that, i had another thing with hw_rescue_bus=scsi | |
| 13:09:27 | lyarwood | is there the original boot element higher in the XML? | |
| 13:09:44 | jrosser | in the instance console bus=scsi gives "no bootable device" | |
| 13:14:13 | jrosser | lyarwood: i've just redone it with the bus=usb http://paste.openstack.org/show/803378/ | |
| 13:14:44 | sean-k-mooney | lyarwood: we dont really suppor t mixing buses properly | |
| 13:14:59 | sean-k-mooney | i dont think we generate the contoler properly in all cases | |
| 13:15:41 | lyarwood | sean-k-mooney: the instance was already using SCSI in the bus=scsi case so that should work | |
| 13:16:00 | lyarwood | sean-k-mooney: and bus=usb worked for me while I was testing this back in the day with virtio disks attached | |
| 13:16:21 | sean-k-mooney | yep but i thik we have some edge cases with scsi and virtio blk | |
| 13:16:50 | lyarwood | yeah | |
| 13:17:05 | sean-k-mooney | usb would likely have worked because the usb contoler was previously always added by libvirt | |
| 13:17:25 | sean-k-mooney | i have seen issue where the scsi contoler was not alwasy added but that might have been fixed by now | |
| 13:17:33 | lyarwood | jrosser: just building an env now to play with this | |
| 13:18:19 | jrosser | lyarwood: oh cool, thankyou :) | |
| 13:18:31 | lyarwood | jrosser: oh wait, does this reproduce if you use a different image? | |
| 13:18:49 | jrosser | i can try that | |
| 13:19:18 | lyarwood | yeah please, it might be the rescue disk is finding the original disk first and mounting it as the root filesystem as the labels match | |
| 13:19:49 | lyarwood | so we are booted into the kernel from the rescue disk using the filesystem from the original | |
| 13:20:38 | jrosser | also somewhat contrary to the stuff right at the end of here, leaving --image off does some fail-y thing i've not yet found https://docs.openstack.org/nova/latest/user/rescue.html | |
| 13:22:07 | lyarwood | hmm with a boot from volume instance it should try to boot from the original image referenced by the volume, if one is present. I forget what the behaviour is if an image wasn't used to create the volume. | |
| 13:24:46 | jrosser | from the wording i'd inferred that no --image would make it use the 'default', i.e the one from the instance being rescued | |
| 13:28:00 | sean-k-mooney | do we support rescue for BFV | |
| 13:28:05 | sean-k-mooney | we didnt for a long time | |
| 13:28:28 | sean-k-mooney | i think it was added in the last 2-3 cycle but cant recall if it landed | |
| 13:28:41 | jrosser | ussuri i think | |
| 13:29:00 | sean-k-mooney | ya i know we still have no supprot for rebuild with bfv | |
| 13:30:11 | jrosser | ah thats interesting, changing the rescue image to one != the original instance makes things work a whole lot better | |
| 13:30:32 | sean-k-mooney | jrosser: the image for rescue if you dont pass an image is the image use to boot the vm unless a rescue image is set in the nova.conf https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.rescue_image_id | |
| 13:30:37 | jrosser | i rescued a focal vm with bionic image and thats now as i expect | |
| 13:30:39 | sean-k-mooney | jrosser: yep it normally does | |
| 13:31:15 | sean-k-mooney | it should work in both casees but you often dont have the same disk lable issues | |
| 13:31:35 | jrosser | theres a bunch of trap doors for the unwary here :) | |
| 13:32:11 | sean-k-mooney | for what its worth i have generally not had issues with this. its typeically just worked | |
| 13:32:23 | sean-k-mooney | even before the stable rescuse work | |
| 13:35:43 | jrosser | heres what i get if i don't pass --image http://paste.openstack.org/show/803379/ | |
| 13:36:24 | lyarwood | jrosser: kk that's a bug | |
| 13:36:34 | lyarwood | jrosser: but glad the original issue is resolved at least | |
| 13:38:50 | lyarwood | https://github.com/openstack/nova/blob/31889ce296d1e1a62fe5825292479009118ddfab/nova/compute/manager.py#L4123-L4130 doesn't look right | |
| 13:38:52 | jrosser | lyarwood: would you expect hw_rescue_bus=scsi to work? | |
| 13:39:17 | lyarwood | jrosser: with a different image and an instance that already had a disk attached via SCSI yes | |
| 13:39:51 | jrosser | feels like something else there as i get "No Bootable device" in the instance console | |
| 13:44:00 | lyarwood | jrosser: would you mind raising a bug for that and the API error above when --image is missing? | |