| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-09 | |||
| 11:01:37 | lyarwood | jawad_axd: you do need access to the volume or a downloaded image to decrypt the actual data held within them | |
| 11:02:29 | jawad_axd | @lyarwood I am actually concerned about passphrase here. How to get it from key manager.. I | |
| 11:03:03 | lyarwood | jawad_axd: kk then it's this part | |
| 11:03:13 | lyarwood | openstack secret get --payload_content_type 'application/octet-stream' http://192.168.122.208/key-manager/v1/secrets/6fd4f879-005d-4b7d-9e5f-2505f010be7c --file mysecret.key | |
| 11:03:37 | lyarwood | jawad_axd: ^ where the URL is provided by `openstack secret list` and lists your secret UUID | |
| 11:04:25 | lyarwood | ah yeah it does | |
| 11:04:28 | lyarwood | https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=show-a-volume-s-details-detail#show-a-volume-s-details | |
| 11:04:39 | lyarwood | if you do an `openstack volume show $volume` | |
| 11:04:46 | lyarwood | encryption_key_id is the secret uuid | |
| 11:05:27 | lyarwood | jawad_axd: `hexdump -e '16/1 "%02x"' mysecret.key` will then give you the passphrase used to unlock the LUKS header | |
| 11:05:39 | lyarwood | which is awful | |
| 11:05:41 | lyarwood | but here we are | |
| 11:06:45 | lyarwood | jawad_axd: FWIW by default only the owner of the volume should have access to this secret | |
| 11:12:08 | jawad_axd | @lyarwood Yesss... I just tested and got passphrase from "hexdump -e '16/1 "%02x"' mysecret.key" . Very cool. Thanks alot. | |
| 12:50:02 | stephenfin | gibi: Done | |
| 12:52:41 | jrosser | stephenfin: thankyou for the api version tip before, i am now able to put a boot-from-volume instance into rescue | |
| 12:52:52 | stephenfin | hth | |
| 12:53:02 | jrosser | it doesnt quite behave as i expect though :) | |
| 12:53:17 | jrosser | i seem to get the original instance root disk as the root disk in the rescue instance | |
| 12:54:01 | jrosser | this is the xml i get for the disks http://paste.openstack.org/show/803376/ | |
| 13:05:25 | lyarwood | jrosser: yup that's by design, it's using stable device rescue so the original disks are presented first | |
| 13:05:33 | lyarwood | jrosser: with the rescue disk last | |
| 13:06:01 | jrosser | but i would expect it to boot off the USB disk in that case? | |
| 13:06:18 | lyarwood | jrosser: note the boot order element, it boots from the rescue disk | |
| 13:06:35 | gibi | stephenfin: thanks +2 | |
| 13:06:51 | jrosser | yeah, so with hw_rescue_bus=usb i don't see it boot from the usb device | |
| 13:07:06 | lyarwood | jrosser: is it the same image in the volume and rescue image? | |
| 13:07:28 | jrosser | it should be, +/- the snapshots in the ceph backend | |
| 13:07:55 | lyarwood | jrosser: right so are you sure it's booting from the original volume? | |
| 13:08:04 | lyarwood | this caught be out a few times while testing this | |
| 13:08:48 | jrosser | yes, its confusing, one moment | |
| 13:09:10 | lyarwood | hmm actually I wonder if this is a valid bug when using a different bus | |
| 13:09:20 | jrosser | ok before i check that, i had another thing with hw_rescue_bus=scsi | |
| 13:09:27 | lyarwood | is there the original boot element higher in the XML? | |
| 13:09:44 | jrosser | in the instance console bus=scsi gives "no bootable device" | |
| 13:14:13 | jrosser | lyarwood: i've just redone it with the bus=usb http://paste.openstack.org/show/803378/ | |
| 13:14:44 | sean-k-mooney | lyarwood: we dont really suppor t mixing buses properly | |
| 13:14:59 | sean-k-mooney | i dont think we generate the contoler properly in all cases | |
| 13:15:41 | lyarwood | sean-k-mooney: the instance was already using SCSI in the bus=scsi case so that should work | |
| 13:16:00 | lyarwood | sean-k-mooney: and bus=usb worked for me while I was testing this back in the day with virtio disks attached | |
| 13:16:21 | sean-k-mooney | yep but i thik we have some edge cases with scsi and virtio blk | |
| 13:16:50 | lyarwood | yeah | |
| 13:17:05 | sean-k-mooney | usb would likely have worked because the usb contoler was previously always added by libvirt | |
| 13:17:25 | sean-k-mooney | i have seen issue where the scsi contoler was not alwasy added but that might have been fixed by now | |
| 13:17:33 | lyarwood | jrosser: just building an env now to play with this | |
| 13:18:19 | jrosser | lyarwood: oh cool, thankyou :) | |
| 13:18:31 | lyarwood | jrosser: oh wait, does this reproduce if you use a different image? | |
| 13:18:49 | jrosser | i can try that | |
| 13:19:18 | lyarwood | yeah please, it might be the rescue disk is finding the original disk first and mounting it as the root filesystem as the labels match | |
| 13:19:49 | lyarwood | so we are booted into the kernel from the rescue disk using the filesystem from the original | |
| 13:20:38 | jrosser | also somewhat contrary to the stuff right at the end of here, leaving --image off does some fail-y thing i've not yet found https://docs.openstack.org/nova/latest/user/rescue.html | |
| 13:22:07 | lyarwood | hmm with a boot from volume instance it should try to boot from the original image referenced by the volume, if one is present. I forget what the behaviour is if an image wasn't used to create the volume. | |
| 13:24:46 | jrosser | from the wording i'd inferred that no --image would make it use the 'default', i.e the one from the instance being rescued | |
| 13:28:00 | sean-k-mooney | do we support rescue for BFV | |
| 13:28:05 | sean-k-mooney | we didnt for a long time | |
| 13:28:28 | sean-k-mooney | i think it was added in the last 2-3 cycle but cant recall if it landed | |
| 13:28:41 | jrosser | ussuri i think | |
| 13:29:00 | sean-k-mooney | ya i know we still have no supprot for rebuild with bfv | |
| 13:30:11 | jrosser | ah thats interesting, changing the rescue image to one != the original instance makes things work a whole lot better | |
| 13:30:32 | sean-k-mooney | jrosser: the image for rescue if you dont pass an image is the image use to boot the vm unless a rescue image is set in the nova.conf https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.rescue_image_id | |
| 13:30:37 | jrosser | i rescued a focal vm with bionic image and thats now as i expect | |
| 13:30:39 | sean-k-mooney | jrosser: yep it normally does | |
| 13:31:15 | sean-k-mooney | it should work in both casees but you often dont have the same disk lable issues | |
| 13:31:35 | jrosser | theres a bunch of trap doors for the unwary here :) | |
| 13:32:11 | sean-k-mooney | for what its worth i have generally not had issues with this. its typeically just worked | |
| 13:32:23 | sean-k-mooney | even before the stable rescuse work | |
| 13:35:43 | jrosser | heres what i get if i don't pass --image http://paste.openstack.org/show/803379/ | |
| 13:36:24 | lyarwood | jrosser: kk that's a bug | |
| 13:36:34 | lyarwood | jrosser: but glad the original issue is resolved at least | |
| 13:38:50 | lyarwood | https://github.com/openstack/nova/blob/31889ce296d1e1a62fe5825292479009118ddfab/nova/compute/manager.py#L4123-L4130 doesn't look right | |
| 13:38:52 | jrosser | lyarwood: would you expect hw_rescue_bus=scsi to work? | |
| 13:39:17 | lyarwood | jrosser: with a different image and an instance that already had a disk attached via SCSI yes | |
| 13:39:51 | jrosser | feels like something else there as i get "No Bootable device" in the instance console | |
| 13:44:00 | lyarwood | jrosser: would you mind raising a bug for that and the API error above when --image is missing? | |
| 13:44:17 | lyarwood | I'm not sure about the SCSI failure to find a boot device tbh | |
| 13:44:32 | lyarwood | unless again it's something weird with the image | |
| 13:44:47 | sean-k-mooney | jrosser: which image did you set the rescue bus on? | |
| 13:45:02 | jrosser | the one i'm specifying with --image | |
| 13:45:05 | sean-k-mooney | if you dont specify it it would have to be on the original image | |
| 13:45:20 | sean-k-mooney | ya ok that should be the image whos metadata we use | |
| 13:46:55 | jrosser | in this case turns out those options are set on both images i've tried as the rescue image now | |
| 13:47:12 | jrosser | one of which is the original instance image | |
| 14:13:52 | jrosser | lyarwood: bug reports done | |
| 14:14:42 | jrosser | thankyou again for your help, i have something usable now with the usb bus and understanding the need for a different rescue image | |
| 14:17:00 | lyarwood | jrosser: np and thanks for the bugs, I'll try to get them resolved after feature freeze later this week | |
| 14:36:40 | sean-k-mooney | lyarwood: by the way do you know why the volumn detach is sometime failing in the live migration job | |
| 14:40:05 | sean-k-mooney | looks like its hitting nova.exception.DeviceDetachFailed: Device detach failed for vdb: Unable to detach the device from the live config. | |
| 14:41:53 | lyarwood | sean-k-mooney: no, I've been trying to push gibi's rework along to see if that resolved it tbh | |
| 14:42:24 | sean-k-mooney | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unable%20to%20detach%20the%20device%20from%20the%20live%20config%5C%22%20AND%20loglevel%3A%20ERROR | |
| 14:43:00 | lyarwood | sean-k-mooney: there's nothing obvious in the logs but I wonder if it could be related to https://bugs.launchpad.net/cinder/+bug/1917750 | |
| 14:43:01 | openstack | Launchpad bug 1917750 in Cinder "Running parallel iSCSI/LVM c-vol backends is causing random failures in CI" [Undecided,New] | |
| 14:43:07 | sean-k-mooney | ~300 hits in 30 days | |
| 14:43:52 | sean-k-mooney | im seeing some libvirt issue on the contoler too not the node with teh detach issue | |
| 14:44:47 | lyarwood | do you have an example to hand? | |
| 14:45:19 | sean-k-mooney | am its hitting my vdpa pataches but also neutron let me get one | |
| 14:47:54 | sean-k-mooney | so ya https://review.opendev.org/c/openstack/nova/+/778350/4 https://zuul.opendev.org/t/openstack/build/fb643b53835341ac8589afeadfa7044d/logs | |
| 14:49:08 | sean-k-mooney | its showing up in neutron too https://review.opendev.org/c/openstack/neutron/+/777785 | |
| 14:49:50 | sean-k-mooney | in https://zuul.opendev.org/t/openstack/build/32f3dd64008b469eb9fb8b13ed33f137 | |
| 14:50:04 | sean-k-mooney | so i think this is just an issue with master in general | |
| 14:50:27 | sean-k-mooney | it could be related to https://bugs.launchpad.net/cinder/+bug/1917750 maybe havent looked at it yet | |