| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-09 | |||
| 18:00:02 | jmccarthy | Hmm trying to setup an encrypted volume, anyone know where does it come up with this device name ? /dev/disk/by-id/scsi-360014057e4c08a456174ed99b53aa5d6 (paste.openstack.org/show/720695/) | |
| 18:00:18 | jmccarthy | It all goes wrong after this, and I can't find that device anywhere .. | |
| 18:00:21 | mriedem | jmccarthy: os-brick library | |
| 18:02:35 | jmccarthy | mriedem: Hmm ok and it's making it up ? Should I be able to 'see' it someplace ? | |
| 18:02:55 | mriedem | lyarwood: is your guy for encrypted volume stuff in os-brick | |
| 18:03:25 | jmccarthy | mriedem: kk ! | |
| 18:03:27 | lyarwood | jmccarthy: what are you trying to find, the original volume on the compute host? | |
| 18:04:07 | jmccarthy | lyarwood: I'm just trying to find where it's going wrong, I can attach unencrpyted, but not encrypted volumes paste.openstack.org/show/720695/ | |
| 18:04:27 | jmccarthy | It goes wrong after here - as this device can't be found anywhere ? Or I'm looking in wrong places | |
| 18:05:54 | lyarwood | jmccarthy: paste.openstack.org isn't working for me, can you use https://paste.fedoraproject.org/ | |
| 18:06:56 | jmccarthy | lyarwood: https://paste.fedoraproject.org/paste/nL0L92U4ArsPfyVkgQlX-Q | |
| 18:07:24 | jmccarthy | It's brief .. but in the case where it works, I'm pretty sure I find that device under /dev/disk/by-id on the compute host | |
| 18:07:37 | lyarwood | jmccarthy: yeah that's expected the first time you attach the volume | |
| 18:07:54 | lyarwood | jmccarthy: well, with the older flow where os-brick formats the volume | |
| 18:07:59 | jmccarthy | That's expected to return 1 is it ? | |
| 18:08:23 | jmccarthy | lyarwood: It's expected to return 1 I mean is it ? | |
| 18:08:36 | jmccarthy | lyarwood: Ok let me try and attach it again | |
| 18:08:55 | lyarwood | jmccarthy: yeah I think so, 0 is the device is encrypted already, 1 if it isn't | |
| 18:09:22 | mriedem | mdbooth: https://review.openstack.org/#/c/528362/14/nova/tests/unit/virt/test_block_device.py@283 | |
| 18:09:27 | mriedem | am i missing something? | |
| 18:09:32 | jmccarthy | lyarwood: Ok my bad, I assumed that was the start of where my attach failed | |
| 18:10:01 | jmccarthy | lyarwood: I'll look at logs more, it's going wrong but I'm doing a bad job at figuring why lol | |
| 18:10:35 | lyarwood | jmccarthy: np, it's an odd flow tbh, feel free to throw another pastebin my way if you want a hand :) | |
| 18:11:41 | jmccarthy | lyarwood: Ok thanks ! Will do :) | |
| 18:13:57 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add missing encryption_secret_uuid tests https://review.openstack.org/540679 | |
| 18:15:27 | jmccarthy | lyarwood: Maybe this is where I'm going wrong ? https://paste.fedoraproject.org/paste/oSRFKMBJiuCEifxfLXUxhg the volume says it's attaching for a short time, and then just goes back to available | |
| 18:16:21 | lyarwood | jhesketh: yeah it's disconnecting after the libvirt failure to attach the disk to the domain | |
| 18:18:48 | lyarwood | jmccarthy: can you pastebin the entire req-138f35c8-ed21-4430-bf68-525574657926 flow ? | |
| 18:19:15 | jmccarthy | lyarwood: Sure - let me get that together | |
| 18:24:31 | jmccarthy | lyarwood: I think this is all of it ? https://paste.fedoraproject.org/paste/6sdkA90XTlkpftZL5j5D~w | |
| 18:28:34 | lyarwood | jmccarthy: odd, which version of Nova is this? | |
| 18:29:04 | jmccarthy | lyarwood: This is a kolla deploy, queens | |
| 18:29:07 | lyarwood | attach device xml: <disk type="block" device="disk"> is wrong | |
| 18:30:04 | jmccarthy | lyarwood: Ok let me check that a bit more - I do have one setup that is working and one that isn't - I just can't see wtf is going on | |
| 18:30:40 | jmccarthy | lyarwood: Great - Ok, I'll go try and see what is the deal with that - thanks ! | |
| 18:32:02 | lyarwood | jmccarthy: can you just double check that line and make sure it wasn't cut off in your terminal etc | |
| 18:32:22 | jmccarthy | lyarwood: One sec | |
| 18:34:16 | jmccarthy | lyarwood: Ok sorry - part of that bit was missed - this ? https://paste.fedoraproject.org/paste/jR3KP6NlhvTjQZrTl~yK~w | |
| 18:34:47 | lyarwood | jmccarthy: yeah that's better | |
| 18:34:53 | lyarwood | jmccarthy: okay so the XML looks good | |
| 18:35:27 | lyarwood | jmccarthy: are you using fixed_key in nova.conf? | |
| 18:36:07 | jmccarthy | lyarwood: Nope - should I be ? I'm going by this here: https://docs.openstack.org/cinder/queens/configuration/block-storage/volume-encryption.html | |
| 18:36:42 | jmccarthy | lyarwood: I'm pretty sure they are both configured the same, but setup1 works and setup2 doesn't - I have the logs I'm just going crosseyed with where it's going wrong | |
| 18:37:05 | lyarwood | jmccarthy: is it the same key in both envs? | |
| 18:37:14 | lyarwood | jmccarthy: and is the key the same in nova.conf and cinder.conf? | |
| 18:37:33 | lyarwood | ah sorry that doc is using barbican | |
| 18:38:11 | jmccarthy | lyarwood: the setups are not connected to each other at all as such, just in the same lab | |
| 18:38:29 | jmccarthy | lyarwood: nothing is shared | |
| 18:38:55 | lyarwood | jmccarthy: kk can you do a cinder show of the volume? | |
| 18:40:07 | jmccarthy | lyarwood: Should be this here: https://paste.fedoraproject.org/paste/SQ5quYSt8wKOV~-eTqLNMA | |
| 18:40:41 | jmccarthy | lyarwood: I have gone through this a few times, I think the logs aligh with where it's at atm | |
| 18:40:56 | jmccarthy | lyarwood: * s/aligh/align | |
| 18:42:54 | jmccarthy | lyarwood: maybe the log I sent earlier doesn't make senses .. it was grepped on req-138f35c8-ed21-4430-bf68-525574657926 | |
| 18:43:04 | lyarwood | jmccarthy: yeah there's an underlying libvirt issue here with the attach, I'm just trying to rule out some older known issues with passphrase length etc, for example https://bugzilla.redhat.com/show_bug.cgi?id=1447297 | |
| 18:43:04 | openstack | bugzilla.redhat.com bug 1447297 in libvirt "Unable to use LUKS passphrase that is exactly 16 bytes long" [Unspecified,Closed: errata] - Assigned to pkrempa | |
| 18:44:22 | lyarwood | jmccarthy: final ask, `openstack volume type show luks` just to confirm the key_size | |
| 18:44:33 | lyarwood | ^ I think that's the correct command at least | |
| 18:45:02 | lyarwood | https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/volume-type.html#volume-type-show yeah it is | |
| 18:45:32 | lyarwood | openstack volume type show --encryption-type luks | |
| 18:47:30 | jmccarthy | lyarwood: https://paste.fedoraproject.org/paste/34qbpIwDx867wt66JCnUlA | |
| 18:48:25 | lyarwood | jmccarthy: yeah 256 should be fine, odd. | |
| 18:48:36 | jmccarthy | lyarwood: I should be lioadm for the scsi_helper I think it is, right ? | |
| 18:48:37 | lyarwood | jmccarthy: is the same version of libvirt installed in both envs? | |
| 18:49:10 | lyarwood | jmccarthy: that really depends on the underlying env but that isn't the issue here | |
| 18:49:29 | lyarwood | jmccarthy: we can see the raw volume, crypsetup is able to format it etc | |
| 18:49:44 | lyarwood | jmccarthy: libvirt just borks when you try to attach it to the instance | |
| 18:50:00 | jmccarthy | lyarwood: I should have maybe mentioned earlier :/ one is tgdadm I think ? vs lioadm | |
| 18:50:47 | jmccarthy | lyarwood: Should be same version, have to confirm | |
| 18:51:02 | lyarwood | jmccarthy: I still can't see how that could cause this failure tbh | |
| 18:52:25 | jmccarthy | lyarwood: virsh --version says 3.9.0 on both | |
| 18:54:00 | jmccarthy | lyarwood: I know ? | |
| 18:54:24 | lyarwood | jmccarthy: kk, I'm out of ideas on the OpenStack / Nova side tbh, do you have libvirtd logs? | |
| 18:54:40 | lyarwood | jmccarthy: I need to drop in 5 btw before my wife kills me | |
| 18:55:10 | jmccarthy | lyarwood: Oh I did earlier lemme check again - they didn't say much - it's all good I'm fading also - I'll step through them tomorrow | |
| 18:56:09 | jmccarthy | lyarwood: Just more of this here `error : qemuMonitorJSONCheckError:389 : internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'` | |
| 18:56:31 | jmccarthy | lyarwood: Ok maybe I can get some more debug log out of that | |
| 18:56:55 | jmccarthy | lyarwood: Thanks for the help ! I thought maybe it was something obvious I was just missing, but I guess not | |
| 18:57:36 | lyarwood | jmccarthy: yeah increase the logging there and it might shed some light, iirc I've seen this before when the passphrase wouldn't unlock the volume, I just can't see how that's possible with the current flow tbh. | |
| 18:58:08 | jmccarthy | lyarwood: o/ | |
| 18:58:25 | jmccarthy | lyarwood: \o lol | |
| 18:58:36 | jmccarthy | Zzz | |
| 19:53:11 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_instance_objects_sorted() be smart about cells https://review.openstack.org/509003 | |
| 19:55:11 | mriedem | dansmith: on this heal allocations CLI, i'm currently pulling instaces per cell and filtering by task_state=None since we don't want to mess with instances that are going through some task state transition, | |
| 19:55:33 | dansmith | yeah | |
| 19:55:37 | mriedem | however, for the purpose of reporting why we didn't create allocations for a given instance (like we do if it didn't have a host, e.g. shelved offloaded), | |
| 19:55:55 | mriedem | i'm thinking about removing that filter so we can just check instance.task_state in code and if not None, we log something and continue | |
| 19:56:30 | mriedem | because it would be weird to say we healed all instances when we don't know that we actually got all instances in a cell | |
| 19:56:50 | dansmith | okay | |
| 19:57:14 | dansmith | in many cases the instances with a task state outstanding will be a fraction of the total | |
| 19:57:22 | dansmith | not all cases, but probably reasonable as long as you're batching | |
| 19:58:18 | mriedem | by default we pull 50 instances per iteration | |
| 19:59:02 | dansmith | yeah, I mean I was stating that you were batching, not asking | |
| 20:01:13 | esberglu | efried: melwitt: mriedem: I've enabled run_validation and added scenario testing in PowerVM CI. This should unblock these three in the runway queue | |
| 20:01:16 | esberglu | Snapshot: https://review.openstack.org/#/c/543023/ | |
| 20:01:20 | esberglu | Localdisk: https://review.openstack.org/#/c/549053/ & https://review.openstack.org/#/c/549300/ | |
| 20:01:49 | esberglu | vSCSI is still blocked due to lack of CI, I have noted that on the etherpad | |
| 20:02:52 | dansmith | I thought it was going to make the runs take a long time? | |
| 20:02:58 | mriedem | ++lib/tempest:configure_tempest:429 [m iniset /opt/stack/tempest/etc/tempest.conf validation run_validation True | |
| 20:04:05 | openstackgerrit | Jay Pipes proposed openstack/nova master: add consumer.generation/create missing consumers https://review.openstack.org/557958 | |
| 20:04:06 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: add Project, User and Consumer objects https://review.openstack.org/565403 | |