| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-14 | |||
| 17:52:36 | jaypipes | efried: I'm still super-confused why we need that. | |
| 17:52:55 | jaypipes | efried: and why we can't just instantiate the reportclient variable on __init__ of the compute manager | |
| 17:53:25 | jaypipes | efried: is this to minimize the number of ProviderTree caches or something? | |
| 18:06:12 | jaypipes | efried: never mind... I see you've addressed it in https://review.openstack.org/#/c/620711/13 | |
| 18:16:19 | mriedem | it's funny that an az description field has never been requested for the api | |
| 18:25:10 | jaypipes | mriedem: perhaps because nobody knows wtf an AZ is in Nova? | |
| 18:31:46 | sean-k-mooney | jaypipes: its like a aws AZ right with all the semantics or a totally seperate fault domain and is tototally not a metadata key on an aggregate | |
| 18:32:02 | efried | jaypipes: sorry was afk. Assume any remaining action items are denoted in review comments at this point? | |
| 18:36:28 | jaypipes | efried: yuupers. | |
| 18:36:31 | jaypipes | yuppers | |
| 18:36:46 | jaypipes | sean-k-mooney: lol, yeah right :) | |
| 18:36:55 | efried | jaypipes: Thanks for the review. I'm pretty excited to get this series going. | |
| 18:37:30 | efried | cdent: tape that ^ | |
| 18:40:54 | eandersson | Are there any actual dependencies on the new libvirt / qemu versions for nova-compute? | |
| 18:44:56 | cdent | efried: no sir | |
| 18:54:43 | lennyb | jangutter, moshele. yes. internal environment . I will check it | |
| 18:55:17 | moshele | lennyb: ? | |
| 18:56:58 | mriedem | jaypipes: heh, well, all the more reason to allow a cloud operator to describe their AZs | |
| 18:57:52 | mriedem | eandersson: what do you mean? the libvirt driver has a minimum required libvirt/qemu otherwise nova-compute won't start | |
| 18:58:07 | mriedem | we bump that minimum every other release or so | |
| 18:58:45 | mriedem | some features in the libvirt driver require newer versions of the binaries and those have conditional version checks on them | |
| 18:58:59 | mriedem | e.g. file-backed memory | |
| 19:41:59 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Add free for claimed, allocated devices https://review.openstack.org/616120 | |
| 19:42:00 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Add get_instance_pci_request_from_vif https://review.openstack.org/619929 | |
| 19:42:00 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Allow per-port modification of vnic_type and profile https://review.openstack.org/607365 | |
| 19:42:01 | openstackgerrit | Adrian Chiris proposed openstack/nova master: SR-IOV Live migration indirect port support https://review.openstack.org/620115 | |
| 19:47:57 | adrianc | sean-k-mooney: ^ | |
| 20:04:24 | lennyb | jangutter, moshele. fixed and passed | |
| 20:04:48 | jangutter | lennyb: thanks very much! | |
| 20:36:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Share image membership with instance owner https://review.openstack.org/630769 | |
| 20:38:21 | ade_lee | mriedem, hey - any idea who I could talk to about voume attachments? | |
| 20:38:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Share snapshot image membership with instance owner https://review.openstack.org/630769 | |
| 20:38:34 | mriedem | ade_lee: likely me | |
| 20:38:50 | ade_lee | mriedem, so -- I have a gate job that is failing | |
| 20:39:14 | ade_lee | because I'm trying to attach a volume and then use it - and there is some timing issue | |
| 20:39:30 | ade_lee | whereby it looks like we're trying to use it before its fully attached. | |
| 20:39:49 | ade_lee | we're waiting to see if the volume is "in-use", | |
| 20:40:08 | ade_lee | but thats checking if cinder thinks the volume is in use | |
| 20:40:23 | ade_lee | and not if the instance thinks the volume is attached .. | |
| 20:40:44 | ade_lee | so how/what do I wait on instead? | |
| 20:41:19 | ade_lee | mriedem, the specific gate job that is failing is here -- http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/ | |
| 20:41:22 | mriedem | that should be sufficient, | |
| 20:41:31 | mriedem | nova tells cinder when the local connection happens for the guest vm, | |
| 20:41:38 | mriedem | and that's what should change the volume status to in-use | |
| 20:41:40 | smcginnis | Yeah, I didn't think we marked it as in-use until things were ready. | |
| 20:42:26 | ade_lee | mriedem, smcginnis interesting -- any idea whats going on in the above test then? | |
| 20:42:28 | smcginnis | Could it be that the local device is getting set wrong for some reason? http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/job-output.txt.gz#_2019-01-14_15_22_54_281033 | |
| 20:43:49 | ade_lee | note - the code we are executing is here -- https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/manager.py line 326 | |
| 20:45:12 | mriedem | is something specifically waiting on /dev/vdb ? | |
| 20:45:28 | mriedem | because nova, or at least the libvirt driver, totally ignores the user-requested device name for the attach | |
| 20:45:35 | mriedem | so https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/manager.py#L328 is kind of pointless to pass the device kwarg | |
| 20:46:00 | mriedem | https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/manager.py#L346 | |
| 20:46:03 | mriedem | looks like it probably is | |
| 20:46:05 | ade_lee | mriedem, well - that fails when we actually try to write to the device .. | |
| 20:46:32 | smcginnis | There's only a /dev/vda | |
| 20:46:56 | mriedem | this is the POST to attach the volume it looks like http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/job-output.txt.gz#_2019-01-14_15_21_32_860642 | |
| 20:47:01 | ade_lee | mriedem, so this is the test -- https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/test_volume_encryption.py line 101 | |
| 20:47:41 | mriedem | wtf this is nice http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_14_46_115508 | |
| 20:47:43 | mriedem | unrelated though | |
| 20:48:36 | mriedem | http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_20_47_770302 | |
| 20:48:40 | mriedem | Jan 14 15:20:47.770302 ubuntu-xenial-rax-ord-0001683696 nova-compute[343]: INFO nova.virt.libvirt.driver [None req-c127b555-17c3-4ce7-9c75-b3df8b653f76 tempest-VolumeEncryptionTest-1809383190 tempest-VolumeEncryptionTest-1809383190] [instance: 1f45a2d6-3811-4b85-a13b-68c3eb605dea] Ignoring supplied device name: /dev/vdb | |
| 20:49:00 | mriedem | http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_20_48_012939 | |
| 20:49:00 | mriedem | says it's attaching it to /dev/vdb though | |
| 20:49:05 | mriedem | Jan 14 15:20:48.012939 ubuntu-xenial-rax-ord-0001683696 nova-compute[343]: INFO nova.compute.manager [None req-c127b555-17c3-4ce7-9c75-b3df8b653f76 tempest-VolumeEncryptionTest-1809383190 tempest-VolumeEncryptionTest-1809383190] [instance: 1f45a2d6-3811-4b85-a13b-68c3eb605dea] Attaching volume ed7fc802-ba1c-4639-8723-47ed02bd11de to /dev/vdb | |
| 20:51:11 | lyarwood | ade_lee: are you only seeing this with the _cryptsetup test? | |
| 20:51:13 | mriedem | hmm, i wonder if it's connected at sda | |
| 20:51:14 | mriedem | http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_20_51_033570 | |
| 20:51:38 | ade_lee | lyarwood, seems so | |
| 20:53:22 | mriedem | something sure enjoys calling the barbican secrets API a lot during this attach flow | |
| 20:54:08 | lyarwood | ade_lee: if the _luks version of the test is passing I'd put money on there being an issue with how os-brick is wiring up the encrypted volume on the underlying host | |
| 20:54:22 | mriedem | Jan 14 15:20:51.786765 ubuntu-xenial-rax-ord-0001683696 nova-compute[343]: DEBUG os_brick.encryptors.cryptsetup [None req-c127b555-17c3-4ce7-9c75-b3df8b653f76 tempest-VolumeEncryptionTest-1809383190 tempest-VolumeEncryptionTest-1809383190] opening encrypted volume /dev/sda {{(pid=343) _open_volume /usr/local/lib/python2.7/dist-packages/os_brick/encryptors/cryptsetup.py:109}} | |
| 20:54:29 | mriedem | it's on sda but the test is looking for dev/vdb | |
| 20:54:48 | lyarwood | that's on the compute host itself | |
| 20:54:50 | mriedem | http://logs.openstack.org/67/628667/13/check/grenade-devstack-barbican/613998f/logs/screen-n-cpu.txt.gz#_Jan_14_15_20_51_786765 | |
| 20:55:03 | mriedem | <target bus="virtio" dev="vdb"/> | |
| 20:56:09 | ade_lee | mriedem, not sure if I understand what that means -- does that mean we asked for it to be on vdb and it put it on sda instead? | |
| 20:57:09 | mriedem | maybe not, tempest is sshi'ing into the guest looking for it at vdb | |
| 20:57:14 | mriedem | if the sda thing is host only, then ignore me | |
| 20:57:24 | mriedem | lyarwood seems to know something about cryptsetup | |
| 20:57:37 | mriedem | i remember something about that coming up in berlin too | |
| 20:58:09 | lyarwood | unfortunately, yeah I'd like to drop support for it tbh | |
| 20:58:16 | mriedem | L22 https://etherpad.openstack.org/p/BER-volume-encryption-forum | |
| 20:59:03 | lyarwood | ade_lee: so os-brick currently hacks the device paths on the host to point the instance at the decrypted volume using the original encrypted device path. | |
| 21:01:21 | ade_lee | lyarwood, so os-brick is setting it up sda instead of vdb -- and luks is setting it up on vdb? | |
| 21:02:58 | smcginnis | I don't think so. /dev/vda should be the boot volume. | |
| 21:05:08 | lyarwood | ade_lee: os-brick points /dev/disk/by-id/scsi-360000000000000000e00000000010001 to the decrypted device on the host, for some reason I can't see the actual commands in the logs anymore. | |
| 21:13:03 | ade_lee | lyarwood, sorry - did you find anything? I'm looking at a couple runs -- one where it worked and one where it didn't - but not sure how to make heads or tails of it | |
| 21:13:40 | lyarwood | ade_lee: yeah sorry I can't see anything obvious on the host / os-brick side of things tbh | |
| 21:14:16 | lyarwood | ade_lee: /dev/disk/by-id/scsi-360000000000000000e00000000010001 should point to the decrypted device that in turn points to encrypted /dev/sda | |
| 21:15:09 | ade_lee | lyarwood, well sdb right? | |
| 21:18:31 | lyarwood | ade_lee: nope, this is on the host sorry | |
| 21:19:03 | lyarwood | ade_lee: like I said there's a load of hacks with this legacy encryptor to decrypt the volume on the host itself | |
| 21:19:35 | ade_lee | lyarwood, it seems to work intermittently .. so not sure whats going on | |
| 21:21:54 | lyarwood | ade_lee: hard to say but if we can't read/write to the device once it's connected within the instance then I'd start by looking at the devicemapper layer on the compute host itself. | |
| 21:23:16 | lyarwood | ade_lee: if it's intermittent it might be a race between os-brick connecting another device while also setting up the decrypt devices for this volume. | |
| 21:24:54 | ade_lee | lyarwood, would that show up in the logs somehow? | |
| 21:25:31 | efried | mriedem: Doesn't deleting the compute service remove its node provider(s) from placement? | |
| 21:27:43 | lyarwood | ade_lee: yeah, os-brick keeps reporting the device path as /dev/disk/by-id/scsi-360000000000000000e00000000010001 once each volume is connected on the host. | |
| 21:28:17 | lyarwood | ade_lee: we use `scsi-360000000000000000e00000000010001` as part of the device name when using cryptsetup to decrypt | |
| 21:31:27 | lyarwood | ade_lee: I need to head offline now but it does look like we are overloading the /dev/disk/by-id/scsi-360000000000000000e00000000010001 path during these tests somehow | |