Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-31
21:59:26 ameeda mriedem: its okay, Thank you for your time and helps.
21:59:43 ameeda mriedem: please don't forget to review this for me https://review.openstack.org/#/c/528385/
22:01:29 mriedem ameeda: at this point i dream about the opportunity to review that patch :)
22:01:33 mriedem it is my destiny
22:05:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_access_get_by_flavor_id db api method https://review.openstack.org/539700
22:05:52 ameeda mriedem: lol, I appreciate that from you,I know that its hard to you to find time to check all reviews
22:06:38 ameeda mriedem: lastly I see +2 on one of my patches ^_^, lol
22:07:44 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_extra_specs_delete db api method https://review.openstack.org/539702
22:13:30 melwitt mriedem: on this backport, we're only concerned about holding it from ocata because of the possible related network-vif-plugged event timeout, but not for pike? https://review.openstack.org/#/c/531422
22:13:53 mriedem melwitt: i am concerned for pike, but we have more time for pike
22:13:56 mriedem so less concerned
22:14:01 melwitt okay
22:14:27 mriedem i thought about dropping my +2
22:14:32 mriedem but i didn't think about it very hard
22:14:46 melwitt fair enough
22:15:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_extra_specs_delete db api method https://review.openstack.org/539702
22:15:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_extra_specs_get db api method https://review.openstack.org/539706
22:17:45 Roamer` hm, has anybody reported problems with the "libvirt: QEMU native LUKS decryption for encrypted volumes" change recently?
22:18:36 melwitt not that I know of. have you experienced problems with it?
22:18:50 Roamer` the StorPool Cinder CI started failing each and every test run with the same problem: test_encrypted_cinder_volumes_luks fails and there is a nova-compute exception: Failed to detach volume 645fd643-89fc-4b3d-9ea5-59c764fc39a2 from /dev/vdb: AttributeError: 'NoneType' object has no attribute 'format_dom'
22:19:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_extra_specs_delete db api method https://review.openstack.org/539702
22:19:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_extra_specs_get db api method https://review.openstack.org/539706
22:19:11 Roamer` just a minute, let me point you at a stack trace
22:19:57 Roamer` http://logs.ci-openstack.storpool.com/18/539318/1/check/dsvm-tempest-storpool/c3daf58/logs/screen-n-cpu.txt.gz#_Jan_31_18_07_27_971552 and a couple of lines further down
22:20:50 Roamer` melwitt: ^^ - it looks like the LibvirtConfigGuestDiskEncryption is missing a "secret" (not initialized; maybe not passed somewhere?)
22:20:56 Roamer` and this happens on every run
22:21:30 melwitt Roamer`: okay. could you please open a bug against nova with a link to the log and excerpt of the stack trace?
22:21:54 Roamer` melwitt, sure, I'll open it right now
22:21:56 melwitt we'll have lyarwood take a look at it
22:22:01 melwitt thank you
22:24:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_access_remove db api method https://review.openstack.org/539709
22:34:09 Roamer` lyarwood, melwitt, https://bugs.launchpad.net/nova/+bug/1746609
22:34:10 openstack Launchpad bug 1746609 in OpenStack Compute (nova) "test_boot_server_from_encrypted_volume_luks cannot detach an encrypted StorPool-backed volume" [Undecided,New]
22:34:23 Roamer` it's a bit brief, but it has links to the logs...
22:34:35 openstackgerrit Eric Fried proposed openstack/nova master: Avoid inventory DELETE API (no conflict detection) https://review.openstack.org/539712
22:34:48 efried jaypipes: How about them apples ^
22:35:12 efried Note approx -400LOC net.
22:35:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_access_add db api methods https://review.openstack.org/539714
22:35:30 melwitt Roamer`: thanks. we'll use this to investigate what's going wrong
22:35:38 openstackgerrit Eric Berglund proposed openstack/nova master: Use dict.get() when accessing capabilities dict https://review.openstack.org/539715
22:41:23 mriedem Roamer`: is the storpool volume driver passing a secret in it's connection_info dict when using an encrypted volume type
22:42:01 mriedem i'm not actually sure if that's required
22:42:27 mriedem should probably check lee's patches to see if he had to change anything in devstack for this stuff to work, like with the key manager across both nova and cinder
22:43:00 mriedem https://review.openstack.org/#/q/topic:bp/libvirt-qemu-native-luks+(status:open+OR+status:merged)
22:43:18 mriedem melwitt: you know,
22:43:22 mriedem now that i think about it,
22:43:39 mriedem the tempest test that tests encrypted volumes is marked 'slow' so i don't think it actually runs in a job in our check queue...
22:43:47 mriedem i think there is a job that runs it in the experimental queue
22:44:36 mriedem this one https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L10841
22:46:07 mriedem passes here though http://logs.openstack.org/60/539260/2/experimental/legacy-tempest-dsvm-neutron-scenario-multinode-lvm-multibackend/77a82c8/job-output.txt.gz#_2018-01-31_20_59_15_161538
22:48:01 melwitt hm, okay
22:48:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove old flavor_access_get_by_flavor_id db api method https://review.openstack.org/539720
22:53:10 Roamer` mriedem, the StorPool driver does not generate any structures by itself, at your suggestion it inherits LibvirtVolumeDriver's get_config() method
22:53:52 Roamer` and from what I can see of the libvirt LUKS patch, the secret should be stored by LibvirtBaseVolumeDriver's get_config()
22:54:38 mriedem u'data': {u'device_path': u'/dev/storpool/os--volume-645fd643-89fc-4b3d-9ea5-59c764fc39a2', u'encrypted': True, u'volume': u'645fd643-89fc-4b3d-9ea5-59c764fc39a2', u'qos_specs': None, u'client_id': 12, u'access_mode': u'rw'}
22:54:50 mriedem https://review.openstack.org/#/c/523958/18/nova/virt/libvirt/driver.py@1342
22:55:04 mriedem looks like the libvirt driver is expecting a connection_info['data']['volume_id'] entry
22:56:02 melwitt there must be one, else we'd see a KeyError, right? is it None?
22:56:29 Roamer` hm, ISTR that there was a "volume_id" one level up, on the same level as "data", but let me look again
22:56:37 mriedem https://review.openstack.org/#/c/523958/18/nova/virt/libvirt/volume/volume.py
22:57:08 mriedem melwitt: the libvirt driver code is using .get() to avoid a keyerror
22:57:47 melwitt I see it now
22:58:59 melwitt so is volume_id supposed to always be set in connect_info['data'] or is it valid not to have it?
22:59:17 mriedem shrug,
22:59:31 mriedem connection_info is a wild west of whatever from the bazillion volume drivers in cinder
22:59:48 mriedem i do know that in nova, when we attach, we shove 'serial' into the connection_info if it's not already there
22:59:51 mriedem and serial == volume_id
22:59:56 melwitt sigh
23:00:10 smcginnis I don't think the serial is always the volume_id.
23:00:25 smcginnis RemoteFS based storage I think that's the case, but not others.
23:00:28 mriedem {u'status': u'reserved', u'instance': u'9a7f0101-917d-43f2-ac18-0b7d77c14fda', u'attached_at': u'', u'attach_mode': None, u'driver_volume_type': u'storpool', u'detached_at': u'', u'volume_id': u'645fd643-89fc-4b3d-9ea5-59c764fc39a2', u'serial': u'645fd643-89fc-4b3d-9ea5-59c764fc39a2', u'data': {u'device_path': u'/dev/storpool/os--volume-645fd643-89fc-4b3d-9ea5-59c764fc39a2', u'encrypted': True, u'volume': u'645fd643-89fc-4b3d
23:00:29 mriedem 5-59c764fc39a2', u'qos_specs': None, u'client_id': 12, u'access_mode': u'rw'}}
23:00:49 mriedem smcginnis: it's the volume_id if it's not already in the dict from cinder
23:01:02 smcginnis Ah, OK.
23:01:03 mriedem https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L425
23:01:07 mriedem b/c we shove it in there
23:01:09 mriedem shove it real good
23:01:29 smcginnis Nah, just not the same as push it. Nice try.
23:01:39 melwitt oh, heh. smcginnis came here. I just asked the question in #openstack-cinder if it's valid not to have 'volume_id' in connection_info['data']
23:01:43 mriedem so all of this new encryptor code from lyarwood should probably be falling back to looking for the serial in connection_info if connection_info['data'] doesn't have 'volume_id'
23:02:14 smcginnis I do not think it's safe to assume that.
23:02:23 melwitt mdbooth ^
23:05:28 openstackgerrit Eric Berglund proposed openstack/nova master: Use dict.get() when accessing capabilities dict https://review.openstack.org/539715
23:07:53 mriedem i don't really know how this is failing
23:08:01 mriedem if there is no secret, it shouldn't be set in the conf
23:09:03 melwitt oh, you mean find_secret should return None or something
23:09:23 mriedem idk, i get lost following the damn xml conversion crap in the libvirt driver
23:09:31 melwitt er, sorry it wouldn't even call find_secret
23:11:28 mriedem my guess is we don't get here https://review.openstack.org/#/c/464008/10/nova/virt/libvirt/config.py@1149
23:13:08 melwitt yeah, and we wouldn't because there's no secret in the config
23:13:29 melwitt if c.tag == 'secret': would be False
23:14:32 melwitt so since there was no volume_id, there's no secret in the config, so there's no secret to be parsed into the config object, then it blows up when it goes to format the domain later
23:15:39 mriedem http://logs.ci-openstack.storpool.com/18/539318/1/check/dsvm-tempest-storpool/c3daf58/logs/screen-n-cpu.txt.gz#_Jan_31_18_07_21_994385
23:15:56 mriedem that's the secret xml
23:15:59 mriedem sshhhh
23:16:17 melwitt lol
23:17:07 melwitt so there IS secret xml? sigh, I'm confused
23:17:38 melwitt back to not knowing how this is failing
23:18:14 mriedem right i have no idea
23:18:24 mriedem basically need to wait for lyarwood to wake up

Earlier   Later