| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-09 | |||
| 16:16:56 | dansmith | yes | |
| 16:27:02 | mriedem | artom: ok +2 on queens, +1 on pike | |
| 16:32:00 | artom | mriedem, dansmith, thank you gentlemen | |
| 16:39:01 | openstackgerrit | Dan Smith proposed openstack/nova master: Add CellMapping.get_by_project_id() query method https://review.openstack.org/509002 | |
| 16:39:02 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_instance_objects_sorted() be smart about cells https://review.openstack.org/509003 | |
| 17:01:29 | mdbooth | mriedem: If you get a sec could you check I haven't misrepresented you in my response to jaypipes here: https://review.openstack.org/#/c/528363/15/nova/virt/block_device.py ? | |
| 17:04:43 | mriedem | replied | |
| 17:05:43 | mriedem | i really need to be reviewing that series given my somewhat grossly intimate relationship with that code now | |
| 17:06:23 | mriedem | which i plan on making my afternoon | |
| 17:08:04 | mriedem | mdbooth: the entire goal of that series is to get a serial value in the non-volume block devices right? since we only have serial == volume id today, so for the non-volumes we'll use the bdm.uuid | |
| 17:08:30 | mriedem | and the 20 patch refactor series is because that's how you roll :) | |
| 17:08:58 | mriedem | no offense, i know this code is horrible | |
| 17:11:14 | melwitt | mriedem: cool, thanks for the etherpad for the stable release needs | |
| 17:14:41 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework how we pass candidate request information https://review.openstack.org/566166 | |
| 17:16:14 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework how we pass candidate request information https://review.openstack.org/566166 | |
| 17:22:01 | openstackgerrit | Merged openstack/nova master: Fix detach_volume calls when rolling back a failed attach https://review.openstack.org/563213 | |
| 17:47:33 | mriedem | dansmith: lyarwood: https://review.openstack.org/#/c/567232/ should be good to go now - change on master just merged | |
| 18:00:02 | jmccarthy | Hmm trying to setup an encrypted volume, anyone know where does it come up with this device name ? /dev/disk/by-id/scsi-360014057e4c08a456174ed99b53aa5d6 (paste.openstack.org/show/720695/) | |
| 18:00:18 | jmccarthy | It all goes wrong after this, and I can't find that device anywhere .. | |
| 18:00:21 | mriedem | jmccarthy: os-brick library | |
| 18:02:35 | jmccarthy | mriedem: Hmm ok and it's making it up ? Should I be able to 'see' it someplace ? | |
| 18:02:55 | mriedem | lyarwood: is your guy for encrypted volume stuff in os-brick | |
| 18:03:25 | jmccarthy | mriedem: kk ! | |
| 18:03:27 | lyarwood | jmccarthy: what are you trying to find, the original volume on the compute host? | |
| 18:04:07 | jmccarthy | lyarwood: I'm just trying to find where it's going wrong, I can attach unencrpyted, but not encrypted volumes paste.openstack.org/show/720695/ | |
| 18:04:27 | jmccarthy | It goes wrong after here - as this device can't be found anywhere ? Or I'm looking in wrong places | |
| 18:05:54 | lyarwood | jmccarthy: paste.openstack.org isn't working for me, can you use https://paste.fedoraproject.org/ | |
| 18:06:56 | jmccarthy | lyarwood: https://paste.fedoraproject.org/paste/nL0L92U4ArsPfyVkgQlX-Q | |
| 18:07:24 | jmccarthy | It's brief .. but in the case where it works, I'm pretty sure I find that device under /dev/disk/by-id on the compute host | |
| 18:07:37 | lyarwood | jmccarthy: yeah that's expected the first time you attach the volume | |
| 18:07:54 | lyarwood | jmccarthy: well, with the older flow where os-brick formats the volume | |
| 18:07:59 | jmccarthy | That's expected to return 1 is it ? | |
| 18:08:23 | jmccarthy | lyarwood: It's expected to return 1 I mean is it ? | |
| 18:08:36 | jmccarthy | lyarwood: Ok let me try and attach it again | |
| 18:08:55 | lyarwood | jmccarthy: yeah I think so, 0 is the device is encrypted already, 1 if it isn't | |
| 18:09:22 | mriedem | mdbooth: https://review.openstack.org/#/c/528362/14/nova/tests/unit/virt/test_block_device.py@283 | |
| 18:09:27 | mriedem | am i missing something? | |
| 18:09:32 | jmccarthy | lyarwood: Ok my bad, I assumed that was the start of where my attach failed | |
| 18:10:01 | jmccarthy | lyarwood: I'll look at logs more, it's going wrong but I'm doing a bad job at figuring why lol | |
| 18:10:35 | lyarwood | jmccarthy: np, it's an odd flow tbh, feel free to throw another pastebin my way if you want a hand :) | |
| 18:11:41 | jmccarthy | lyarwood: Ok thanks ! Will do :) | |
| 18:13:57 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add missing encryption_secret_uuid tests https://review.openstack.org/540679 | |
| 18:15:27 | jmccarthy | lyarwood: Maybe this is where I'm going wrong ? https://paste.fedoraproject.org/paste/oSRFKMBJiuCEifxfLXUxhg the volume says it's attaching for a short time, and then just goes back to available | |
| 18:16:21 | lyarwood | jhesketh: yeah it's disconnecting after the libvirt failure to attach the disk to the domain | |
| 18:18:48 | lyarwood | jmccarthy: can you pastebin the entire req-138f35c8-ed21-4430-bf68-525574657926 flow ? | |
| 18:19:15 | jmccarthy | lyarwood: Sure - let me get that together | |
| 18:24:31 | jmccarthy | lyarwood: I think this is all of it ? https://paste.fedoraproject.org/paste/6sdkA90XTlkpftZL5j5D~w | |
| 18:28:34 | lyarwood | jmccarthy: odd, which version of Nova is this? | |
| 18:29:04 | jmccarthy | lyarwood: This is a kolla deploy, queens | |
| 18:29:07 | lyarwood | attach device xml: <disk type="block" device="disk"> is wrong | |
| 18:30:04 | jmccarthy | lyarwood: Ok let me check that a bit more - I do have one setup that is working and one that isn't - I just can't see wtf is going on | |
| 18:30:40 | jmccarthy | lyarwood: Great - Ok, I'll go try and see what is the deal with that - thanks ! | |
| 18:32:02 | lyarwood | jmccarthy: can you just double check that line and make sure it wasn't cut off in your terminal etc | |
| 18:32:22 | jmccarthy | lyarwood: One sec | |
| 18:34:16 | jmccarthy | lyarwood: Ok sorry - part of that bit was missed - this ? https://paste.fedoraproject.org/paste/jR3KP6NlhvTjQZrTl~yK~w | |
| 18:34:47 | lyarwood | jmccarthy: yeah that's better | |
| 18:34:53 | lyarwood | jmccarthy: okay so the XML looks good | |
| 18:35:27 | lyarwood | jmccarthy: are you using fixed_key in nova.conf? | |
| 18:36:07 | jmccarthy | lyarwood: Nope - should I be ? I'm going by this here: https://docs.openstack.org/cinder/queens/configuration/block-storage/volume-encryption.html | |
| 18:36:42 | jmccarthy | lyarwood: I'm pretty sure they are both configured the same, but setup1 works and setup2 doesn't - I have the logs I'm just going crosseyed with where it's going wrong | |
| 18:37:05 | lyarwood | jmccarthy: is it the same key in both envs? | |
| 18:37:14 | lyarwood | jmccarthy: and is the key the same in nova.conf and cinder.conf? | |
| 18:37:33 | lyarwood | ah sorry that doc is using barbican | |
| 18:38:11 | jmccarthy | lyarwood: the setups are not connected to each other at all as such, just in the same lab | |
| 18:38:29 | jmccarthy | lyarwood: nothing is shared | |
| 18:38:55 | lyarwood | jmccarthy: kk can you do a cinder show of the volume? | |
| 18:40:07 | jmccarthy | lyarwood: Should be this here: https://paste.fedoraproject.org/paste/SQ5quYSt8wKOV~-eTqLNMA | |
| 18:40:41 | jmccarthy | lyarwood: I have gone through this a few times, I think the logs aligh with where it's at atm | |
| 18:40:56 | jmccarthy | lyarwood: * s/aligh/align | |
| 18:42:54 | jmccarthy | lyarwood: maybe the log I sent earlier doesn't make senses .. it was grepped on req-138f35c8-ed21-4430-bf68-525574657926 | |
| 18:43:04 | lyarwood | jmccarthy: yeah there's an underlying libvirt issue here with the attach, I'm just trying to rule out some older known issues with passphrase length etc, for example https://bugzilla.redhat.com/show_bug.cgi?id=1447297 | |
| 18:43:04 | openstack | bugzilla.redhat.com bug 1447297 in libvirt "Unable to use LUKS passphrase that is exactly 16 bytes long" [Unspecified,Closed: errata] - Assigned to pkrempa | |
| 18:44:22 | lyarwood | jmccarthy: final ask, `openstack volume type show luks` just to confirm the key_size | |
| 18:44:33 | lyarwood | ^ I think that's the correct command at least | |
| 18:45:02 | lyarwood | https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/volume-type.html#volume-type-show yeah it is | |
| 18:45:32 | lyarwood | openstack volume type show --encryption-type luks | |
| 18:47:30 | jmccarthy | lyarwood: https://paste.fedoraproject.org/paste/34qbpIwDx867wt66JCnUlA | |
| 18:48:25 | lyarwood | jmccarthy: yeah 256 should be fine, odd. | |
| 18:48:36 | jmccarthy | lyarwood: I should be lioadm for the scsi_helper I think it is, right ? | |
| 18:48:37 | lyarwood | jmccarthy: is the same version of libvirt installed in both envs? | |
| 18:49:10 | lyarwood | jmccarthy: that really depends on the underlying env but that isn't the issue here | |
| 18:49:29 | lyarwood | jmccarthy: we can see the raw volume, crypsetup is able to format it etc | |
| 18:49:44 | lyarwood | jmccarthy: libvirt just borks when you try to attach it to the instance | |
| 18:50:00 | jmccarthy | lyarwood: I should have maybe mentioned earlier :/ one is tgdadm I think ? vs lioadm | |
| 18:50:47 | jmccarthy | lyarwood: Should be same version, have to confirm | |
| 18:51:02 | lyarwood | jmccarthy: I still can't see how that could cause this failure tbh | |
| 18:52:25 | jmccarthy | lyarwood: virsh --version says 3.9.0 on both | |
| 18:54:00 | jmccarthy | lyarwood: I know ? | |
| 18:54:24 | lyarwood | jmccarthy: kk, I'm out of ideas on the OpenStack / Nova side tbh, do you have libvirtd logs? | |
| 18:54:40 | lyarwood | jmccarthy: I need to drop in 5 btw before my wife kills me | |
| 18:55:10 | jmccarthy | lyarwood: Oh I did earlier lemme check again - they didn't say much - it's all good I'm fading also - I'll step through them tomorrow | |
| 18:56:09 | jmccarthy | lyarwood: Just more of this here `error : qemuMonitorJSONCheckError:389 : internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'` | |
| 18:56:31 | jmccarthy | lyarwood: Ok maybe I can get some more debug log out of that | |
| 18:56:55 | jmccarthy | lyarwood: Thanks for the help ! I thought maybe it was something obvious I was just missing, but I guess not | |
| 18:57:36 | lyarwood | jmccarthy: yeah increase the logging there and it might shed some light, iirc I've seen this before when the passphrase wouldn't unlock the volume, I just can't see how that's possible with the current flow tbh. | |
| 18:58:08 | jmccarthy | lyarwood: o/ | |
| 18:58:25 | jmccarthy | lyarwood: \o lol | |
| 18:58:36 | jmccarthy | Zzz | |
| 19:53:11 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_instance_objects_sorted() be smart about cells https://review.openstack.org/509003 | |
| 19:55:11 | mriedem | dansmith: on this heal allocations CLI, i'm currently pulling instaces per cell and filtering by task_state=None since we don't want to mess with instances that are going through some task state transition, | |