| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-09 | |||
| 16:13:00 | dansmith | mriedem: I think warning on pike is reasonable, and I think that's where we hit it downstream.. they didn't realize things had gotten legit out of sync because they had debug off | |
| 16:13:23 | dansmith | when they turned debug on, we realized some instances had allocations on three computes because of legit leaks | |
| 16:13:53 | mriedem | was this pike 16.0.0 GA? | |
| 16:13:58 | artom | dansmith, wait, was that the juicy backported you mentioned in internal IRC? | |
| 16:14:03 | dansmith | I think we made it debug initially because we were pretty sure we'd log that a lot | |
| 16:14:05 | dansmith | artom: no | |
| 16:14:18 | dansmith | but since we've seen it in the wild, it probably needs to be more visible | |
| 16:14:26 | dansmith | mriedem: it was whatever our build is based on | |
| 16:14:45 | dansmith | mriedem: are you asking if it was something past GA because of backports? | |
| 16:15:23 | mriedem | we fixed a lot of leaky allocation stuff post pike GA, | |
| 16:15:28 | dansmith | mriedem: the actual leakage happened some time in the past, they don't know when, but noticed it when computes were refusing to schedule the last bit of resource, which turned out to be because of some stale allocations | |
| 16:15:30 | dansmith | right | |
| 16:15:33 | mriedem | so i'm wondering if the customer hit this on 16.0.0 GA before we fixed the leaks | |
| 16:15:41 | dansmith | entirely possible | |
| 16:15:50 | dansmith | could also have been from ocata, or during/around the upgrade | |
| 16:15:56 | mriedem | i realize the leaks could have been from GA, even if we stopped the leaks later | |
| 16:16:02 | dansmith | yeah | |
| 16:16:13 | dansmith | it wasn't very acute, so I don't think they're just leaking like crazy right now | |
| 16:16:23 | dansmith | it was just "hmm, this should fit, help us figure out why not" | |
| 16:16:26 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347 | |
| 16:16:27 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | |
| 16:16:28 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root_device https://review.openstack.org/567277 | |
| 16:16:28 | mriedem | given we should have fixed those leaks now in pike, if we do hit this, my comment from the change on master probably applies | |
| 16:16:51 | mriedem | i.e. we shouldn't hit this, so if we are leaking, we f'ed up and warning is probably ok | |
| 16:16:56 | dansmith | yes | |
| 16:27:02 | mriedem | artom: ok +2 on queens, +1 on pike | |
| 16:32:00 | artom | mriedem, dansmith, thank you gentlemen | |
| 16:39:01 | openstackgerrit | Dan Smith proposed openstack/nova master: Add CellMapping.get_by_project_id() query method https://review.openstack.org/509002 | |
| 16:39:02 | openstackgerrit | Dan Smith proposed openstack/nova master: Make get_instance_objects_sorted() be smart about cells https://review.openstack.org/509003 | |
| 17:01:29 | mdbooth | mriedem: If you get a sec could you check I haven't misrepresented you in my response to jaypipes here: https://review.openstack.org/#/c/528363/15/nova/virt/block_device.py ? | |
| 17:04:43 | mriedem | replied | |
| 17:05:43 | mriedem | i really need to be reviewing that series given my somewhat grossly intimate relationship with that code now | |
| 17:06:23 | mriedem | which i plan on making my afternoon | |
| 17:08:04 | mriedem | mdbooth: the entire goal of that series is to get a serial value in the non-volume block devices right? since we only have serial == volume id today, so for the non-volumes we'll use the bdm.uuid | |
| 17:08:30 | mriedem | and the 20 patch refactor series is because that's how you roll :) | |
| 17:08:58 | mriedem | no offense, i know this code is horrible | |
| 17:11:14 | melwitt | mriedem: cool, thanks for the etherpad for the stable release needs | |
| 17:14:41 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework how we pass candidate request information https://review.openstack.org/566166 | |
| 17:16:14 | openstackgerrit | Jay Pipes proposed openstack/nova master: rework how we pass candidate request information https://review.openstack.org/566166 | |
| 17:22:01 | openstackgerrit | Merged openstack/nova master: Fix detach_volume calls when rolling back a failed attach https://review.openstack.org/563213 | |
| 17:47:33 | mriedem | dansmith: lyarwood: https://review.openstack.org/#/c/567232/ should be good to go now - change on master just merged | |
| 18:00:02 | jmccarthy | Hmm trying to setup an encrypted volume, anyone know where does it come up with this device name ? /dev/disk/by-id/scsi-360014057e4c08a456174ed99b53aa5d6 (paste.openstack.org/show/720695/) | |
| 18:00:18 | jmccarthy | It all goes wrong after this, and I can't find that device anywhere .. | |
| 18:00:21 | mriedem | jmccarthy: os-brick library | |
| 18:02:35 | jmccarthy | mriedem: Hmm ok and it's making it up ? Should I be able to 'see' it someplace ? | |
| 18:02:55 | mriedem | lyarwood: is your guy for encrypted volume stuff in os-brick | |
| 18:03:25 | jmccarthy | mriedem: kk ! | |
| 18:03:27 | lyarwood | jmccarthy: what are you trying to find, the original volume on the compute host? | |
| 18:04:07 | jmccarthy | lyarwood: I'm just trying to find where it's going wrong, I can attach unencrpyted, but not encrypted volumes paste.openstack.org/show/720695/ | |
| 18:04:27 | jmccarthy | It goes wrong after here - as this device can't be found anywhere ? Or I'm looking in wrong places | |
| 18:05:54 | lyarwood | jmccarthy: paste.openstack.org isn't working for me, can you use https://paste.fedoraproject.org/ | |
| 18:06:56 | jmccarthy | lyarwood: https://paste.fedoraproject.org/paste/nL0L92U4ArsPfyVkgQlX-Q | |
| 18:07:24 | jmccarthy | It's brief .. but in the case where it works, I'm pretty sure I find that device under /dev/disk/by-id on the compute host | |
| 18:07:37 | lyarwood | jmccarthy: yeah that's expected the first time you attach the volume | |
| 18:07:54 | lyarwood | jmccarthy: well, with the older flow where os-brick formats the volume | |
| 18:07:59 | jmccarthy | That's expected to return 1 is it ? | |
| 18:08:23 | jmccarthy | lyarwood: It's expected to return 1 I mean is it ? | |
| 18:08:36 | jmccarthy | lyarwood: Ok let me try and attach it again | |
| 18:08:55 | lyarwood | jmccarthy: yeah I think so, 0 is the device is encrypted already, 1 if it isn't | |
| 18:09:22 | mriedem | mdbooth: https://review.openstack.org/#/c/528362/14/nova/tests/unit/virt/test_block_device.py@283 | |
| 18:09:27 | mriedem | am i missing something? | |
| 18:09:32 | jmccarthy | lyarwood: Ok my bad, I assumed that was the start of where my attach failed | |
| 18:10:01 | jmccarthy | lyarwood: I'll look at logs more, it's going wrong but I'm doing a bad job at figuring why lol | |
| 18:10:35 | lyarwood | jmccarthy: np, it's an odd flow tbh, feel free to throw another pastebin my way if you want a hand :) | |
| 18:11:41 | jmccarthy | lyarwood: Ok thanks ! Will do :) | |
| 18:13:57 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add missing encryption_secret_uuid tests https://review.openstack.org/540679 | |
| 18:15:27 | jmccarthy | lyarwood: Maybe this is where I'm going wrong ? https://paste.fedoraproject.org/paste/oSRFKMBJiuCEifxfLXUxhg the volume says it's attaching for a short time, and then just goes back to available | |
| 18:16:21 | lyarwood | jhesketh: yeah it's disconnecting after the libvirt failure to attach the disk to the domain | |
| 18:18:48 | lyarwood | jmccarthy: can you pastebin the entire req-138f35c8-ed21-4430-bf68-525574657926 flow ? | |
| 18:19:15 | jmccarthy | lyarwood: Sure - let me get that together | |
| 18:24:31 | jmccarthy | lyarwood: I think this is all of it ? https://paste.fedoraproject.org/paste/6sdkA90XTlkpftZL5j5D~w | |
| 18:28:34 | lyarwood | jmccarthy: odd, which version of Nova is this? | |
| 18:29:04 | jmccarthy | lyarwood: This is a kolla deploy, queens | |
| 18:29:07 | lyarwood | attach device xml: <disk type="block" device="disk"> is wrong | |
| 18:30:04 | jmccarthy | lyarwood: Ok let me check that a bit more - I do have one setup that is working and one that isn't - I just can't see wtf is going on | |
| 18:30:40 | jmccarthy | lyarwood: Great - Ok, I'll go try and see what is the deal with that - thanks ! | |
| 18:32:02 | lyarwood | jmccarthy: can you just double check that line and make sure it wasn't cut off in your terminal etc | |
| 18:32:22 | jmccarthy | lyarwood: One sec | |
| 18:34:16 | jmccarthy | lyarwood: Ok sorry - part of that bit was missed - this ? https://paste.fedoraproject.org/paste/jR3KP6NlhvTjQZrTl~yK~w | |
| 18:34:47 | lyarwood | jmccarthy: yeah that's better | |
| 18:34:53 | lyarwood | jmccarthy: okay so the XML looks good | |
| 18:35:27 | lyarwood | jmccarthy: are you using fixed_key in nova.conf? | |
| 18:36:07 | jmccarthy | lyarwood: Nope - should I be ? I'm going by this here: https://docs.openstack.org/cinder/queens/configuration/block-storage/volume-encryption.html | |
| 18:36:42 | jmccarthy | lyarwood: I'm pretty sure they are both configured the same, but setup1 works and setup2 doesn't - I have the logs I'm just going crosseyed with where it's going wrong | |
| 18:37:05 | lyarwood | jmccarthy: is it the same key in both envs? | |
| 18:37:14 | lyarwood | jmccarthy: and is the key the same in nova.conf and cinder.conf? | |
| 18:37:33 | lyarwood | ah sorry that doc is using barbican | |
| 18:38:11 | jmccarthy | lyarwood: the setups are not connected to each other at all as such, just in the same lab | |
| 18:38:29 | jmccarthy | lyarwood: nothing is shared | |
| 18:38:55 | lyarwood | jmccarthy: kk can you do a cinder show of the volume? | |
| 18:40:07 | jmccarthy | lyarwood: Should be this here: https://paste.fedoraproject.org/paste/SQ5quYSt8wKOV~-eTqLNMA | |
| 18:40:41 | jmccarthy | lyarwood: I have gone through this a few times, I think the logs aligh with where it's at atm | |
| 18:40:56 | jmccarthy | lyarwood: * s/aligh/align | |
| 18:42:54 | jmccarthy | lyarwood: maybe the log I sent earlier doesn't make senses .. it was grepped on req-138f35c8-ed21-4430-bf68-525574657926 | |
| 18:43:04 | openstack | bugzilla.redhat.com bug 1447297 in libvirt "Unable to use LUKS passphrase that is exactly 16 bytes long" [Unspecified,Closed: errata] - Assigned to pkrempa | |
| 18:43:04 | lyarwood | jmccarthy: yeah there's an underlying libvirt issue here with the attach, I'm just trying to rule out some older known issues with passphrase length etc, for example https://bugzilla.redhat.com/show_bug.cgi?id=1447297 | |
| 18:44:22 | lyarwood | jmccarthy: final ask, `openstack volume type show luks` just to confirm the key_size | |
| 18:44:33 | lyarwood | ^ I think that's the correct command at least | |
| 18:45:02 | lyarwood | https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/volume-type.html#volume-type-show yeah it is | |
| 18:45:32 | lyarwood | openstack volume type show --encryption-type luks | |