Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-09
15:33:26 openstackgerrit Dan Smith proposed openstack/nova master: Make get_instance_objects_sorted() be smart about cells https://review.openstack.org/509003
15:34:08 mriedem stephenfin: left some comments, the commit message clearly needs more context
15:34:23 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: Skip placement on rebuild in same host https://review.openstack.org/561015
15:42:50 openstackgerrit Merged openstack/nova stable/queens: Fix dropped check for boot_index 0 in _validate_bdm https://review.openstack.org/564461
15:43:36 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove '_apply_instance_name_template' https://review.openstack.org/567257
15:43:37 openstackgerrit Stephen Finucane proposed openstack/nova master: Simplify instance name generation https://review.openstack.org/516573
15:52:24 openstackgerrit Balazs Gibizer proposed openstack/nova master: add resource_requests field to RequestSpec https://review.openstack.org/567267
15:52:25 openstackgerrit Balazs Gibizer proposed openstack/nova master: transfer port.resource_request to the scheduler https://review.openstack.org/567268
15:52:25 stephenfin Anyone have any idea why this change could be causing Tempest to fail? https://review.openstack.org/#/c/564444/ http://logs.openstack.org/44/564444/2/check/tempest-full/762ada1/testr_results.html.gz
15:53:56 stephenfin The only functional thing I've changed is that we now fetch physnet information for any network IDs we passed in network requests - not just port IDs. Can't reproduce locally though :/
15:55:10 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: log stale allocations as WARNING instead of DEBUG https://review.openstack.org/567271
15:55:56 openstackgerrit Artom Lifshitz proposed openstack/nova stable/queens: log stale allocations as WARNING instead of DEBUG https://review.openstack.org/567273
15:56:45 openstackgerrit Artom Lifshitz proposed openstack/nova stable/pike: log stale allocations as WARNING instead of DEBUG https://review.openstack.org/567271
15:57:22 gibi stephenfin: now you include the meta network 'auto' to the query http://logs.openstack.org/44/564444/2/check/tempest-full/762ada1/controller/logs/screen-n-api.txt.gz?level=ERROR#_May_07_16_24_34_423219
15:59:15 stephenfin gibi: Dang it. Thanks! I've been trying to suss that all day and couldn't find the required log 😅
15:59:22 lbragstad jaypipes: re: the scope type patches for nova, i reworked the ksm (gating) and the oslo.context patches (still in review) - hopefully they're a bit more clear from a consuming service perspective
15:59:30 lbragstad oslo.context patch - https://review.openstack.org/#/c/530509/
16:00:02 mdbooth dansmith: I think the explanation you requested is better in a class docstring rather than the commit message. That ok with you?
16:00:18 mdbooth Means we can find it later without git spelunking.
16:00:47 dansmith sure
16:01:30 mriedem stephenfin: the NetworkRequest(List) has helper methods to determine if it's an auto-allocate request
16:05:07 openstack bugzilla.redhat.com bug 1572836 in openstack-nova "nova-compute should log messages about stale resource allocations at warning priority" [Medium,Post] - Assigned to alifshit
16:05:07 openstackgerrit Merged openstack/nova master: Followup for multiple member_of qparams support https://review.openstack.org/567209
16:05:07 artom mriedem, to be honest I just mechanically did the backport since our downstream bug was against 12 (https://bugzilla.redhat.com/show_bug.cgi?id=1572836 FWIW)
16:05:50 mriedem artom: see my comments when i merged the change on master
16:06:18 mriedem artom: queens and rocky shouldn't be hitting that code so warning is ok, and we can't rely on ocata/pike computes to heal the allocations,
16:06:41 mriedem but that gets fuzzy when you're in pike and could have ocata computes, and we also fixed a bunch of places post-pike GA that didn't cleanup allocations on failure,
16:06:51 mriedem so from that bz, it's possible they hit one of those places that are now fixed
16:07:27 artom mriedem, dammit, I was hoping I wouldn't have to use my brain for this one ;)
16:07:36 mriedem defer your brain to dansmith
16:07:41 mriedem he knows that code well
16:08:53 artom I... I can't tell if you're serious or just messing with Dan :/
16:11:38 openstackgerrit Artom Lifshitz proposed openstack/nova master: Do not use SameHostFilter in API sample tests https://review.openstack.org/563037
16:12:45 mriedem artom: i'm phoning a friend
16:13:00 dansmith mriedem: I think warning on pike is reasonable, and I think that's where we hit it downstream.. they didn't realize things had gotten legit out of sync because they had debug off
16:13:23 dansmith when they turned debug on, we realized some instances had allocations on three computes because of legit leaks
16:13:53 mriedem was this pike 16.0.0 GA?
16:13:58 artom dansmith, wait, was that the juicy backported you mentioned in internal IRC?
16:14:03 dansmith I think we made it debug initially because we were pretty sure we'd log that a lot
16:14:05 dansmith artom: no
16:14:18 dansmith but since we've seen it in the wild, it probably needs to be more visible
16:14:26 dansmith mriedem: it was whatever our build is based on
16:14:45 dansmith mriedem: are you asking if it was something past GA because of backports?
16:15:23 mriedem we fixed a lot of leaky allocation stuff post pike GA,
16:15:28 dansmith mriedem: the actual leakage happened some time in the past, they don't know when, but noticed it when computes were refusing to schedule the last bit of resource, which turned out to be because of some stale allocations
16:15:30 dansmith right
16:15:33 mriedem so i'm wondering if the customer hit this on 16.0.0 GA before we fixed the leaks
16:15:41 dansmith entirely possible
16:15:50 dansmith could also have been from ocata, or during/around the upgrade
16:15:56 mriedem i realize the leaks could have been from GA, even if we stopped the leaks later
16:16:02 dansmith yeah
16:16:13 dansmith it wasn't very acute, so I don't think they're just leaking like crazy right now
16:16:23 dansmith it was just "hmm, this should fit, help us figure out why not"
16:16:26 openstackgerrit Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347
16:16:27 openstackgerrit Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029
16:16:28 openstackgerrit Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root_device https://review.openstack.org/567277
16:16:28 mriedem given we should have fixed those leaks now in pike, if we do hit this, my comment from the change on master probably applies
16:16:51 mriedem i.e. we shouldn't hit this, so if we are leaking, we f'ed up and warning is probably ok
16:16:56 dansmith yes
16:27:02 mriedem artom: ok +2 on queens, +1 on pike
16:32:00 artom mriedem, dansmith, thank you gentlemen
16:39:01 openstackgerrit Dan Smith proposed openstack/nova master: Add CellMapping.get_by_project_id() query method https://review.openstack.org/509002
16:39:02 openstackgerrit Dan Smith proposed openstack/nova master: Make get_instance_objects_sorted() be smart about cells https://review.openstack.org/509003
17:01:29 mdbooth mriedem: If you get a sec could you check I haven't misrepresented you in my response to jaypipes here: https://review.openstack.org/#/c/528363/15/nova/virt/block_device.py ?
17:04:43 mriedem replied
17:05:43 mriedem i really need to be reviewing that series given my somewhat grossly intimate relationship with that code now
17:06:23 mriedem which i plan on making my afternoon
17:08:04 mriedem mdbooth: the entire goal of that series is to get a serial value in the non-volume block devices right? since we only have serial == volume id today, so for the non-volumes we'll use the bdm.uuid
17:08:30 mriedem and the 20 patch refactor series is because that's how you roll :)
17:08:58 mriedem no offense, i know this code is horrible
17:11:14 melwitt mriedem: cool, thanks for the etherpad for the stable release needs
17:14:41 openstackgerrit Jay Pipes proposed openstack/nova master: rework how we pass candidate request information https://review.openstack.org/566166
17:16:14 openstackgerrit Jay Pipes proposed openstack/nova master: rework how we pass candidate request information https://review.openstack.org/566166
17:22:01 openstackgerrit Merged openstack/nova master: Fix detach_volume calls when rolling back a failed attach https://review.openstack.org/563213
17:47:33 mriedem dansmith: lyarwood: https://review.openstack.org/#/c/567232/ should be good to go now - change on master just merged
18:00:02 jmccarthy Hmm trying to setup an encrypted volume, anyone know where does it come up with this device name ? /dev/disk/by-id/scsi-360014057e4c08a456174ed99b53aa5d6 (paste.openstack.org/show/720695/)
18:00:18 jmccarthy It all goes wrong after this, and I can't find that device anywhere ..
18:00:21 mriedem jmccarthy: os-brick library
18:02:35 jmccarthy mriedem: Hmm ok and it's making it up ? Should I be able to 'see' it someplace ?
18:02:55 mriedem lyarwood: is your guy for encrypted volume stuff in os-brick
18:03:25 jmccarthy mriedem: kk !
18:03:27 lyarwood jmccarthy: what are you trying to find, the original volume on the compute host?
18:04:07 jmccarthy lyarwood: I'm just trying to find where it's going wrong, I can attach unencrpyted, but not encrypted volumes paste.openstack.org/show/720695/
18:04:27 jmccarthy It goes wrong after here - as this device can't be found anywhere ? Or I'm looking in wrong places
18:05:54 lyarwood jmccarthy: paste.openstack.org isn't working for me, can you use https://paste.fedoraproject.org/
18:06:56 jmccarthy lyarwood: https://paste.fedoraproject.org/paste/nL0L92U4ArsPfyVkgQlX-Q
18:07:24 jmccarthy It's brief .. but in the case where it works, I'm pretty sure I find that device under /dev/disk/by-id on the compute host
18:07:37 lyarwood jmccarthy: yeah that's expected the first time you attach the volume
18:07:54 lyarwood jmccarthy: well, with the older flow where os-brick formats the volume
18:07:59 jmccarthy That's expected to return 1 is it ?
18:08:23 jmccarthy lyarwood: It's expected to return 1 I mean is it ?
18:08:36 jmccarthy lyarwood: Ok let me try and attach it again
18:08:55 lyarwood jmccarthy: yeah I think so, 0 is the device is encrypted already, 1 if it isn't
18:09:22 mriedem mdbooth: https://review.openstack.org/#/c/528362/14/nova/tests/unit/virt/test_block_device.py@283
18:09:27 mriedem am i missing something?
18:09:32 jmccarthy lyarwood: Ok my bad, I assumed that was the start of where my attach failed
18:10:01 jmccarthy lyarwood: I'll look at logs more, it's going wrong but I'm doing a bad job at figuring why lol
18:10:35 lyarwood jmccarthy: np, it's an odd flow tbh, feel free to throw another pastebin my way if you want a hand :)
18:11:41 jmccarthy lyarwood: Ok thanks ! Will do :)
18:13:57 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Add missing encryption_secret_uuid tests https://review.openstack.org/540679

Earlier   Later