Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-20
17:04:10 efried dansmith: Anyway, the reason I wouldn't always want to do that _ensure within this method is because it will go do a potential buttload of placement calls to populate the report client's cache. But if ALL you cared about was updating agg associations, and you were never going to use any of that other stuff, all of that would be wasted.
17:04:14 efried yes
17:04:20 dansmith wouldn't you want the refresh of the associations first anyway so the cache ends up consistent with the server?
17:04:55 efried Well, yes, actually, once I get around to it, I'll need to make this guy take generation into account.
17:05:13 efried At which point you'll HAVE to have previously retrieved the RP, cause that's where you'll get the generation from.
17:06:26 efried But I agree that a public method such as this one ought to be able to be called without you having to know you needed to populate the cache beforehand.
17:06:48 efried So yeah, I'll need to change it (and its brethren) to _ensure/_refresh.
17:07:02 efried Bummer.
17:08:24 dansmith ack
17:10:23 melwitt has anyone seen this error before "EndpointNotFound: Could not find requested endpoint for any of the following interfaces: ['internal', 'public']" when we try to send a notification?
17:11:24 gibi melwitt: I did see this before
17:12:38 mriedem mnaser: figured it out; it's only a problem for resizing an instance that has multiattach volumes attached; i don't know why the newer libvirt package versions have anything to do with it, because it looks like a latent bug in how we actually handle resize + multiattach
17:12:56 melwitt gibi: do you know if it's a problem on our end or is it a problem with a deployment?
17:13:04 gibi melwitt: is this EndpointNotFound a keystone exception?
17:13:26 melwitt gibi: here's the full trace https://bugs.launchpad.net/nova/+bug/1753550/comments/3
17:13:27 openstack Launchpad bug 1753550 in OpenStack Compute (nova) "Status does not update to "Shutoff" when instance shuts down itself" [Undecided,Incomplete]
17:13:34 mriedem melwitt: for the given endpoint, which interfaces are available for it in the service catalog
17:13:55 mriedem the new ksa code will try to use internal and public by default if you don't specify a specific interface in config, like 'admin'
17:14:19 mriedem it's trying to hit glance
17:14:31 melwitt okay. they said they have services listed under 'public' when they do a 'openstack endpoint list'
17:14:34 mriedem so presumably they have glance configured in nova on an admin interface but don't have it configured that way
17:14:46 melwitt oh, hm
17:15:27 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#glance
17:15:30 mriedem see 'valid_interfaces'
17:15:59 melwitt woo, thanks
17:16:51 gibi mriedem, melwitt: hm, this means that sending a notification causes a REST call to glance. interesting...
17:17:07 melwitt the trace says 'internal' and 'public' though so it seems like they haven't changed that
17:17:12 mriedem   File "/opt/stack/nova/nova/image/api.py", line 65, in generate_image_url
17:17:12 mriedem     context)
17:17:12 mriedem File "/opt/stack/nova/nova/notifications/base.py", line 398, in info_from_instance
17:17:33 melwitt gibi: yeah, that surprised me
17:17:57 gibi melwitt: me too :)
17:18:33 mriedem there was also something related to this which we fixed at the end of queens
17:18:48 gibi mriedem: this one? https://review.openstack.org/#/c/511397/14/nova/notifications/base.py
17:18:50 mriedem https://github.com/openstack/nova/commit/62ef6cfcf01d84813f71d1e8252b86c170ee39f0
17:19:20 gibi mriedem: OK yours is different and seems more relevant
17:20:41 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384
17:20:42 mriedem they likely need to update nova, or restack their devstack env, and try again
17:22:59 melwitt the commit id they pulled is from march 4 though, so they should have that fix
17:23:30 openstackgerrit Merged openstack/nova master: Remove old flavor_extra_specs_get db api method https://review.openstack.org/539706
17:24:33 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Correct documentation for '[pci] passthrough_whitelist' https://review.openstack.org/552874
17:26:11 mriedem ildikov: a real hum dinger https://bugs.launchpad.net/nova/+bug/1757190
17:26:13 openstack Launchpad bug 1757190 in OpenStack Compute (nova) "resize fails with volume multiattach using with libvirt 4.0.0 (and qemu 2.11.1): Failed to get shared "write" lock" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem)
17:26:14 mriedem will fix after lunch
17:26:32 mriedem from what i can tell, it has nothing to do with the actual libvirt/qemu versions, except maybe older libvirt/qemu were masking a bug in nova
17:27:00 mriedem during resize we blow away our special 'multiattach' flag in the connection_info here https://github.com/openstack/nova/blob/f80b4e50093002f84b43ff245a605fbe44d34711/nova/virt/block_device.py#L639
17:27:02 mriedem derp
17:28:00 ildikov mriedem: nasty... :/
17:28:53 ildikov mriedem: I guess it gives us another reason to find a better way to pass the multiattach info
17:30:16 gibi melwitt, mriedem: we call glance API from the notification sending due to the else branch here https://github.com/openstack/nova/blob/24379f1822e3ae1d4f7c8398e60af6e52b386c32/nova/image/glance.py#L120
17:30:45 gibi melwitt,mriedem: s/glance/keystone/
17:31:48 gibi melwitt, mriedem: and that ksa code was introduced here https://review.openstack.org/#/c/488137/23/nova/image/glance.py
17:33:18 melwitt okay, yeah, and we're failing on L126 on endpoint = utils.get_endpoint(ksa_adap)
17:33:19 gibi melwitt, mriedem: before that ^^ the notification url generation only depened on the config params but after that it can fall back to keystone
17:33:51 openstackgerrit Surya Seetharaman proposed openstack/nova master: Add CellMappingList.get_enabled_or_disabled(disabled=False) query method https://review.openstack.org/550188
17:33:52 openstackgerrit Surya Seetharaman proposed openstack/nova master: Allow scheduling only to enabled cells (Filter Scheduler) https://review.openstack.org/550527
17:34:00 gibi melwitt: I think so yes
17:34:43 melwitt I meant that's what's in the trace. so that fails before we ever get to the code that will try to strip the version from the url
17:35:41 gibi melwitt: this also means that you can workaround the problem by setting CONF.glance.api_servers in the nova.conf
17:35:59 openstackgerrit Merged openstack/nova master: Remove old flavor_access_remove db api method https://review.openstack.org/539709
17:37:50 melwitt yeah, true. on the surface, it seems like there must be something wrong with their service catalog, because this is a straight call to keystone, or so it seems
17:38:13 openstackgerrit Merged openstack/nova master: Remove old flavor_access_add db api methods https://review.openstack.org/539714
17:38:56 gibi melwitt: yes, hence my classification of this change as only a workaround and not a real solution
17:39:15 gibi mriedem, melwitt: I have to leave now, but I will read back tomorrow to see if we have to do someting about not talking to keyston from the notification sending codepath
17:39:32 melwitt thanks gibi
17:48:14 edmondsw melwitt there are a ton of unapproved specs... is there any focus on reviewing those and knocking down that queue?
17:48:24 edmondsw I know the powervm spec has been sitting without comment since Feb
17:57:10 melwitt edmondsw: I think several people have been reviewing specs. I've been busy with PTG summary writeups and getting the runways proposal going. spec freeze is Apr 19 so I'm going to send email this week to get input on a spec review day date where everyone will focus on spec reviews
17:58:56 edmondsw melwitt I know a bunch of spec reviews are going on. Just wondering if there is a concerted focus on burning down the queue. Sounds like you're thinking about that
17:59:04 edmondsw kinda a prereq for the runways stuff
17:59:09 edmondsw gotta get the spec approved first :)
18:00:04 dansmith edleafe: around?
18:01:33 melwitt yeah. as usual, the beginning of the cycle people are reviewing specs more because as you said, have to approve things before we can focus on reviewing the implementations. I think people are already doing that. last cycle we had a 79% approved spec/bp completion percentage so I think this cycle we're probably going to approve fewer things to increase that percentage significantly
18:02:19 melwitt like last cycle, we'll have a dedicated spec review day to burn through a lot of them before spec freeze. that's what I'm going to send email about this week so we can pick a date that works for most
18:02:59 edleafe dansmith: yeah - somewhat distracted atm
18:03:41 dansmith edleafe: okay, I'm having trouble getting member_of to work and there's kindof a missing case in your tests, which I thought maybe was covering up a bug, but I tweaked it and it still passes, so .. I'm still digging
18:04:01 dansmith edleafe: but, I've associated a provider with an aggregate, but when I member_of it, I get back no candidates
18:04:36 edleafe dansmith: that's... odd
18:04:47 edmondsw melwitt just a little anxious not seeing reviews on the powervm spec. I'd have thought it was a fairly easy review. I believe the nova team is already committed to this effort and it's just a matter of how much or how little we bite off per release, not whether. I'd like to get the rocky content locked down.
18:05:08 edleafe dansmith: the gabbi tests show that exact case
18:05:31 edmondsw we've been putting up commits for review, but until the spec is approved hard to ask anyone to look at functional commits
18:05:33 melwitt edmondsw: has it been previously approved? I don't know the history about it
18:05:49 edmondsw melwitt yes, and the history is all in the spec at the end
18:05:55 dansmith edleafe: yeah, I know, that's why I'm trying to diff what I'm doing with those
18:06:02 edmondsw melwitt it's a multirelease effort
18:06:21 edmondsw melwitt since it was considered too much to bite off in one release from a reviewer bandwidth perspective
18:06:25 dansmith edleafe: you don't have a case where you only have one thing in member_of that actually returns results, but I shaved one down and it still seems to work (although it's hard to know if it's returning the thing we expect
18:06:48 edmondsw melwitt https://review.openstack.org/#/c/545111/
18:08:08 artom So, do we handle *any* NUMA stuff during live migration? I know hugepages and CPU pinning aren't, but what about plain old NUMA topology?
18:09:03 openstackgerrit Merged openstack/nova master: Remove old flavor_access_get_by_flavor_id db api method https://review.openstack.org/539720
18:09:04 melwitt edmondsw: okay, probably just needs some reminding then. looks like mriedem has reviewed it last month so maybe he can take another look at it soon
18:09:21 edleafe dansmith: you could inspect the returned a-c, but that's always too complicated to write in gabbi
18:09:46 dansmith edleafe: yeah, I'm doing that in my real test and getting back [], but I dunno how to do much with the gabbit
18:09:47 cdent edleafe: untrue, either make the test verbose: True or make it fail intentionally in the response_json_paths
18:09:54 openstackgerrit Merged openstack/nova master: Remove old flavor_destroy db api method https://review.openstack.org/539729
18:10:00 dansmith but it's checking the length of allocation_requests
18:10:04 cdent and the error message will show the full response
18:10:23 openstackgerrit Merged openstack/nova master: Remove old flavor_get_by_flavor_id db api method https://review.openstack.org/539733
18:10:35 openstackgerrit Merged openstack/nova master: Remove old flavor_get_by_name db api method https://review.openstack.org/544379
18:10:36 dansmith cdent: ah, duh, trying

Earlier   Later