Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-20
16:54:07 openstackgerrit Mathieu Gagné proposed openstack/nova master: Fix rebuild of baremetal instance when vm_state is ERROR https://review.openstack.org/523559
16:54:38 dansmith efried: I shall stick a #FIXME(efried) in my test
16:54:56 efried dansmith: wfm. Or even open a bug.
16:55:09 dansmith well, just this for the moment
16:55:28 efried I'd like to fix this on top of the upt stack, cause I want to take advantage of the move of the cache refresh timer business.
16:55:50 dansmith I can muck with internals in my test, but for other uses it'd be more of a wart
16:57:38 efried dansmith: I think the answer is going to be to trap such errors and invalidate the cache for that RP. Though after the aforementioned "move of the cache refresh timer business", the cache is already invalidated in this case. So... we'll just wind up ignoring that exception, pretty much. But there's similar code elsewhere. Quite a bit of it, I fear.
16:58:04 dansmith efried: why wouldn't we just _ensure_resource_provider(uuid) in the set_aggregates call?
16:58:47 efried dansmith: That would *usually* be an extraneous REST call. Which I don't like (but cdent does).
16:58:54 sean-k-mooney dansmith: currently we cant remove all the old vif plugging stuff yet. but i think we could get there in rocky
16:59:15 dansmith efried: ah I figured it wouldn't do so if it's already in our known tree
16:59:27 dansmith sean-k-mooney: ack, I got caught up by the others, thanks
16:59:35 efried dansmith: That may be true, looking...
17:00:51 efried dansmith: Well, you're correct that it wouldn't retrieve it if it's already in the cache. But it does call _refresh_associations.
17:01:06 dansmith which we're about to do after the put?
17:01:36 efried no, where?
17:01:37 sean-k-mooney dansmith: ok cool, we have been trying to remove the old methods as they get converted to use os-vif. eventully they will all go.
17:01:53 efried dansmith: I mean, if you were gonna do that via some other call after set_aggregates_for_provider, that's different.
17:03:40 dansmith efried: we're refreshing associations after the put, which is failing now, right? or is that something else?
17:03:50 dansmith efried: anyway, it's your call on how to fix it, but I think this needs to be a thing you can do
17:03:53 efried No, that's just updating the cache
17:03:57 efried which is local and fast.
17:04:05 dansmith okay, from the result?
17:04:10 efried dansmith: Anyway, the reason I wouldn't always want to do that _ensure within this method is because it will go do a potential buttload of placement calls to populate the report client's cache. But if ALL you cared about was updating agg associations, and you were never going to use any of that other stuff, all of that would be wasted.
17:04:14 efried yes
17:04:20 dansmith wouldn't you want the refresh of the associations first anyway so the cache ends up consistent with the server?
17:04:55 efried Well, yes, actually, once I get around to it, I'll need to make this guy take generation into account.
17:05:13 efried At which point you'll HAVE to have previously retrieved the RP, cause that's where you'll get the generation from.
17:06:26 efried But I agree that a public method such as this one ought to be able to be called without you having to know you needed to populate the cache beforehand.
17:06:48 efried So yeah, I'll need to change it (and its brethren) to _ensure/_refresh.
17:07:02 efried Bummer.
17:08:24 dansmith ack
17:10:23 melwitt has anyone seen this error before "EndpointNotFound: Could not find requested endpoint for any of the following interfaces: ['internal', 'public']" when we try to send a notification?
17:11:24 gibi melwitt: I did see this before
17:12:38 mriedem mnaser: figured it out; it's only a problem for resizing an instance that has multiattach volumes attached; i don't know why the newer libvirt package versions have anything to do with it, because it looks like a latent bug in how we actually handle resize + multiattach
17:12:56 melwitt gibi: do you know if it's a problem on our end or is it a problem with a deployment?
17:13:04 gibi melwitt: is this EndpointNotFound a keystone exception?
17:13:26 melwitt gibi: here's the full trace https://bugs.launchpad.net/nova/+bug/1753550/comments/3
17:13:27 openstack Launchpad bug 1753550 in OpenStack Compute (nova) "Status does not update to "Shutoff" when instance shuts down itself" [Undecided,Incomplete]
17:13:34 mriedem melwitt: for the given endpoint, which interfaces are available for it in the service catalog
17:13:55 mriedem the new ksa code will try to use internal and public by default if you don't specify a specific interface in config, like 'admin'
17:14:19 mriedem it's trying to hit glance
17:14:31 melwitt okay. they said they have services listed under 'public' when they do a 'openstack endpoint list'
17:14:34 mriedem so presumably they have glance configured in nova on an admin interface but don't have it configured that way
17:14:46 melwitt oh, hm
17:15:27 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#glance
17:15:30 mriedem see 'valid_interfaces'
17:15:59 melwitt woo, thanks
17:16:51 gibi mriedem, melwitt: hm, this means that sending a notification causes a REST call to glance. interesting...
17:17:07 melwitt the trace says 'internal' and 'public' though so it seems like they haven't changed that
17:17:12 mriedem   File "/opt/stack/nova/nova/image/api.py", line 65, in generate_image_url
17:17:12 mriedem     context)
17:17:12 mriedem File "/opt/stack/nova/nova/notifications/base.py", line 398, in info_from_instance
17:17:33 melwitt gibi: yeah, that surprised me
17:17:57 gibi melwitt: me too :)
17:18:33 mriedem there was also something related to this which we fixed at the end of queens
17:18:48 gibi mriedem: this one? https://review.openstack.org/#/c/511397/14/nova/notifications/base.py
17:18:50 mriedem https://github.com/openstack/nova/commit/62ef6cfcf01d84813f71d1e8252b86c170ee39f0
17:19:20 gibi mriedem: OK yours is different and seems more relevant
17:20:41 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384
17:20:42 mriedem they likely need to update nova, or restack their devstack env, and try again
17:22:59 melwitt the commit id they pulled is from march 4 though, so they should have that fix
17:23:30 openstackgerrit Merged openstack/nova master: Remove old flavor_extra_specs_get db api method https://review.openstack.org/539706
17:24:33 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Correct documentation for '[pci] passthrough_whitelist' https://review.openstack.org/552874
17:26:11 mriedem ildikov: a real hum dinger https://bugs.launchpad.net/nova/+bug/1757190
17:26:13 openstack Launchpad bug 1757190 in OpenStack Compute (nova) "resize fails with volume multiattach using with libvirt 4.0.0 (and qemu 2.11.1): Failed to get shared "write" lock" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem)
17:26:14 mriedem will fix after lunch
17:26:32 mriedem from what i can tell, it has nothing to do with the actual libvirt/qemu versions, except maybe older libvirt/qemu were masking a bug in nova
17:27:00 mriedem during resize we blow away our special 'multiattach' flag in the connection_info here https://github.com/openstack/nova/blob/f80b4e50093002f84b43ff245a605fbe44d34711/nova/virt/block_device.py#L639
17:27:02 mriedem derp
17:28:00 ildikov mriedem: nasty... :/
17:28:53 ildikov mriedem: I guess it gives us another reason to find a better way to pass the multiattach info
17:30:16 gibi melwitt, mriedem: we call glance API from the notification sending due to the else branch here https://github.com/openstack/nova/blob/24379f1822e3ae1d4f7c8398e60af6e52b386c32/nova/image/glance.py#L120
17:30:45 gibi melwitt,mriedem: s/glance/keystone/
17:31:48 gibi melwitt, mriedem: and that ksa code was introduced here https://review.openstack.org/#/c/488137/23/nova/image/glance.py
17:33:18 melwitt okay, yeah, and we're failing on L126 on endpoint = utils.get_endpoint(ksa_adap)
17:33:19 gibi melwitt, mriedem: before that ^^ the notification url generation only depened on the config params but after that it can fall back to keystone
17:33:51 openstackgerrit Surya Seetharaman proposed openstack/nova master: Add CellMappingList.get_enabled_or_disabled(disabled=False) query method https://review.openstack.org/550188
17:33:52 openstackgerrit Surya Seetharaman proposed openstack/nova master: Allow scheduling only to enabled cells (Filter Scheduler) https://review.openstack.org/550527
17:34:00 gibi melwitt: I think so yes
17:34:43 melwitt I meant that's what's in the trace. so that fails before we ever get to the code that will try to strip the version from the url
17:35:41 gibi melwitt: this also means that you can workaround the problem by setting CONF.glance.api_servers in the nova.conf
17:35:59 openstackgerrit Merged openstack/nova master: Remove old flavor_access_remove db api method https://review.openstack.org/539709
17:37:50 melwitt yeah, true. on the surface, it seems like there must be something wrong with their service catalog, because this is a straight call to keystone, or so it seems
17:38:13 openstackgerrit Merged openstack/nova master: Remove old flavor_access_add db api methods https://review.openstack.org/539714
17:38:56 gibi melwitt: yes, hence my classification of this change as only a workaround and not a real solution
17:39:15 gibi mriedem, melwitt: I have to leave now, but I will read back tomorrow to see if we have to do someting about not talking to keyston from the notification sending codepath
17:39:32 melwitt thanks gibi
17:48:14 edmondsw melwitt there are a ton of unapproved specs... is there any focus on reviewing those and knocking down that queue?
17:48:24 edmondsw I know the powervm spec has been sitting without comment since Feb
17:57:10 melwitt edmondsw: I think several people have been reviewing specs. I've been busy with PTG summary writeups and getting the runways proposal going. spec freeze is Apr 19 so I'm going to send email this week to get input on a spec review day date where everyone will focus on spec reviews
17:58:56 edmondsw melwitt I know a bunch of spec reviews are going on. Just wondering if there is a concerted focus on burning down the queue. Sounds like you're thinking about that
17:59:04 edmondsw kinda a prereq for the runways stuff
17:59:09 edmondsw gotta get the spec approved first :)
18:00:04 dansmith edleafe: around?
18:01:33 melwitt yeah. as usual, the beginning of the cycle people are reviewing specs more because as you said, have to approve things before we can focus on reviewing the implementations. I think people are already doing that. last cycle we had a 79% approved spec/bp completion percentage so I think this cycle we're probably going to approve fewer things to increase that percentage significantly
18:02:19 melwitt like last cycle, we'll have a dedicated spec review day to burn through a lot of them before spec freeze. that's what I'm going to send email about this week so we can pick a date that works for most
18:02:59 edleafe dansmith: yeah - somewhat distracted atm
18:03:41 dansmith edleafe: okay, I'm having trouble getting member_of to work and there's kindof a missing case in your tests, which I thought maybe was covering up a bug, but I tweaked it and it still passes, so .. I'm still digging
18:04:01 dansmith edleafe: but, I've associated a provider with an aggregate, but when I member_of it, I get back no candidates

Earlier   Later