Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-20
16:45:00 kashyap lyarwood: Err, let me try that
16:45:30 lyarwood kashyap: as listed here - https://git.openstack.org/cgit/openstack/nova
16:47:52 kashyap lyarwood: No luck for me; very bizarre; yesterday it worked briefly, but not anymore
16:47:53 dansmith efried: around?
16:47:53 efried dansmith: yeaux
16:47:53 efried Am I reading backscroll or...?
16:47:53 dansmith efried: trying to write a functional test that talks to placement and do some aggregate stuff
16:47:53 kashyap lyarwood: Both (.git or otherwise) worked for me in the past. Not anymore. /me digs further
16:47:53 dansmith efried: and the reportclient I have is complaining that it doesn't know about a provider, which I think might be a local cache thing and I need to refresh its view or something?
16:47:53 efried dansmith: Which module are you writing your test in?
16:47:53 dansmith but the provider definitely exists
16:47:53 efried dansmith: Wanna post what you've got?
16:48:58 dansmith efried: see if this is enough: https://pastebin.com/ekYNZJWR
16:49:15 dansmith I did a raw get against placement and the provider uuid I'm using is definitely there
16:49:46 efried dansmith: Yeah, set_aggregates_for_provider does require that you've loaded up the provider previously. I think it says so in the docstring. Stand by...
16:50:08 efried oh, it says the provider must exist. Which isn't the same at all.
16:50:08 dansmith I don't think it does
16:50:10 dansmith yeah
16:50:31 dansmith presumably I need a .refresh() or some such?
16:50:36 efried dansmith: Well, the PUT ought to be working.
16:50:43 dansmith t'aint
16:50:43 efried cause that's definitely not relying on the cache.
16:51:14 efried yeah, 'tis. It's the cache update that ain't.
16:51:31 efried provider_tree.update_aggregates is what's complaining about the provider not being found.
16:51:58 dansmith you mean the aggregate association completed and it's just trying to refresh its view and failing?
16:52:03 efried Correct.
16:52:10 dansmith okay, what I meant is "this call is failing"
16:52:12 dansmith ;)
16:52:22 efried If you put an _ensure_resource_provider before your set_aggs, it'll fix it.
16:52:34 dansmith okay, but that's a bug yes?
16:52:51 efried Thinking...
16:53:05 melwitt bauzas: runways draft, please add feedback if you have any, want to kick off using it later this week https://etherpad.openstack.org/p/nova-runways-rocky
16:53:52 dansmith efried: I think the deal here is that this makes report client unable to be generally useful to manage things, as it seems opinionated about only knowing about one provider
16:53:53 efried dansmith: It's not unreasonable for you to expect to be able to use that method as you are. So yeah, I think it's a bug. But tbh I don't know what the right fix would be.
16:53:57 dansmith where it used to work well
16:54:07 openstackgerrit Mathieu Gagné proposed openstack/nova master: Fix rebuild of baremetal instance when vm_state is ERROR https://review.openstack.org/523559
16:54:38 dansmith efried: I shall stick a #FIXME(efried) in my test
16:54:56 efried dansmith: wfm. Or even open a bug.
16:55:09 dansmith well, just this for the moment
16:55:28 efried I'd like to fix this on top of the upt stack, cause I want to take advantage of the move of the cache refresh timer business.
16:55:50 dansmith I can muck with internals in my test, but for other uses it'd be more of a wart
16:57:38 efried dansmith: I think the answer is going to be to trap such errors and invalidate the cache for that RP. Though after the aforementioned "move of the cache refresh timer business", the cache is already invalidated in this case. So... we'll just wind up ignoring that exception, pretty much. But there's similar code elsewhere. Quite a bit of it, I fear.
16:58:04 dansmith efried: why wouldn't we just _ensure_resource_provider(uuid) in the set_aggregates call?
16:58:47 efried dansmith: That would *usually* be an extraneous REST call. Which I don't like (but cdent does).
16:58:54 sean-k-mooney dansmith: currently we cant remove all the old vif plugging stuff yet. but i think we could get there in rocky
16:59:15 dansmith efried: ah I figured it wouldn't do so if it's already in our known tree
16:59:27 dansmith sean-k-mooney: ack, I got caught up by the others, thanks
16:59:35 efried dansmith: That may be true, looking...
17:00:51 efried dansmith: Well, you're correct that it wouldn't retrieve it if it's already in the cache. But it does call _refresh_associations.
17:01:06 dansmith which we're about to do after the put?
17:01:36 efried no, where?
17:01:37 sean-k-mooney dansmith: ok cool, we have been trying to remove the old methods as they get converted to use os-vif. eventully they will all go.
17:01:53 efried dansmith: I mean, if you were gonna do that via some other call after set_aggregates_for_provider, that's different.
17:03:40 dansmith efried: we're refreshing associations after the put, which is failing now, right? or is that something else?
17:03:50 dansmith efried: anyway, it's your call on how to fix it, but I think this needs to be a thing you can do
17:03:53 efried No, that's just updating the cache
17:03:57 efried which is local and fast.
17:04:05 dansmith okay, from the result?
17:04:10 efried dansmith: Anyway, the reason I wouldn't always want to do that _ensure within this method is because it will go do a potential buttload of placement calls to populate the report client's cache. But if ALL you cared about was updating agg associations, and you were never going to use any of that other stuff, all of that would be wasted.
17:04:14 efried yes
17:04:20 dansmith wouldn't you want the refresh of the associations first anyway so the cache ends up consistent with the server?
17:04:55 efried Well, yes, actually, once I get around to it, I'll need to make this guy take generation into account.
17:05:13 efried At which point you'll HAVE to have previously retrieved the RP, cause that's where you'll get the generation from.
17:06:26 efried But I agree that a public method such as this one ought to be able to be called without you having to know you needed to populate the cache beforehand.
17:06:48 efried So yeah, I'll need to change it (and its brethren) to _ensure/_refresh.
17:07:02 efried Bummer.
17:08:24 dansmith ack
17:10:23 melwitt has anyone seen this error before "EndpointNotFound: Could not find requested endpoint for any of the following interfaces: ['internal', 'public']" when we try to send a notification?
17:11:24 gibi melwitt: I did see this before
17:12:38 mriedem mnaser: figured it out; it's only a problem for resizing an instance that has multiattach volumes attached; i don't know why the newer libvirt package versions have anything to do with it, because it looks like a latent bug in how we actually handle resize + multiattach
17:12:56 melwitt gibi: do you know if it's a problem on our end or is it a problem with a deployment?
17:13:04 gibi melwitt: is this EndpointNotFound a keystone exception?
17:13:26 melwitt gibi: here's the full trace https://bugs.launchpad.net/nova/+bug/1753550/comments/3
17:13:27 openstack Launchpad bug 1753550 in OpenStack Compute (nova) "Status does not update to "Shutoff" when instance shuts down itself" [Undecided,Incomplete]
17:13:34 mriedem melwitt: for the given endpoint, which interfaces are available for it in the service catalog
17:13:55 mriedem the new ksa code will try to use internal and public by default if you don't specify a specific interface in config, like 'admin'
17:14:19 mriedem it's trying to hit glance
17:14:31 melwitt okay. they said they have services listed under 'public' when they do a 'openstack endpoint list'
17:14:34 mriedem so presumably they have glance configured in nova on an admin interface but don't have it configured that way
17:14:46 melwitt oh, hm
17:15:27 mriedem https://docs.openstack.org/nova/latest/configuration/config.html#glance
17:15:30 mriedem see 'valid_interfaces'
17:15:59 melwitt woo, thanks
17:16:51 gibi mriedem, melwitt: hm, this means that sending a notification causes a REST call to glance. interesting...
17:17:07 melwitt the trace says 'internal' and 'public' though so it seems like they haven't changed that
17:17:12 mriedem File "/opt/stack/nova/nova/notifications/base.py", line 398, in info_from_instance
17:17:12 mriedem     context)
17:17:12 mriedem   File "/opt/stack/nova/nova/image/api.py", line 65, in generate_image_url
17:17:33 melwitt gibi: yeah, that surprised me
17:17:57 gibi melwitt: me too :)
17:18:33 mriedem there was also something related to this which we fixed at the end of queens
17:18:48 gibi mriedem: this one? https://review.openstack.org/#/c/511397/14/nova/notifications/base.py
17:18:50 mriedem https://github.com/openstack/nova/commit/62ef6cfcf01d84813f71d1e8252b86c170ee39f0
17:19:20 gibi mriedem: OK yours is different and seems more relevant
17:20:41 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384
17:20:42 mriedem they likely need to update nova, or restack their devstack env, and try again
17:22:59 melwitt the commit id they pulled is from march 4 though, so they should have that fix
17:23:30 openstackgerrit Merged openstack/nova master: Remove old flavor_extra_specs_get db api method https://review.openstack.org/539706
17:24:33 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Correct documentation for '[pci] passthrough_whitelist' https://review.openstack.org/552874
17:26:11 mriedem ildikov: a real hum dinger https://bugs.launchpad.net/nova/+bug/1757190

Earlier   Later