| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-21 | |||
| 22:23:14 | efried | fine. | |
| 22:23:18 | mriedem | sure, so GET /os-hypervisors/detail?service_host=CONF.host | |
| 22:23:25 | mriedem | ^ is the RFE i mentioned | |
| 22:23:27 | efried | oh, that's a thing... | |
| 22:23:31 | mriedem | ha, no | |
| 22:23:33 | mriedem | oh eric | |
| 22:23:51 | mriedem | https://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-details-detail#list-hypervisors-details | |
| 22:24:14 | efried | yes, I'm looking at that. You're taking advantage of my weakened mental state to f with me. Is it April? | |
| 22:24:33 | mriedem | why is your mental state weak? | |
| 22:25:06 | efried | is it ever not? | |
| 22:25:23 | mriedem | i can't find the bug, i could have sworn a guy opened one though | |
| 22:25:52 | mriedem | but that was the idea, he was trying to filter hypervisors by ironic compute service host but was only getting nodes based on the ironic node uuid which wasn't helpful | |
| 22:25:54 | efried | okay, anyway, yes, that would be a nice way to make this work that involves an API change with a microversion. | |
| 22:26:38 | efried | but for the sake of discussion... | |
| 22:27:13 | efried | would it be so wrong for neutron/cyborg to simply use the result of `gethostname()` instead of CONF.host? | |
| 22:27:37 | efried | I guess that becomes coupled to the virt driver implementation. | |
| 22:27:59 | efried | though arguably using CONF.host at all already is. | |
| 22:28:14 | efried | in that it at least will never work for ironic | |
| 22:29:29 | mriedem | we don't care about ironic for nested providers | |
| 22:29:31 | mriedem | or most things | |
| 22:30:07 | efried | I can totally see needing to do... *something* with bandwidth for ironic. | |
| 22:30:16 | efried | Though I imagine the providers would be shared in that case, not nested. | |
| 22:30:29 | efried | nevertheless we would have to discover the ironic nodes for aggregation purposes. | |
| 22:30:31 | efried | anyway... | |
| 22:33:06 | mriedem | idk | |
| 22:33:35 | mriedem | filtering hypervisors by service host seems useful in general so if it could be used here then i don't see a reason not to add that earlier than later | |
| 22:33:55 | mriedem | always nice to get to N+3 release from now and be like, "we have this problem, oh but we added X in N so we an use that" | |
| 22:33:56 | efried | okay. Not backportable tho | |
| 22:34:00 | mriedem | nope | |
| 22:34:36 | efried | For backport purposes, I guess since the scope is known and constrained to libvirt, we could just ask neutron to use gethostname() instead of CONF.host. | |
| 22:35:05 | mriedem | again, idk... | |
| 22:35:10 | mriedem | does neutron have a concept of workarounds options? | |
| 22:35:22 | mriedem | sounds like either way the list of 'used as' here should be updated https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.host | |
| 22:35:33 | mriedem | or something mentioned about how this is linked across services for certain features | |
| 22:36:07 | mriedem | > does neutron have a concept of workarounds options? - meaning, could neutron be configured to say if it should use CONF.host or gethostname() | |
| 22:36:23 | mriedem | or the sake of linking nested providers i mean | |
| 22:36:30 | efried | yup. I actually mentioned that (the DEFAULT.host help text) while you were offline. | |
| 22:36:33 | mriedem | b/c that would be backportable | |
| 22:37:01 | efried | what are the rules about neutron and n-cpu versions on a given host? | |
| 22:37:12 | efried | are they allowed to differ? by how much? | |
| 22:37:29 | mriedem | i would guess (1) yes they should be able to differ and (2) assume N-1 | |
| 22:37:48 | mriedem | that's part of the idea behind passing os-vif negotiated objects around so you can do rolling upgrades of those | |
| 22:37:57 | mriedem | over the rest api i mean | |
| 22:38:07 | efried | which one gets to be -1? or are they both allowed to be? | |
| 22:38:16 | efried | ugh, am I making sense? | |
| 22:38:24 | mriedem | i know what you're asking, but i don't have a good answer | |
| 22:38:29 | efried | k | |
| 22:38:29 | efried | well | |
| 22:38:38 | mriedem | i doubt that level of upgrade granularity is documented or tested anywhere | |
| 22:39:17 | mriedem | nova and neutron and cinder and keystone etc should all be able to work with each other at wildly different versions but we don't test upgrades that way | |
| 22:39:29 | mriedem | not because we can't | |
| 22:39:35 | efried | any neutron that does bw as currently written knows that it will work properly with `gethostname()` | |
| 22:39:35 | efried | and | |
| 22:39:35 | efried | it can condition the "new" thing simply on whether nova is exposing the microversion providing the new os-hypervisors qparam | |
| 22:40:05 | efried | try: | |
| 22:40:05 | efried | new thing | |
| 22:40:05 | efried | except NoSuchMicroversion: | |
| 22:40:05 | efried | old thing | |
| 22:40:30 | efried | unless neutron does better discovery than that (which it should be able to, but that doesn't mean it does) | |
| 22:40:34 | efried | Okay, | |
| 22:40:50 | mriedem | i would probably start with just a simple workaround optoin in neutron which is backportable without the microversion stuff | |
| 22:41:00 | mriedem | and the way to deprecate the workaround optoin in neutron is the microversion in nova when it's available | |
| 22:41:03 | efried | why is a workaround necessary? | |
| 22:41:17 | mriedem | because neutron has to do a thing based on how nova CONF.host is set yeah? | |
| 22:41:20 | efried | no | |
| 22:41:27 | efried | CONF.host is only ever right by chance | |
| 22:41:33 | efried | `gethostname()` is always right | |
| 22:41:49 | efried | ...for the cases it cares about in existing code, which is what we care about for backportability. | |
| 22:42:28 | efried | Anyway, since gibi and sean-k-mooney aren't here right now, and I don't know whether there's a bug yet, I guess I'll throw out a ML post summarizing the discussion and let it fester from there. | |
| 22:46:11 | mriedem | ok | |
| 22:47:27 | melwitt | speaking of downstream bugs, | |
| 22:48:29 | melwitt | we hit a problem downstream where compute node orphan removal was happening and destroyed the compute node record but failed to delete the RP bc keystone or placement was down, | |
| 22:49:06 | melwitt | and after that, nova-compute could not start up again bc it was trying to create compute node record and then failed with 409 dupe from placement when trying to create the same provider | |
| 22:50:07 | mriedem | unrelated, i just wanted to say this makes us look dumb https://docs.openstack.org/api-guide/compute/general_info.html#relationship-with-volume-api | |
| 22:50:14 | melwitt | searching for ResourceProviderCreationFailed led me to mriedem's patch https://review.opendev.org/#/c/678100/1/nova/compute/manager.py@8332 where he posed the question, should we swap the destroy and RP delete ordering, | |
| 22:50:17 | mriedem | and this https://docs.openstack.org/api-guide/compute/server_concepts.html#server-creation | |
| 22:50:23 | melwitt | and I think the answer is yes | |
| 22:51:19 | melwitt | that's odd. I'm not why someone added empty docs. and I hope it wasn't me | |
| 22:51:23 | melwitt | *not sure | |
| 22:51:51 | mriedem | the api-guide was imported i think so no i'm not saying you, i just was looking for some stuff and noticed these | |
| 22:52:04 | mriedem | these giant todo gaps in our user-facing docs are embarrassing | |
| 22:52:11 | mriedem | i'd rather we just delete them than leave them | |
| 22:52:20 | melwitt | yeah, I was gonna say, just remove em | |
| 22:52:36 | melwitt | that was a joke, sometimes I see a thing and be like, wtf who did this and find it was me | |
| 22:54:25 | mriedem | your provider issue is also because in queens we didn't link the ironic node id to the compute node uuid to the provider uuid, we started that in rocky | |
| 22:54:37 | mriedem | so your recourse in queens is deleting the old providers so compute on restart can re-create them | |
| 22:54:40 | mriedem | but you'll have to heal allocations | |
| 22:54:43 | mriedem | which isn't in quenes | |
| 22:54:45 | mriedem | *queens | |
| 22:55:18 | mriedem | there is a pretty beefy ML thread about all of this orphaned provider stuff months back | |
| 22:55:38 | mriedem | i've been slowly polishing these turds | |
| 22:55:54 | melwitt | yeah, sean-k-mooney mentioned that | |
| 22:55:59 | melwitt | the turd polishing | |
| 22:56:07 | mriedem | http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007135.html is the tl;dr of the first mega beef thread | |
| 22:56:25 | mriedem | http://lists.openstack.org/pipermail/openstack-discuss/2019-June/thread.html#7097 | |
| 22:56:37 | melwitt | thank you. my brain is like about to explode so tl;dr is majorly appreciated | |
| 22:56:57 | mriedem | http://lists.openstack.org/pipermail/openstack-discuss/2019-November/thread.html#10642 is the post-ptg summary | |
| 22:57:02 | melwitt | I'm adding a comment to your review just so.... it's there | |
| 22:57:25 | mriedem | yeah so related to this, | |
| 22:57:33 | mriedem | https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:stable/train+topic:bug/1852610 | |