| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-21 | |||
| 22:15:51 | mriedem | oopsy daisy | |
| 22:16:28 | efried | [1] https://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-detail#id306 | |
| 22:16:28 | efried | sean-k-mooney: o hey, it looks like you were maybe getting the hypervisor ID as a short ID because you were using the default microversion. Per [1] it'll come back as a UUID starting at 2.53. | |
| 22:16:38 | efried | which phew gives us a path forward without further changes. | |
| 22:16:46 | efried | I think | |
| 22:17:12 | efried | Having to hit the /os-hypervisors API is heavier than just looking at CONF.host, but that UUID is the UUID of the provider you want. | |
| 22:17:19 | mriedem | yup | |
| 22:18:35 | efried | hm, but mriedem, how do I ask /os-hypervisors for the entry for my current node? | |
| 22:19:14 | efried | cause the qparams and output still talk in terms of hypervisor_hostname :( | |
| 22:19:18 | mriedem | GET /os-hypervisors/detail?hypervisor_hostname_pattern=<hostname> | |
| 22:19:54 | efried | right, exactly. | |
| 22:20:17 | efried | I mean, if I'm going to rely on running `gethostname()` on the host, I might as well just do that from the start. | |
| 22:20:19 | mriedem | if not that, then i guess GET /os-hypervisors/detail and you iterate all until you find the one with the service.host that matches what you care about if hypervisor_hostname is not a match | |
| 22:21:17 | mriedem | i think service.host there is CONF.host | |
| 22:21:31 | mriedem | someone had an RFE at one point to be able to filter hypervisors by service.host for ironic | |
| 22:21:35 | mriedem | it would be pretty easy to do i think | |
| 22:21:50 | mriedem | it's not clear to me if you need that though | |
| 22:22:01 | mriedem | you being neutron/cyborg | |
| 22:22:14 | efried | that would be a sure, but heavy, way to correlate CONF.host. | |
| 22:22:22 | efried | esp in a big env, that's a big payload | |
| 22:22:39 | mriedem | which thing? getting all hypervisors and then finding the service.host? | |
| 22:23:00 | mriedem | yeah it would, and likely limited to 1000 results by default so you'd have to page | |
| 22:23:03 | efried | yeah, GET /os-hypervisors/detail is going to be a big response for CERn. | |
| 22:23:13 | efried | use sdk and it pages for you, which is fin. | |
| 22:23:14 | efried | fine. | |
| 22:23:18 | mriedem | sure, so GET /os-hypervisors/detail?service_host=CONF.host | |
| 22:23:25 | mriedem | ^ is the RFE i mentioned | |
| 22:23:27 | efried | oh, that's a thing... | |
| 22:23:31 | mriedem | ha, no | |
| 22:23:33 | mriedem | oh eric | |
| 22:23:51 | mriedem | https://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-details-detail#list-hypervisors-details | |
| 22:24:14 | efried | yes, I'm looking at that. You're taking advantage of my weakened mental state to f with me. Is it April? | |
| 22:24:33 | mriedem | why is your mental state weak? | |
| 22:25:06 | efried | is it ever not? | |
| 22:25:23 | mriedem | i can't find the bug, i could have sworn a guy opened one though | |
| 22:25:52 | mriedem | but that was the idea, he was trying to filter hypervisors by ironic compute service host but was only getting nodes based on the ironic node uuid which wasn't helpful | |
| 22:25:54 | efried | okay, anyway, yes, that would be a nice way to make this work that involves an API change with a microversion. | |
| 22:26:38 | efried | but for the sake of discussion... | |
| 22:27:13 | efried | would it be so wrong for neutron/cyborg to simply use the result of `gethostname()` instead of CONF.host? | |
| 22:27:37 | efried | I guess that becomes coupled to the virt driver implementation. | |
| 22:27:59 | efried | though arguably using CONF.host at all already is. | |
| 22:28:14 | efried | in that it at least will never work for ironic | |
| 22:29:29 | mriedem | we don't care about ironic for nested providers | |
| 22:29:31 | mriedem | or most things | |
| 22:30:07 | efried | I can totally see needing to do... *something* with bandwidth for ironic. | |
| 22:30:16 | efried | Though I imagine the providers would be shared in that case, not nested. | |
| 22:30:29 | efried | nevertheless we would have to discover the ironic nodes for aggregation purposes. | |
| 22:30:31 | efried | anyway... | |
| 22:33:06 | mriedem | idk | |
| 22:33:35 | mriedem | filtering hypervisors by service host seems useful in general so if it could be used here then i don't see a reason not to add that earlier than later | |
| 22:33:55 | mriedem | always nice to get to N+3 release from now and be like, "we have this problem, oh but we added X in N so we an use that" | |
| 22:33:56 | efried | okay. Not backportable tho | |
| 22:34:00 | mriedem | nope | |
| 22:34:36 | efried | For backport purposes, I guess since the scope is known and constrained to libvirt, we could just ask neutron to use gethostname() instead of CONF.host. | |
| 22:35:05 | mriedem | again, idk... | |
| 22:35:10 | mriedem | does neutron have a concept of workarounds options? | |
| 22:35:22 | mriedem | sounds like either way the list of 'used as' here should be updated https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.host | |
| 22:35:33 | mriedem | or something mentioned about how this is linked across services for certain features | |
| 22:36:07 | mriedem | > does neutron have a concept of workarounds options? - meaning, could neutron be configured to say if it should use CONF.host or gethostname() | |
| 22:36:23 | mriedem | or the sake of linking nested providers i mean | |
| 22:36:30 | efried | yup. I actually mentioned that (the DEFAULT.host help text) while you were offline. | |
| 22:36:33 | mriedem | b/c that would be backportable | |
| 22:37:01 | efried | what are the rules about neutron and n-cpu versions on a given host? | |
| 22:37:12 | efried | are they allowed to differ? by how much? | |
| 22:37:29 | mriedem | i would guess (1) yes they should be able to differ and (2) assume N-1 | |
| 22:37:48 | mriedem | that's part of the idea behind passing os-vif negotiated objects around so you can do rolling upgrades of those | |
| 22:37:57 | mriedem | over the rest api i mean | |
| 22:38:07 | efried | which one gets to be -1? or are they both allowed to be? | |
| 22:38:16 | efried | ugh, am I making sense? | |
| 22:38:24 | mriedem | i know what you're asking, but i don't have a good answer | |
| 22:38:29 | efried | well | |
| 22:38:29 | efried | k | |
| 22:38:38 | mriedem | i doubt that level of upgrade granularity is documented or tested anywhere | |
| 22:39:17 | mriedem | nova and neutron and cinder and keystone etc should all be able to work with each other at wildly different versions but we don't test upgrades that way | |
| 22:39:29 | mriedem | not because we can't | |
| 22:39:35 | efried | it can condition the "new" thing simply on whether nova is exposing the microversion providing the new os-hypervisors qparam | |
| 22:39:35 | efried | and | |
| 22:39:35 | efried | any neutron that does bw as currently written knows that it will work properly with `gethostname()` | |
| 22:40:05 | efried | old thing | |
| 22:40:05 | efried | except NoSuchMicroversion: | |
| 22:40:05 | efried | new thing | |
| 22:40:05 | efried | try: | |
| 22:40:30 | efried | unless neutron does better discovery than that (which it should be able to, but that doesn't mean it does) | |
| 22:40:34 | efried | Okay, | |
| 22:40:50 | mriedem | i would probably start with just a simple workaround optoin in neutron which is backportable without the microversion stuff | |
| 22:41:00 | mriedem | and the way to deprecate the workaround optoin in neutron is the microversion in nova when it's available | |
| 22:41:03 | efried | why is a workaround necessary? | |
| 22:41:17 | mriedem | because neutron has to do a thing based on how nova CONF.host is set yeah? | |
| 22:41:20 | efried | no | |
| 22:41:27 | efried | CONF.host is only ever right by chance | |
| 22:41:33 | efried | `gethostname()` is always right | |
| 22:41:49 | efried | ...for the cases it cares about in existing code, which is what we care about for backportability. | |
| 22:42:28 | efried | Anyway, since gibi and sean-k-mooney aren't here right now, and I don't know whether there's a bug yet, I guess I'll throw out a ML post summarizing the discussion and let it fester from there. | |
| 22:46:11 | mriedem | ok | |
| 22:47:27 | melwitt | speaking of downstream bugs, | |
| 22:48:29 | melwitt | we hit a problem downstream where compute node orphan removal was happening and destroyed the compute node record but failed to delete the RP bc keystone or placement was down, | |
| 22:49:06 | melwitt | and after that, nova-compute could not start up again bc it was trying to create compute node record and then failed with 409 dupe from placement when trying to create the same provider | |
| 22:50:07 | mriedem | unrelated, i just wanted to say this makes us look dumb https://docs.openstack.org/api-guide/compute/general_info.html#relationship-with-volume-api | |
| 22:50:14 | melwitt | searching for ResourceProviderCreationFailed led me to mriedem's patch https://review.opendev.org/#/c/678100/1/nova/compute/manager.py@8332 where he posed the question, should we swap the destroy and RP delete ordering, | |
| 22:50:17 | mriedem | and this https://docs.openstack.org/api-guide/compute/server_concepts.html#server-creation | |