| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-03 | |||
| 20:46:49 | clarkb | it could be just run the tempest doc job with the version of sphinx that you want | |
| 20:47:06 | clarkb | I think you might need to edit requirements upper-constraints and do a depends on for that | |
| 20:47:45 | clarkb | its possible that should be os.path.abspath(os.path.join(__file__, '../../')) | |
| 20:47:55 | clarkb | then you have to refer to things as tempest.cmd not just cmd | |
| 20:49:15 | zigo | BINGO !!! | |
| 20:49:17 | zigo | I was right. | |
| 20:51:25 | zigo | I see no reason why the conf.py does this ... When I remove it, building the doc continues to work. | |
| 20:51:59 | clarkb | the git log might give a hint. But if it works without it then removing it seems like a good option | |
| 20:52:23 | zigo | Yeah, let me git blame. | |
| 20:52:59 | clarkb | If extensions (or modules to document with autodoc) are in another directory, add these directories to sys.path here. If the directory is relative to the documentation root, use os.path.abspath to make it absolute, like shown here. | |
| 20:53:28 | clarkb | thats from the comment in conf.py. So its related to autodoc. It is possible that it is required to make autodoc work and then that breaks sphinx trying to use cmd | |
| 20:55:07 | clarkb | zigo: if you look at the automodule directives they all start with tempest | |
| 20:55:35 | clarkb | so my hunch here is that the path manipulation is unnecessary (because its off by a level anyway) and that the doc build must be relying on tox installing the package before building | |
| 20:55:48 | clarkb | from https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#ensuring-the-code-can-be-imported the code just need to be importable | |
| 20:56:39 | clarkb | if you need to keep it then I think you just need __file__ + '../../' then tempest is in that directory and the automodule directives should find it is importable from there. But also we can stop polluting the nova channel as this is a qa problem | |
| 20:57:34 | zigo | Let's see how it goes: https://review.opendev.org/c/openstack/tempest/+/959390 :) | |
| 20:58:43 | zigo | clarkb: Yeah, wrong chan, appologies. | |
| 21:08:59 | zigo | Comes from here: https://review.opendev.org/c/openstack/tempest/+/735619 | |
| #openstack-nova - 2025-09-04 | |||
| 06:53:46 | opendevreview | Rajesh Tailor proposed openstack/nova stable/2025.1: Fix 'nova-manage image_property set' command https://review.opendev.org/c/openstack/nova/+/958834 | |
| 08:33:58 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2115905 https://review.opendev.org/c/openstack/nova/+/954336 | |
| 08:33:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: [PCI tracker]Remove non configured devs when freed https://review.opendev.org/c/openstack/nova/+/954613 | |
| 08:33:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149 | |
| 08:34:59 | gibi | sean-k-mooney: ^^ I fixed the commit messages and pulled out two constants for the log message asserts | |
| 08:36:23 | sean-k-mooney[m] | cool ill review it after i grab a coffee and start my day | |
| 09:01:48 | mmessah | Hi all, `/v2.1/os-hypervisors/detail` requests take too long time after upgrade to Epoxy. I saw similar issue for server list like https://bugs.launchpad.net/nova/+bug/2106380 but we dont have issue for server list. We tested manually with curl request. if I use OpenStack-API-Version is 2.88 and above, we wait about 20 seconds. if I use OpenStack-API-Version is 2.87, result is returned immediately. Is there anyone | |
| 09:01:48 | mmessah | experiencing this situation? | |
| 09:06:15 | jkulik | 2.88 added the "uptime" field into the hypervisor details. we experienced slowness with that, too ... and thus added a config setting downstream to patch it out: https://github.com/sapcc/nova/commit/686f44c4c65eba6c851c852804e82a3e00309945 | |
| 09:11:41 | frickler | jkulik: interesting, that looks like a different source of slowness to me, did you ever try to upstream your patch? it does seem pretty reasonable to me (caveat I'm no nova developer, just another deployer) | |
| 09:12:39 | jkulik | frickler: no, it stems from a time with less upstream-involvement. especially after an upgrade, downstream times are quite ... stressful :) I'd be happy to propose that, though | |
| 09:13:11 | masahito | we hit the same issue from the v2.88. calling the RPC to all hypervisors in serial causes slow os-hypervisor/detail API response. | |
| 09:15:04 | masahito | large scale deployment hits the same issue. we just wait few minutes for 10K HV situation, or use limit and maker to separate API. | |
| 09:19:53 | frickler | yes, iterating over all hypervisors for an API call (assuming there's no caching, didn't check the code) sounds pretty much not scalable | |
| 09:26:49 | jkulik | hm ... Isn't there some policy that the API response shouldn't change depending on operator configs? our downstream patch would break that policy. | |
| 09:28:16 | jkulik | api-ref says "Only reported for active hosts where the virt driver supports this feature." so ... maybe it's still fine, because clients would have to handle those cases anyways? | |
| 09:34:18 | bauzas | hey folks, do we have any upstream tracker for that bug ? | |
| 09:34:41 | bauzas | just overlooked the code and yeah I'm afraid of the fact this is indeed a performance issue | |
| 09:35:10 | bauzas | when calling _view_hypervisor(), we then call the host API for each of the hypervisors | |
| 09:35:16 | bauzas | gibi: ^ | |
| 09:35:21 | jkulik | I haven't found one. Also no review-request. | |
| 09:36:18 | bauzas | https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L100-L101 | |
| 09:40:47 | opendevreview | Johannes Kulik proposed openstack/nova master: Conditionally disable hypervisor uptime in details API https://review.opendev.org/c/openstack/nova/+/959420 | |
| 09:42:15 | jkulik | ready to work on making ^ better. I guess it needs a releasenote at least? but before that, I'd like to get some feedback if this approach should be continue | |
| 09:42:51 | bauzas | eek, https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/virt/libvirt/driver.py#L12319-L12322 | |
| 09:43:11 | bauzas | so now when calling statistics, we do N calls to an external process call | |
| 09:43:39 | bauzas | jkulik: I need to review your patch but we can't have API behaviour changed by configuration | |
| 09:44:13 | jkulik | yeah, figured. it's already behaving like that for a subset of the HVs, though | |
| 09:45:15 | bauzas | hmmm, unfortunately I need to leave by the next mins | |
| 09:46:07 | bauzas | jkulik: where do we conditionally change the API behaviour ? | |
| 09:46:12 | bauzas | I mean, already | |
| 09:46:33 | jkulik | depending on if the calls succeeed or rais NotImplemented, we return None instead of an actual uptime string | |
| 09:46:50 | jkulik | so this patch changes to "always None" | |
| 09:48:00 | bauzas | atm, if we return None, this is not because we change the configuration, right? this is just whether the call works or not | |
| 09:48:28 | bauzas | the concern with an confiugrable API is about interop | |
| 09:48:29 | jkulik | yes. for some hypervisors, this call never works. api-ref states for "uptime": "Only reported for active hosts where the virt driver supports this feature." | |
| 09:49:03 | jkulik | so imho, a client cannot rely on this being set | |
| 09:57:03 | stephenfin | jkulik: You're focused on the vmware driver, right? And I assume the noise you're talking about is the tracebacks in nova-compute? Couldn't you just implement the method for that driver but return None? | |
| 09:57:20 | stephenfin | That could likely be backported too | |
| 10:01:40 | jkulik | sure. doesn't bring any speedup at all though, so the _actual_ issue of os-hypervisor/detail being slow doesn't go away | |
| 10:05:23 | opendevreview | sean mooney proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149 | |
| 10:19:28 | opendevreview | Johannes Kulik proposed openstack/nova master: Conditionally disable hypervisor uptime in details API https://review.opendev.org/c/openstack/nova/+/959420 | |
| 10:34:17 | sean-k-mooney | jkulik: yes we have a general rule about no api driven behvaiaor | |
| 10:34:33 | sean-k-mooney | jkulik: also the uptime prerfroamce issue is known | |
| 10:35:14 | sean-k-mooney | jkulik: the way we plan to fix this is to add it to the stats in the compute node db table and read it form there instead od doing the rpc | |
| 10:35:33 | sean-k-mooney | so in the future you will get a cached value but wont have the perfomacne hit | |
| 10:35:48 | sean-k-mooney | it will be in the data revvived with teh ohte rcompute node values in the same sql query | |
| 10:36:39 | sean-k-mooney | jkulik: gibi and i debugged this a month or two ago but didnt have time tow rok on proposing the fix | |
| 10:38:14 | sean-k-mooney | basiclly the plan to make this "just work" across upgrades is to make the rpc consditional and to get the uptime value only if its not in the stats blob we get form teh db | |
| 10:55:15 | jkulik | sean-k-mooney: thanks for sharing. I'll abandon my RR then. | |
| 10:55:47 | jkulik | sean-k-mooney: so this means, all drivers currently not implementing get_host_uptime() should at least provide a None value into the stats to not still hit the performance problem, right? | |
| 11:02:27 | sean-k-mooney | yep that or we could jsut remove the fallback entirly | |
| 11:02:44 | sean-k-mooney | jkulik: we breifly dicsused just removing uptime form the api in a new microversion | |
| 11:03:00 | sean-k-mooney | we kind of wanted to get wider input on that before moving forward however | |
| 11:03:43 | sean-k-mooney | jkulik: we orginally wanted to remove uptime in the orgial spec but some operator wanted use to keep it so we did | |
| 11:03:52 | sean-k-mooney | but im not sure that was the right choice | |
| 11:04:31 | sean-k-mooney | jkulik: does vmware supprot uptime at all | |
| 11:04:52 | sean-k-mooney | jkulik: we could also make the fallback to rpc condtional on if the driver suprot it | |
| 11:05:01 | jkulik | sean-k-mooney: no, not at all, because in VMware we have a whole cluster as hypervisor in Nova and each node in the cluster has different uptime | |
| 11:05:02 | sean-k-mooney | and then just exclude vmware if it does not | |
| 11:05:32 | jkulik | we're also introducing libvirt, so I'd still like to get the details to become fast ;) | |
| 11:05:36 | sean-k-mooney | right so we know the hypervior type so we coudl limit the fallback to just libvirt(qemu) | |
| 11:06:36 | sean-k-mooney | so for all ohter driver we coudl retrun None | |
| 11:06:45 | sean-k-mooney | if not in stats | |
| 11:07:39 | jkulik | ... or patch everyone to return None, which would be fine for me, too | |
| 11:08:07 | sean-k-mooney | well again we propsoed removing uptime when we removed the hypervior stats api | |
| 11:08:26 | sean-k-mooney | and some folks objected to that so we cant actully do that without a new microversion | |
| 11:08:52 | sean-k-mooney | i.e. it woudl break our api rules ot just stop reutring without a new microversion | |
| 11:09:19 | sean-k-mooney | but instead of a config opiotn https://review.opendev.org/c/openstack/nova/+/959420/2/nova/api/openstack/compute/hypervisors.py#105 | |
| 11:09:40 | sean-k-mooney | you coudl have checked if the hypervior type was qemu and only get the uptime if it was | |
| 11:10:20 | sean-k-mooney | evenutaly we want to also stash it in the stats to make libvirt fast but that would fix vmware for now | |
| 11:10:50 | jkulik | I mean, we have the patch downstream. there's no pressure from our side. I'd go with the proper "put in stats" solution. | |
| 11:11:29 | sean-k-mooney | ack. we may want to put this in the ptg adjenda to not forget or have a dedicated bug for it | |
| 11:12:07 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/2106380 is about server list not the hyperviors api so we sould avoid resuing it | |
| 11:12:25 | sean-k-mooney | with that said i wonder if gibi already filed an upstream bug | |
| 11:38:28 | opendevreview | Rajesh Tailor proposed openstack/nova stable/2024.2: Fix 'nova-manage image_property set' command https://review.opendev.org/c/openstack/nova/+/959429 | |
| 11:41:43 | gibi | bauzas: jkulik: the uptime issue ring bells to me but I think it was only reported downstream. I agree that this needs to be fixed | |
| 11:42:04 | gibi | until then the WA is using older microversion | |
| 11:43:09 | gibi | jkulik: yeah a config driven API is a no-no. But we can revisit why we added the uptime to the main GET call, or we can try to speed it up via scatter gather, or revers the relationship and send up the uptime data to the DB periodicall from the compute so reading it from the API will be quick | |
| 11:43:21 | gibi | hm that latter rings another bell that this was dansmith's idea | |
| 11:43:29 | gibi | so we might discussed this already upstream as well | |
| 11:48:22 | gibi | aand there is another semi related thread of sending that uptime RPC to computes that are down, but not yet marked as down in the DB is another source of slowness | |