Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-09-03
18:38:07 gmaan Uggla: thanks +2 on both
19:55:04 zigo sean-k-mooney: It's normal to do PYTHONPATH=., because the package isn't installed in /usr/lib/python3/dist-packages when calling sphinx, that's a way to add the module in the path.
19:55:14 zigo I do that very often in my packages, with no issue.
19:55:43 zigo As for my error here, the weird thing is that it happens it crashes in Unstable, but not in Trixie, so something must have changed.
19:55:54 zigo I'll try to do a pip freeze and compare.
19:58:28 clarkb zigo: why not add the path to the location rather than .?
19:59:12 clarkb then cmd would only be resolvable as tempest.cmd or the stdlib stuff
20:31:47 zigo clarkb: It produces the same result, that's not the issue.
20:33:14 clarkb zigo: I think that means you're a directory level too deep?
20:33:45 clarkb youre in tempest/ so cmd is there and . allows you to import cmd that way. Another approach would be to put it at the end of your path if python resolves the path in order
20:33:53 zigo I'm not.
20:33:58 zigo I'm on top of that folder.
20:39:07 clarkb its possible that the sphinx build changes its cwd maybe? (but I would expect the rooted version instead of '.' to fix that)
20:40:50 zigo Ok, the diff is because of sphinx version.
20:41:19 zigo 8.1.3 in Trixie, vs 8.2.3 in Unstable.
20:41:58 zigo I checked building in Trixie, adding sphinx as --extra-package in the sbuild command line.
20:42:17 zigo And then it fails ...
20:42:38 zigo clarkb: Is there a way I can do such demonstration with a patch in gerrit?
20:46:03 zigo In conf.py, there's:
20:46:03 zigo sys.path.insert(0, os.path.abspath('../../tempest'))
20:46:26 zigo That's a likely guilty thingy ... :P
20:46:49 clarkb it could be just run the tempest doc job with the version of sphinx that you want
20:47:06 clarkb I think you might need to edit requirements upper-constraints and do a depends on for that
20:47:45 clarkb its possible that should be os.path.abspath(os.path.join(__file__, '../../'))
20:47:55 clarkb then you have to refer to things as tempest.cmd not just cmd
20:49:15 zigo BINGO !!!
20:49:17 zigo I was right.
20:51:25 zigo I see no reason why the conf.py does this ... When I remove it, building the doc continues to work.
20:51:59 clarkb the git log might give a hint. But if it works without it then removing it seems like a good option
20:52:23 zigo Yeah, let me git blame.
20:52:59 clarkb If extensions (or modules to document with autodoc) are in another directory, add these directories to sys.path here. If the directory is relative to the documentation root, use os.path.abspath to make it absolute, like shown here.
20:53:28 clarkb thats from the comment in conf.py. So its related to autodoc. It is possible that it is required to make autodoc work and then that breaks sphinx trying to use cmd
20:55:07 clarkb zigo: if you look at the automodule directives they all start with tempest
20:55:35 clarkb so my hunch here is that the path manipulation is unnecessary (because its off by a level anyway) and that the doc build must be relying on tox installing the package before building
20:55:48 clarkb from https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#ensuring-the-code-can-be-imported the code just need to be importable
20:56:39 clarkb if you need to keep it then I think you just need __file__ + '../../' then tempest is in that directory and the automodule directives should find it is importable from there. But also we can stop polluting the nova channel as this is a qa problem
20:57:34 zigo Let's see how it goes: https://review.opendev.org/c/openstack/tempest/+/959390 :)
20:58:43 zigo clarkb: Yeah, wrong chan, appologies.
21:08:59 zigo Comes from here: https://review.opendev.org/c/openstack/tempest/+/735619
#openstack-nova - 2025-09-04
06:53:46 opendevreview Rajesh Tailor proposed openstack/nova stable/2025.1: Fix 'nova-manage image_property set' command https://review.opendev.org/c/openstack/nova/+/958834
08:33:58 opendevreview Balazs Gibizer proposed openstack/nova master: Reproduce bug/2115905 https://review.opendev.org/c/openstack/nova/+/954336
08:33:59 opendevreview Balazs Gibizer proposed openstack/nova master: [PCI tracker]Remove non configured devs when freed https://review.opendev.org/c/openstack/nova/+/954613
08:33:59 opendevreview Balazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149
08:34:59 gibi sean-k-mooney: ^^ I fixed the commit messages and pulled out two constants for the log message asserts
08:36:23 sean-k-mooney[m] cool ill review it after i grab a coffee and start my day
09:01:48 mmessah Hi all, `/v2.1/os-hypervisors/detail` requests take too long time after upgrade to Epoxy. I saw similar issue for server list like https://bugs.launchpad.net/nova/+bug/2106380 but we dont have issue for server list. We tested manually with curl request. if I use OpenStack-API-Version is 2.88 and above, we wait about 20 seconds. if I use OpenStack-API-Version is 2.87, result is returned immediately. Is there anyone
09:01:48 mmessah experiencing this situation?
09:06:15 jkulik 2.88 added the "uptime" field into the hypervisor details. we experienced slowness with that, too ... and thus added a config setting downstream to patch it out: https://github.com/sapcc/nova/commit/686f44c4c65eba6c851c852804e82a3e00309945
09:11:41 frickler jkulik: interesting, that looks like a different source of slowness to me, did you ever try to upstream your patch? it does seem pretty reasonable to me (caveat I'm no nova developer, just another deployer)
09:12:39 jkulik frickler: no, it stems from a time with less upstream-involvement. especially after an upgrade, downstream times are quite ... stressful :) I'd be happy to propose that, though
09:13:11 masahito we hit the same issue from the v2.88. calling the RPC to all hypervisors in serial causes slow os-hypervisor/detail API response.
09:15:04 masahito large scale deployment hits the same issue. we just wait few minutes for 10K HV situation, or use limit and maker to separate API.
09:19:53 frickler yes, iterating over all hypervisors for an API call (assuming there's no caching, didn't check the code) sounds pretty much not scalable
09:26:49 jkulik hm ... Isn't there some policy that the API response shouldn't change depending on operator configs? our downstream patch would break that policy.
09:28:16 jkulik api-ref says "Only reported for active hosts where the virt driver supports this feature." so ... maybe it's still fine, because clients would have to handle those cases anyways?
09:34:18 bauzas hey folks, do we have any upstream tracker for that bug ?
09:34:41 bauzas just overlooked the code and yeah I'm afraid of the fact this is indeed a performance issue
09:35:10 bauzas when calling _view_hypervisor(), we then call the host API for each of the hypervisors
09:35:16 bauzas gibi: ^
09:35:21 jkulik I haven't found one. Also no review-request.
09:36:18 bauzas https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L100-L101
09:40:47 opendevreview Johannes Kulik proposed openstack/nova master: Conditionally disable hypervisor uptime in details API https://review.opendev.org/c/openstack/nova/+/959420
09:42:15 jkulik ready to work on making ^ better. I guess it needs a releasenote at least? but before that, I'd like to get some feedback if this approach should be continue
09:42:51 bauzas eek, https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/virt/libvirt/driver.py#L12319-L12322
09:43:11 bauzas so now when calling statistics, we do N calls to an external process call
09:43:39 bauzas jkulik: I need to review your patch but we can't have API behaviour changed by configuration
09:44:13 jkulik yeah, figured. it's already behaving like that for a subset of the HVs, though
09:45:15 bauzas hmmm, unfortunately I need to leave by the next mins
09:46:07 bauzas jkulik: where do we conditionally change the API behaviour ?
09:46:12 bauzas I mean, already
09:46:33 jkulik depending on if the calls succeeed or rais NotImplemented, we return None instead of an actual uptime string
09:46:50 jkulik so this patch changes to "always None"
09:48:00 bauzas atm, if we return None, this is not because we change the configuration, right? this is just whether the call works or not
09:48:28 bauzas the concern with an confiugrable API is about interop
09:48:29 jkulik yes. for some hypervisors, this call never works. api-ref states for "uptime": "Only reported for active hosts where the virt driver supports this feature."
09:49:03 jkulik so imho, a client cannot rely on this being set
09:57:03 stephenfin jkulik: You're focused on the vmware driver, right? And I assume the noise you're talking about is the tracebacks in nova-compute? Couldn't you just implement the method for that driver but return None?
09:57:20 stephenfin That could likely be backported too
10:01:40 jkulik sure. doesn't bring any speedup at all though, so the _actual_ issue of os-hypervisor/detail being slow doesn't go away
10:05:23 opendevreview sean mooney proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149
10:19:28 opendevreview Johannes Kulik proposed openstack/nova master: Conditionally disable hypervisor uptime in details API https://review.opendev.org/c/openstack/nova/+/959420
10:34:17 sean-k-mooney jkulik: yes we have a general rule about no api driven behvaiaor
10:34:33 sean-k-mooney jkulik: also the uptime prerfroamce issue is known
10:35:14 sean-k-mooney jkulik: the way we plan to fix this is to add it to the stats in the compute node db table and read it form there instead od doing the rpc
10:35:33 sean-k-mooney so in the future you will get a cached value but wont have the perfomacne hit
10:35:48 sean-k-mooney it will be in the data revvived with teh ohte rcompute node values in the same sql query
10:36:39 sean-k-mooney jkulik: gibi and i debugged this a month or two ago but didnt have time tow rok on proposing the fix
10:38:14 sean-k-mooney basiclly the plan to make this "just work" across upgrades is to make the rpc consditional and to get the uptime value only if its not in the stats blob we get form teh db
10:55:15 jkulik sean-k-mooney: thanks for sharing. I'll abandon my RR then.
10:55:47 jkulik sean-k-mooney: so this means, all drivers currently not implementing get_host_uptime() should at least provide a None value into the stats to not still hit the performance problem, right?
11:02:27 sean-k-mooney yep that or we could jsut remove the fallback entirly
11:02:44 sean-k-mooney jkulik: we breifly dicsused just removing uptime form the api in a new microversion
11:03:00 sean-k-mooney we kind of wanted to get wider input on that before moving forward however
11:03:43 sean-k-mooney jkulik: we orginally wanted to remove uptime in the orgial spec but some operator wanted use to keep it so we did
11:03:52 sean-k-mooney but im not sure that was the right choice
11:04:31 sean-k-mooney jkulik: does vmware supprot uptime at all
11:04:52 sean-k-mooney jkulik: we could also make the fallback to rpc condtional on if the driver suprot it
11:05:01 jkulik sean-k-mooney: no, not at all, because in VMware we have a whole cluster as hypervisor in Nova and each node in the cluster has different uptime
11:05:02 sean-k-mooney and then just exclude vmware if it does not
11:05:32 jkulik we're also introducing libvirt, so I'd still like to get the details to become fast ;)

Earlier   Later