Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-09-04
08:36:23 sean-k-mooney[m] cool ill review it after i grab a coffee and start my day
09:01:48 mmessah Hi all, `/v2.1/os-hypervisors/detail` requests take too long time after upgrade to Epoxy. I saw similar issue for server list like https://bugs.launchpad.net/nova/+bug/2106380 but we dont have issue for server list. We tested manually with curl request. if I use OpenStack-API-Version is 2.88 and above, we wait about 20 seconds. if I use OpenStack-API-Version is 2.87, result is returned immediately. Is there anyone
09:01:48 mmessah experiencing this situation?
09:06:15 jkulik 2.88 added the "uptime" field into the hypervisor details. we experienced slowness with that, too ... and thus added a config setting downstream to patch it out: https://github.com/sapcc/nova/commit/686f44c4c65eba6c851c852804e82a3e00309945
09:11:41 frickler jkulik: interesting, that looks like a different source of slowness to me, did you ever try to upstream your patch? it does seem pretty reasonable to me (caveat I'm no nova developer, just another deployer)
09:12:39 jkulik frickler: no, it stems from a time with less upstream-involvement. especially after an upgrade, downstream times are quite ... stressful :) I'd be happy to propose that, though
09:13:11 masahito we hit the same issue from the v2.88. calling the RPC to all hypervisors in serial causes slow os-hypervisor/detail API response.
09:15:04 masahito large scale deployment hits the same issue. we just wait few minutes for 10K HV situation, or use limit and maker to separate API.
09:19:53 frickler yes, iterating over all hypervisors for an API call (assuming there's no caching, didn't check the code) sounds pretty much not scalable
09:26:49 jkulik hm ... Isn't there some policy that the API response shouldn't change depending on operator configs? our downstream patch would break that policy.
09:28:16 jkulik api-ref says "Only reported for active hosts where the virt driver supports this feature." so ... maybe it's still fine, because clients would have to handle those cases anyways?
09:34:18 bauzas hey folks, do we have any upstream tracker for that bug ?
09:34:41 bauzas just overlooked the code and yeah I'm afraid of the fact this is indeed a performance issue
09:35:10 bauzas when calling _view_hypervisor(), we then call the host API for each of the hypervisors
09:35:16 bauzas gibi: ^
09:35:21 jkulik I haven't found one. Also no review-request.
09:36:18 bauzas https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L100-L101
09:40:47 opendevreview Johannes Kulik proposed openstack/nova master: Conditionally disable hypervisor uptime in details API https://review.opendev.org/c/openstack/nova/+/959420
09:42:15 jkulik ready to work on making ^ better. I guess it needs a releasenote at least? but before that, I'd like to get some feedback if this approach should be continue
09:42:51 bauzas eek, https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/virt/libvirt/driver.py#L12319-L12322
09:43:11 bauzas so now when calling statistics, we do N calls to an external process call
09:43:39 bauzas jkulik: I need to review your patch but we can't have API behaviour changed by configuration
09:44:13 jkulik yeah, figured. it's already behaving like that for a subset of the HVs, though
09:45:15 bauzas hmmm, unfortunately I need to leave by the next mins
09:46:07 bauzas jkulik: where do we conditionally change the API behaviour ?
09:46:12 bauzas I mean, already
09:46:33 jkulik depending on if the calls succeeed or rais NotImplemented, we return None instead of an actual uptime string
09:46:50 jkulik so this patch changes to "always None"
09:48:00 bauzas atm, if we return None, this is not because we change the configuration, right? this is just whether the call works or not
09:48:28 bauzas the concern with an confiugrable API is about interop
09:48:29 jkulik yes. for some hypervisors, this call never works. api-ref states for "uptime": "Only reported for active hosts where the virt driver supports this feature."
09:49:03 jkulik so imho, a client cannot rely on this being set
09:57:03 stephenfin jkulik: You're focused on the vmware driver, right? And I assume the noise you're talking about is the tracebacks in nova-compute? Couldn't you just implement the method for that driver but return None?
09:57:20 stephenfin That could likely be backported too
10:01:40 jkulik sure. doesn't bring any speedup at all though, so the _actual_ issue of os-hypervisor/detail being slow doesn't go away
10:05:23 opendevreview sean mooney proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149
10:19:28 opendevreview Johannes Kulik proposed openstack/nova master: Conditionally disable hypervisor uptime in details API https://review.opendev.org/c/openstack/nova/+/959420
10:34:17 sean-k-mooney jkulik: yes we have a general rule about no api driven behvaiaor
10:34:33 sean-k-mooney jkulik: also the uptime prerfroamce issue is known
10:35:14 sean-k-mooney jkulik: the way we plan to fix this is to add it to the stats in the compute node db table and read it form there instead od doing the rpc
10:35:33 sean-k-mooney so in the future you will get a cached value but wont have the perfomacne hit
10:35:48 sean-k-mooney it will be in the data revvived with teh ohte rcompute node values in the same sql query
10:36:39 sean-k-mooney jkulik: gibi and i debugged this a month or two ago but didnt have time tow rok on proposing the fix
10:38:14 sean-k-mooney basiclly the plan to make this "just work" across upgrades is to make the rpc consditional and to get the uptime value only if its not in the stats blob we get form teh db
10:55:15 jkulik sean-k-mooney: thanks for sharing. I'll abandon my RR then.
10:55:47 jkulik sean-k-mooney: so this means, all drivers currently not implementing get_host_uptime() should at least provide a None value into the stats to not still hit the performance problem, right?
11:02:27 sean-k-mooney yep that or we could jsut remove the fallback entirly
11:02:44 sean-k-mooney jkulik: we breifly dicsused just removing uptime form the api in a new microversion
11:03:00 sean-k-mooney we kind of wanted to get wider input on that before moving forward however
11:03:43 sean-k-mooney jkulik: we orginally wanted to remove uptime in the orgial spec but some operator wanted use to keep it so we did
11:03:52 sean-k-mooney but im not sure that was the right choice
11:04:31 sean-k-mooney jkulik: does vmware supprot uptime at all
11:04:52 sean-k-mooney jkulik: we could also make the fallback to rpc condtional on if the driver suprot it
11:05:01 jkulik sean-k-mooney: no, not at all, because in VMware we have a whole cluster as hypervisor in Nova and each node in the cluster has different uptime
11:05:02 sean-k-mooney and then just exclude vmware if it does not
11:05:32 jkulik we're also introducing libvirt, so I'd still like to get the details to become fast ;)
11:05:36 sean-k-mooney right so we know the hypervior type so we coudl limit the fallback to just libvirt(qemu)
11:06:36 sean-k-mooney so for all ohter driver we coudl retrun None
11:06:45 sean-k-mooney if not in stats
11:07:39 jkulik ... or patch everyone to return None, which would be fine for me, too
11:08:07 sean-k-mooney well again we propsoed removing uptime when we removed the hypervior stats api
11:08:26 sean-k-mooney and some folks objected to that so we cant actully do that without a new microversion
11:08:52 sean-k-mooney i.e. it woudl break our api rules ot just stop reutring without a new microversion
11:09:19 sean-k-mooney but instead of a config opiotn https://review.opendev.org/c/openstack/nova/+/959420/2/nova/api/openstack/compute/hypervisors.py#105
11:09:40 sean-k-mooney you coudl have checked if the hypervior type was qemu and only get the uptime if it was
11:10:20 sean-k-mooney evenutaly we want to also stash it in the stats to make libvirt fast but that would fix vmware for now
11:10:50 jkulik I mean, we have the patch downstream. there's no pressure from our side. I'd go with the proper "put in stats" solution.
11:11:29 sean-k-mooney ack. we may want to put this in the ptg adjenda to not forget or have a dedicated bug for it
11:12:07 sean-k-mooney https://bugs.launchpad.net/nova/+bug/2106380 is about server list not the hyperviors api so we sould avoid resuing it
11:12:25 sean-k-mooney with that said i wonder if gibi already filed an upstream bug
11:38:28 opendevreview Rajesh Tailor proposed openstack/nova stable/2024.2: Fix 'nova-manage image_property set' command https://review.opendev.org/c/openstack/nova/+/959429
11:41:43 gibi bauzas: jkulik: the uptime issue ring bells to me but I think it was only reported downstream. I agree that this needs to be fixed
11:42:04 gibi until then the WA is using older microversion
11:43:09 gibi jkulik: yeah a config driven API is a no-no. But we can revisit why we added the uptime to the main GET call, or we can try to speed it up via scatter gather, or revers the relationship and send up the uptime data to the DB periodicall from the compute so reading it from the API will be quick
11:43:21 gibi hm that latter rings another bell that this was dansmith's idea
11:43:29 gibi so we might discussed this already upstream as well
11:48:22 gibi aand there is another semi related thread of sending that uptime RPC to computes that are down, but not yet marked as down in the DB is another source of slowness
11:50:09 jkulik I'd go with the "sending to the DB" - not sure how to solve that for computes being down, though.
11:51:05 gibi jkulik: we already have the periodic heartbeat to know when a compute is down, so we can use that info to avoid reporting back a stale uptime data on the API
12:07:09 gibi I did not find this uptime discussion in my IRC logs so this probably happened downstream. I will file a bug right away summarizing both problems and the solution suggestion
12:16:34 gibi aand I found the downstream report https://issues.redhat.com/browse/OSPRH-17563 so we probably have motivation to try to fix it :)
12:23:01 gibi field the bug https://bugs.launchpad.net/nova/+bug/2122036
12:25:30 gibi mmessah: ^^
12:39:21 mmessah Thanks for discussing the situation fastly :) I hope it will be resolved as soon as possible. We recently upgraded our test environment to Epoxy. We will continue to test all services. If we have any situation, we will share IRC.
12:41:29 gibi mmessah: thanks
12:42:55 opendevreview Rajesh Tailor proposed openstack/nova stable/2024.2: Fix case-sensitivity for metadata keys https://review.opendev.org/c/openstack/nova/+/959435
12:49:04 gibi dansmith: sean-k-mooney: when you are referring to "large-ops" job from the past (in relation to scale testing eventlet removal) do you mean https://review.opendev.org/c/openstack/project-config/+/278978 ?
12:50:53 sean-k-mooney i was not refering to that because i did not know that existed
12:50:58 sean-k-mooney but maybe dan was
12:51:20 gibi sean-k-mooney: ahh sorry I did not remember where this input coming from :)
12:51:21 sean-k-mooney gibi: i have just mention that kolla and devstack can fake may compute nodes per phsyical host
12:51:41 sean-k-mooney my expeirnce of this was as part of the osci lab scale testing
12:52:05 sean-k-mooney where intel and rackspace gave access to i think 500? baremental host half x86 the rest ppc
12:52:13 sean-k-mooney for the comutnity to do some scale testing
12:54:02 sean-k-mooney gibi: i wonder if the rally jobs have any existing usage of the fake agents
12:54:24 sean-k-mooney or if the tempest jobs in rally ieslf are usign real agents
12:55:39 gibi yeah rally is also on my radar to look into, especially neutron's jobs
12:55:56 sean-k-mooney so rally and browbeat are the two scaling frameworks we have
12:56:09 sean-k-mooney at leats those are teh two i am vagley aware of
12:56:46 gibi yeah, I never saw a realy browbeat scenario but heard the name before

Earlier   Later