Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-09-04
09:46:33 jkulik depending on if the calls succeeed or rais NotImplemented, we return None instead of an actual uptime string
09:46:50 jkulik so this patch changes to "always None"
09:48:00 bauzas atm, if we return None, this is not because we change the configuration, right? this is just whether the call works or not
09:48:28 bauzas the concern with an confiugrable API is about interop
09:48:29 jkulik yes. for some hypervisors, this call never works. api-ref states for "uptime": "Only reported for active hosts where the virt driver supports this feature."
09:49:03 jkulik so imho, a client cannot rely on this being set
09:57:03 stephenfin jkulik: You're focused on the vmware driver, right? And I assume the noise you're talking about is the tracebacks in nova-compute? Couldn't you just implement the method for that driver but return None?
09:57:20 stephenfin That could likely be backported too
10:01:40 jkulik sure. doesn't bring any speedup at all though, so the _actual_ issue of os-hypervisor/detail being slow doesn't go away
10:05:23 opendevreview sean mooney proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149
10:19:28 opendevreview Johannes Kulik proposed openstack/nova master: Conditionally disable hypervisor uptime in details API https://review.opendev.org/c/openstack/nova/+/959420
10:34:17 sean-k-mooney jkulik: yes we have a general rule about no api driven behvaiaor
10:34:33 sean-k-mooney jkulik: also the uptime prerfroamce issue is known
10:35:14 sean-k-mooney jkulik: the way we plan to fix this is to add it to the stats in the compute node db table and read it form there instead od doing the rpc
10:35:33 sean-k-mooney so in the future you will get a cached value but wont have the perfomacne hit
10:35:48 sean-k-mooney it will be in the data revvived with teh ohte rcompute node values in the same sql query
10:36:39 sean-k-mooney jkulik: gibi and i debugged this a month or two ago but didnt have time tow rok on proposing the fix
10:38:14 sean-k-mooney basiclly the plan to make this "just work" across upgrades is to make the rpc consditional and to get the uptime value only if its not in the stats blob we get form teh db
10:55:15 jkulik sean-k-mooney: thanks for sharing. I'll abandon my RR then.
10:55:47 jkulik sean-k-mooney: so this means, all drivers currently not implementing get_host_uptime() should at least provide a None value into the stats to not still hit the performance problem, right?
11:02:27 sean-k-mooney yep that or we could jsut remove the fallback entirly
11:02:44 sean-k-mooney jkulik: we breifly dicsused just removing uptime form the api in a new microversion
11:03:00 sean-k-mooney we kind of wanted to get wider input on that before moving forward however
11:03:43 sean-k-mooney jkulik: we orginally wanted to remove uptime in the orgial spec but some operator wanted use to keep it so we did
11:03:52 sean-k-mooney but im not sure that was the right choice
11:04:31 sean-k-mooney jkulik: does vmware supprot uptime at all
11:04:52 sean-k-mooney jkulik: we could also make the fallback to rpc condtional on if the driver suprot it
11:05:01 jkulik sean-k-mooney: no, not at all, because in VMware we have a whole cluster as hypervisor in Nova and each node in the cluster has different uptime
11:05:02 sean-k-mooney and then just exclude vmware if it does not
11:05:32 jkulik we're also introducing libvirt, so I'd still like to get the details to become fast ;)
11:05:36 sean-k-mooney right so we know the hypervior type so we coudl limit the fallback to just libvirt(qemu)
11:06:36 sean-k-mooney so for all ohter driver we coudl retrun None
11:06:45 sean-k-mooney if not in stats
11:07:39 jkulik ... or patch everyone to return None, which would be fine for me, too
11:08:07 sean-k-mooney well again we propsoed removing uptime when we removed the hypervior stats api
11:08:26 sean-k-mooney and some folks objected to that so we cant actully do that without a new microversion
11:08:52 sean-k-mooney i.e. it woudl break our api rules ot just stop reutring without a new microversion
11:09:19 sean-k-mooney but instead of a config opiotn https://review.opendev.org/c/openstack/nova/+/959420/2/nova/api/openstack/compute/hypervisors.py#105
11:09:40 sean-k-mooney you coudl have checked if the hypervior type was qemu and only get the uptime if it was
11:10:20 sean-k-mooney evenutaly we want to also stash it in the stats to make libvirt fast but that would fix vmware for now
11:10:50 jkulik I mean, we have the patch downstream. there's no pressure from our side. I'd go with the proper "put in stats" solution.
11:11:29 sean-k-mooney ack. we may want to put this in the ptg adjenda to not forget or have a dedicated bug for it
11:12:07 sean-k-mooney https://bugs.launchpad.net/nova/+bug/2106380 is about server list not the hyperviors api so we sould avoid resuing it
11:12:25 sean-k-mooney with that said i wonder if gibi already filed an upstream bug
11:38:28 opendevreview Rajesh Tailor proposed openstack/nova stable/2024.2: Fix 'nova-manage image_property set' command https://review.opendev.org/c/openstack/nova/+/959429
11:41:43 gibi bauzas: jkulik: the uptime issue ring bells to me but I think it was only reported downstream. I agree that this needs to be fixed
11:42:04 gibi until then the WA is using older microversion
11:43:09 gibi jkulik: yeah a config driven API is a no-no. But we can revisit why we added the uptime to the main GET call, or we can try to speed it up via scatter gather, or revers the relationship and send up the uptime data to the DB periodicall from the compute so reading it from the API will be quick
11:43:21 gibi hm that latter rings another bell that this was dansmith's idea
11:43:29 gibi so we might discussed this already upstream as well
11:48:22 gibi aand there is another semi related thread of sending that uptime RPC to computes that are down, but not yet marked as down in the DB is another source of slowness
11:50:09 jkulik I'd go with the "sending to the DB" - not sure how to solve that for computes being down, though.
11:51:05 gibi jkulik: we already have the periodic heartbeat to know when a compute is down, so we can use that info to avoid reporting back a stale uptime data on the API
12:07:09 gibi I did not find this uptime discussion in my IRC logs so this probably happened downstream. I will file a bug right away summarizing both problems and the solution suggestion
12:16:34 gibi aand I found the downstream report https://issues.redhat.com/browse/OSPRH-17563 so we probably have motivation to try to fix it :)
12:23:01 gibi field the bug https://bugs.launchpad.net/nova/+bug/2122036
12:25:30 gibi mmessah: ^^
12:39:21 mmessah Thanks for discussing the situation fastly :) I hope it will be resolved as soon as possible. We recently upgraded our test environment to Epoxy. We will continue to test all services. If we have any situation, we will share IRC.
12:41:29 gibi mmessah: thanks
12:42:55 opendevreview Rajesh Tailor proposed openstack/nova stable/2024.2: Fix case-sensitivity for metadata keys https://review.opendev.org/c/openstack/nova/+/959435
12:49:04 gibi dansmith: sean-k-mooney: when you are referring to "large-ops" job from the past (in relation to scale testing eventlet removal) do you mean https://review.opendev.org/c/openstack/project-config/+/278978 ?
12:50:53 sean-k-mooney i was not refering to that because i did not know that existed
12:50:58 sean-k-mooney but maybe dan was
12:51:20 gibi sean-k-mooney: ahh sorry I did not remember where this input coming from :)
12:51:21 sean-k-mooney gibi: i have just mention that kolla and devstack can fake may compute nodes per phsyical host
12:51:41 sean-k-mooney my expeirnce of this was as part of the osci lab scale testing
12:52:05 sean-k-mooney where intel and rackspace gave access to i think 500? baremental host half x86 the rest ppc
12:52:13 sean-k-mooney for the comutnity to do some scale testing
12:54:02 sean-k-mooney gibi: i wonder if the rally jobs have any existing usage of the fake agents
12:54:24 sean-k-mooney or if the tempest jobs in rally ieslf are usign real agents
12:55:39 gibi yeah rally is also on my radar to look into, especially neutron's jobs
12:55:56 sean-k-mooney so rally and browbeat are the two scaling frameworks we have
12:56:09 sean-k-mooney at leats those are teh two i am vagley aware of
12:56:46 gibi yeah, I never saw a realy browbeat scenario but heard the name before
12:57:36 sean-k-mooney it might be a wrapper arount rally looking at the repo
12:57:43 sean-k-mooney https://opendev.org/x/browbeat/
12:58:01 sean-k-mooney it has no ci jobs so that wont help us
12:58:07 bauzas gibi: sean-k-mooney: well, I think we should quickly remove the uptime call IMHO
12:58:09 gibi ohh brownbeat referst to "generate_tripleo_inventory"
12:58:18 gibi that turns me down :)
12:58:34 gibi bauzas: in a new microversion? in a non backportable way?
12:58:40 bauzas if you look how this works, then we call the driver for every compute, and then for example the driver directly calls the OS
12:59:00 bauzas so when you have 10000 hypervisors, then each of them calls the OS directly
12:59:12 sean-k-mooney looks like rally does not have any devstack jobs https://github.com/openstack/rally/blob/master/.zuul.d/zuul.yaml although im not sure wht rallty tox-functional does
12:59:42 bauzas gibi: I don't know how to fix this, I'd like to discuss that with gmaan and dansmith
12:59:54 sean-k-mooney bauzas: that would require an api micoversion
13:00:15 bauzas because when I see https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L12319-L12322 I'm pretty done
13:00:31 bauzas sean-k-mooney: well, if that's due to a bug, maybe not
13:00:41 sean-k-mooney bauzas: its not
13:00:49 sean-k-mooney stephen wanted to remove it in the spec
13:00:50 bauzas like, HTTP500 > HTTP200 doesn't need to have a new microversion
13:00:55 sean-k-mooney and we told them no
13:01:05 sean-k-mooney becuase ops asked for it to be kept
13:01:12 bauzas here, we don't have a HTTP500, but it can take more than one minute for getting a response
13:01:23 sean-k-mooney that does nto mean we cant make it better
13:02:00 gibi HTTP500 -> HTTP200 only accepted in a bugfix in my eyes if it still provides the data that the non HTP500 case provides
13:02:03 sean-k-mooney https://specs.openstack.org/openstack/nova-specs/specs/wallaby/implemented/modernize-os-hypervisors-api.html#problem-description
13:02:14 bauzas I saw it
13:02:25 bauzas already and I saw the implementation change too
13:02:40 sean-k-mooney yes htis was done yesar ago to be clear its not recent

Earlier   Later