Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-09-04
11:04:31 sean-k-mooney jkulik: does vmware supprot uptime at all
11:04:52 sean-k-mooney jkulik: we could also make the fallback to rpc condtional on if the driver suprot it
11:05:01 jkulik sean-k-mooney: no, not at all, because in VMware we have a whole cluster as hypervisor in Nova and each node in the cluster has different uptime
11:05:02 sean-k-mooney and then just exclude vmware if it does not
11:05:32 jkulik we're also introducing libvirt, so I'd still like to get the details to become fast ;)
11:05:36 sean-k-mooney right so we know the hypervior type so we coudl limit the fallback to just libvirt(qemu)
11:06:36 sean-k-mooney so for all ohter driver we coudl retrun None
11:06:45 sean-k-mooney if not in stats
11:07:39 jkulik ... or patch everyone to return None, which would be fine for me, too
11:08:07 sean-k-mooney well again we propsoed removing uptime when we removed the hypervior stats api
11:08:26 sean-k-mooney and some folks objected to that so we cant actully do that without a new microversion
11:08:52 sean-k-mooney i.e. it woudl break our api rules ot just stop reutring without a new microversion
11:09:19 sean-k-mooney but instead of a config opiotn https://review.opendev.org/c/openstack/nova/+/959420/2/nova/api/openstack/compute/hypervisors.py#105
11:09:40 sean-k-mooney you coudl have checked if the hypervior type was qemu and only get the uptime if it was
11:10:20 sean-k-mooney evenutaly we want to also stash it in the stats to make libvirt fast but that would fix vmware for now
11:10:50 jkulik I mean, we have the patch downstream. there's no pressure from our side. I'd go with the proper "put in stats" solution.
11:11:29 sean-k-mooney ack. we may want to put this in the ptg adjenda to not forget or have a dedicated bug for it
11:12:07 sean-k-mooney https://bugs.launchpad.net/nova/+bug/2106380 is about server list not the hyperviors api so we sould avoid resuing it
11:12:25 sean-k-mooney with that said i wonder if gibi already filed an upstream bug
11:38:28 opendevreview Rajesh Tailor proposed openstack/nova stable/2024.2: Fix 'nova-manage image_property set' command https://review.opendev.org/c/openstack/nova/+/959429
11:41:43 gibi bauzas: jkulik: the uptime issue ring bells to me but I think it was only reported downstream. I agree that this needs to be fixed
11:42:04 gibi until then the WA is using older microversion
11:43:09 gibi jkulik: yeah a config driven API is a no-no. But we can revisit why we added the uptime to the main GET call, or we can try to speed it up via scatter gather, or revers the relationship and send up the uptime data to the DB periodicall from the compute so reading it from the API will be quick
11:43:21 gibi hm that latter rings another bell that this was dansmith's idea
11:43:29 gibi so we might discussed this already upstream as well
11:48:22 gibi aand there is another semi related thread of sending that uptime RPC to computes that are down, but not yet marked as down in the DB is another source of slowness
11:50:09 jkulik I'd go with the "sending to the DB" - not sure how to solve that for computes being down, though.
11:51:05 gibi jkulik: we already have the periodic heartbeat to know when a compute is down, so we can use that info to avoid reporting back a stale uptime data on the API
12:07:09 gibi I did not find this uptime discussion in my IRC logs so this probably happened downstream. I will file a bug right away summarizing both problems and the solution suggestion
12:16:34 gibi aand I found the downstream report https://issues.redhat.com/browse/OSPRH-17563 so we probably have motivation to try to fix it :)
12:23:01 gibi field the bug https://bugs.launchpad.net/nova/+bug/2122036
12:25:30 gibi mmessah: ^^
12:39:21 mmessah Thanks for discussing the situation fastly :) I hope it will be resolved as soon as possible. We recently upgraded our test environment to Epoxy. We will continue to test all services. If we have any situation, we will share IRC.
12:41:29 gibi mmessah: thanks
12:42:55 opendevreview Rajesh Tailor proposed openstack/nova stable/2024.2: Fix case-sensitivity for metadata keys https://review.opendev.org/c/openstack/nova/+/959435
12:49:04 gibi dansmith: sean-k-mooney: when you are referring to "large-ops" job from the past (in relation to scale testing eventlet removal) do you mean https://review.opendev.org/c/openstack/project-config/+/278978 ?
12:50:53 sean-k-mooney i was not refering to that because i did not know that existed
12:50:58 sean-k-mooney but maybe dan was
12:51:20 gibi sean-k-mooney: ahh sorry I did not remember where this input coming from :)
12:51:21 sean-k-mooney gibi: i have just mention that kolla and devstack can fake may compute nodes per phsyical host
12:51:41 sean-k-mooney my expeirnce of this was as part of the osci lab scale testing
12:52:05 sean-k-mooney where intel and rackspace gave access to i think 500? baremental host half x86 the rest ppc
12:52:13 sean-k-mooney for the comutnity to do some scale testing
12:54:02 sean-k-mooney gibi: i wonder if the rally jobs have any existing usage of the fake agents
12:54:24 sean-k-mooney or if the tempest jobs in rally ieslf are usign real agents
12:55:39 gibi yeah rally is also on my radar to look into, especially neutron's jobs
12:55:56 sean-k-mooney so rally and browbeat are the two scaling frameworks we have
12:56:09 sean-k-mooney at leats those are teh two i am vagley aware of
12:56:46 gibi yeah, I never saw a realy browbeat scenario but heard the name before
12:57:36 sean-k-mooney it might be a wrapper arount rally looking at the repo
12:57:43 sean-k-mooney https://opendev.org/x/browbeat/
12:58:01 sean-k-mooney it has no ci jobs so that wont help us
12:58:07 bauzas gibi: sean-k-mooney: well, I think we should quickly remove the uptime call IMHO
12:58:09 gibi ohh brownbeat referst to "generate_tripleo_inventory"
12:58:18 gibi that turns me down :)
12:58:34 gibi bauzas: in a new microversion? in a non backportable way?
12:58:40 bauzas if you look how this works, then we call the driver for every compute, and then for example the driver directly calls the OS
12:59:00 bauzas so when you have 10000 hypervisors, then each of them calls the OS directly
12:59:12 sean-k-mooney looks like rally does not have any devstack jobs https://github.com/openstack/rally/blob/master/.zuul.d/zuul.yaml although im not sure wht rallty tox-functional does
12:59:42 bauzas gibi: I don't know how to fix this, I'd like to discuss that with gmaan and dansmith
12:59:54 sean-k-mooney bauzas: that would require an api micoversion
13:00:15 bauzas because when I see https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L12319-L12322 I'm pretty done
13:00:31 bauzas sean-k-mooney: well, if that's due to a bug, maybe not
13:00:41 sean-k-mooney bauzas: its not
13:00:49 sean-k-mooney stephen wanted to remove it in the spec
13:00:50 bauzas like, HTTP500 > HTTP200 doesn't need to have a new microversion
13:00:55 sean-k-mooney and we told them no
13:01:05 sean-k-mooney becuase ops asked for it to be kept
13:01:12 bauzas here, we don't have a HTTP500, but it can take more than one minute for getting a response
13:01:23 sean-k-mooney that does nto mean we cant make it better
13:02:00 gibi HTTP500 -> HTTP200 only accepted in a bugfix in my eyes if it still provides the data that the non HTP500 case provides
13:02:03 sean-k-mooney https://specs.openstack.org/openstack/nova-specs/specs/wallaby/implemented/modernize-os-hypervisors-api.html#problem-description
13:02:14 bauzas I saw it
13:02:25 bauzas already and I saw the implementation change too
13:02:40 sean-k-mooney yes htis was done yesar ago to be clear its not recent
13:02:47 sean-k-mooney it has been this way since wallaby
13:02:52 gibi I agree that we need to fix it but maybe we can fix it in a way that preserve the functionality without the performance hit
13:02:55 bauzas but the problem is now we call the OS for all of the hosts automatically
13:03:00 bauzas instead of a specific API
13:03:13 sean-k-mooney you opt into it with the micorversion
13:03:17 sean-k-mooney we dont if you use the old one
13:03:25 bauzas sure
13:03:29 sean-k-mooney but there is a way to achive both
13:03:46 sean-k-mooney first for any host that is not using the qemu virt_type
13:03:51 sean-k-mooney we canc skip the call entirly
13:03:55 bauzas but atm, 2.88 has a performance problem, right?
13:03:57 sean-k-mooney and return none
13:04:18 sean-k-mooney and then we can start adding the uptime to the compute node stats
13:04:31 sean-k-mooney jand if its in the stats we can also skip it even for libvirt
13:04:45 gibi (ahh the idea was from sean-k-mooney not from dansmith sorry for the misattribution above)
13:04:47 sean-k-mooney finelly if we want to remove it in the api entirly we can do that in a new microverison
13:05:47 jkulik instead of having nova-compute call out into the OS on every HV, we could add it once on startup and add a timestamp to it. the API call can then just add `time.time() - timestamp` to the originally reported uptime :)
13:05:48 sean-k-mooney so we have 2 backportable fixes (only do the rpc if the diver supprot it and cache the value in the state generate via update_aviable_resouces) and one non backportable fix removing it in a new micorverion
13:06:10 sean-k-mooney jkulik: well we can also do that too yes
13:06:18 sean-k-mooney but that woudl be diffent
13:06:20 jkulik then we don't need a regular task
13:06:30 jkulik and also no rpc call
13:06:31 sean-k-mooney we check the host so the value does not go backward if you restart the agent on the host
13:06:48 sean-k-mooney jkulik: we have a perodic that is updating the stats already
13:07:04 sean-k-mooney we woudl jus tbe making it add 1 more key to the json blob

Earlier   Later