| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-04 | |||
| 13:02:52 | gibi | I agree that we need to fix it but maybe we can fix it in a way that preserve the functionality without the performance hit | |
| 13:02:55 | bauzas | but the problem is now we call the OS for all of the hosts automatically | |
| 13:03:00 | bauzas | instead of a specific API | |
| 13:03:13 | sean-k-mooney | you opt into it with the micorversion | |
| 13:03:17 | sean-k-mooney | we dont if you use the old one | |
| 13:03:25 | bauzas | sure | |
| 13:03:29 | sean-k-mooney | but there is a way to achive both | |
| 13:03:46 | sean-k-mooney | first for any host that is not using the qemu virt_type | |
| 13:03:51 | sean-k-mooney | we canc skip the call entirly | |
| 13:03:55 | bauzas | but atm, 2.88 has a performance problem, right? | |
| 13:03:57 | sean-k-mooney | and return none | |
| 13:04:18 | sean-k-mooney | and then we can start adding the uptime to the compute node stats | |
| 13:04:31 | sean-k-mooney | jand if its in the stats we can also skip it even for libvirt | |
| 13:04:45 | gibi | (ahh the idea was from sean-k-mooney not from dansmith sorry for the misattribution above) | |
| 13:04:47 | sean-k-mooney | finelly if we want to remove it in the api entirly we can do that in a new microverison | |
| 13:05:47 | jkulik | instead of having nova-compute call out into the OS on every HV, we could add it once on startup and add a timestamp to it. the API call can then just add `time.time() - timestamp` to the originally reported uptime :) | |
| 13:05:48 | sean-k-mooney | so we have 2 backportable fixes (only do the rpc if the diver supprot it and cache the value in the state generate via update_aviable_resouces) and one non backportable fix removing it in a new micorverion | |
| 13:06:10 | sean-k-mooney | jkulik: well we can also do that too yes | |
| 13:06:18 | sean-k-mooney | but that woudl be diffent | |
| 13:06:20 | jkulik | then we don't need a regular task | |
| 13:06:30 | jkulik | and also no rpc call | |
| 13:06:31 | sean-k-mooney | we check the host so the value does not go backward if you restart the agent on the host | |
| 13:06:48 | sean-k-mooney | jkulik: we have a perodic that is updating the stats already | |
| 13:07:04 | sean-k-mooney | we woudl jus tbe making it add 1 more key to the json blob | |
| 13:07:08 | sean-k-mooney | so that pretty much free | |
| 13:07:10 | jkulik | but is it calling an process every time? sounded to me like someone didn't like that very much | |
| 13:07:37 | sean-k-mooney | if we want ot change the meaning to be the compute agent uptime instgead of the host uptime we coudl do the time.time()-timestamp change as well | |
| 13:07:54 | jkulik | out, err = processutils.execute('env', 'LANG=C', 'uptime') | |
| 13:08:24 | jkulik | we call that on agent startup and add the time since agent startup -> meaning stays the same | |
| 13:08:24 | sean-k-mooney | well we can get that info form libivrt too i think or there are other ways to do that hten calling the uptime command | |
| 13:08:41 | sean-k-mooney | that not what uptime is | |
| 13:08:49 | sean-k-mooney | the uptime is nto the uptime since the agent started | |
| 13:08:54 | sean-k-mooney | its the uptiem of the host | |
| 13:09:04 | sean-k-mooney | and if you restart nova-compute without a reboot those are not the same | |
| 13:09:13 | jkulik | yes. hence we add the agent-uptime to the initially-read host uptime | |
| 13:09:33 | sean-k-mooney | or we jsut dont provide this in the furue since you shoudl not be getting this form nova at all | |
| 13:09:42 | jkulik | fine by me :D | |
| 13:10:06 | sean-k-mooney | in wallaby we really wanted to remvoe this from nova but opt wanted ot keep it instead of pullling it ofmr a monitoring tool liek prometheus or collectd | |
| 13:10:41 | sean-k-mooney | the compromies is we can provide a cached value form the existing perodic which is basiclly free | |
| 13:10:55 | sean-k-mooney | for older microversion and delete this in a future one | |
| 13:12:09 | jkulik | sounded like bauzas wouldn't like calling `uptime` in the periodic on all hosts all the time. so we'd also need a cheaper replacement for that. | |
| 13:12:14 | sean-k-mooney | the imprant thing it so fix the perfoamnce reguression. we could also choose in a new microversion to make the retivial fo the uptime contoleable by a query arg and default to not but that fells overly complicaterd | |
| 13:12:28 | sean-k-mooney | jkulik: well we can get that form libvirt as well im pretty sure | |
| 13:12:41 | sean-k-mooney | but that is an entirly seperate issue | |
| 13:12:53 | sean-k-mooney | nova has alwasy done that | |
| 13:13:05 | sean-k-mooney | it was not a regression intorduced by 2.88 | |
| 13:13:28 | jkulik | periodically? only on request, right? and we're going to change how often Nova calls that, if we move it to the periodic task | |
| 13:13:50 | jkulik | it's probably™ not much overhead, so we could also fix that later if we see it being a problem. | |
| 13:14:37 | sean-k-mooney | jkulik: my suggestion is to only update it when the update aviable resouce perodic runs | |
| 13:14:44 | sean-k-mooney | which is every 5 mins | |
| 13:14:56 | sean-k-mooney | we coudl stash the start time of the agent in the stats instead | |
| 13:15:03 | sean-k-mooney | adn compute the uptime in the api | |
| 13:15:03 | bauzas | sean-k-mooney: maybe I'm not right, but https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L206 calls _view_hypervisor for each of the hypervisors lisrt | |
| 13:15:53 | bauzas | then in https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L97-L101 we unconditionnally call the compute API to get the uptime https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L97-L101 | |
| 13:16:20 | sean-k-mooney | bauzas: the rpc happens if you do a detail list or i belive hypervior show with 2.88+ | |
| 13:16:32 | bauzas | not only show right? | |
| 13:16:37 | bauzas | that's my point | |
| 13:16:48 | sean-k-mooney | right its in detail list as well | |
| 13:16:56 | sean-k-mooney | which is what was approvded in the spec | |
| 13:17:26 | bauzas | so for a detailed list of HVs using 2.88, we then call every compute for getting the uptime, right? | |
| 13:17:33 | sean-k-mooney | yep | |
| 13:18:00 | bauzas | each of those computes then eventually calls the driver, but in case of libvirt, it calls an OS process for getting uptime | |
| 13:18:24 | sean-k-mooney | yep | |
| 13:18:34 | sean-k-mooney | no one is saying this si good or scalable | |
| 13:18:40 | sean-k-mooney | but its not an api bug | |
| 13:18:43 | bauzas | so that means that if I have a list of 1000 computes all of them running libvirt, then I'd get 1000 RPC calls and each of them would access the OS before returning | |
| 13:18:59 | bauzas | that's what I call not scalable per say | |
| 13:19:10 | gibi | (I think we all agree that it is bad) | |
| 13:19:41 | sean-k-mooney | right we could also speed it up with sater gather but unless we want to break our api contract we cant fix it by just not including it | |
| 13:19:43 | gibi | we cannot just rip it out from 2.88 something that works for 10 computes but not for 1000 | |
| 13:19:55 | bauzas | previously we were not doing that in 2.87 (calling N RPC calls), that's why I can't say this is not a regression | |
| 13:20:03 | bauzas | if you have to opt in, for sure | |
| 13:21:24 | bauzas | MHO is that either we throw 2.88 as non usable by fixing it in another microversion, or we try somehow to fix 2.88 | |
| 13:21:45 | bauzas | my preference would be the latter for downstream reasons :) | |
| 13:21:54 | sean-k-mooney | bauzas: look im just going to go fix this becuase im -2 on just riping it ou8t | |
| 13:22:27 | bauzas | sean-k-mooney: I'm just awaiting gmaan and dansmith | |
| 13:22:30 | sean-k-mooney | ill file a bug and work on it this/next week | |
| 13:22:46 | dansmith | sorry, I'm on another call | |
| 13:22:59 | bauzas | there are multiple ways to adress that, but I'd rather prefer if we could, as a team, find a solution that we all agree | |
| 13:23:05 | bauzas | before even working on a fix | |
| 13:23:07 | jkulik | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/2122036 bug is filed already | |
| 13:23:24 | bauzas | dansmith: doh, I should attend that call too | |
| 13:23:25 | sean-k-mooney | jkulik: ah perfect thanks gibi | |
| 13:23:41 | sean-k-mooney | oh right | |
| 13:25:22 | sean-k-mooney | jkulik: we already depend on psutiil in nova so i can do time.time() - psutil.boot_time() adn cahce it on the first invocation | |
| 13:25:44 | sean-k-mooney | well i can cache the boot time part | |
| 13:25:53 | jkulik | nice | |
| 13:26:47 | gibi | sean-k-mooney: btw, devstack with fake driver works like a charm. I just rebuilt my two VM devstack with 5 fake compute in each VM | |
| 13:27:11 | sean-k-mooney | gibi: cool its been a very long time since i used it so im glad it still works | |
| 13:28:40 | opendevreview | ribaudr proposed openstack/nova stable/2025.1: Reproducer for bug 2114951 https://review.opendev.org/c/openstack/nova/+/959445 | |
| 13:28:41 | opendevreview | ribaudr proposed openstack/nova stable/2025.1: Fix bug 2114951 https://review.opendev.org/c/openstack/nova/+/959446 | |
| 13:29:36 | sean-k-mooney | apprently time.clock_gettime(time.CLOCK_BOOTTIME) os also a thing without the psutils dep on py 3.7+ | |
| 13:30:42 | sean-k-mooney | we dont have many deps on psutils so if we can aovid ading more that woudl eb good | |
| 13:34:21 | gibi | (and just booted 200 fake VMs successfully) | |
| 13:34:50 | sean-k-mooney | did hte fake networkign agent work by the way or did you do that without networking | |
| 13:35:19 | sean-k-mooney | ironic has a fake driver too but im not sure about cidner | |
| 13:36:13 | sean-k-mooney | glance kind of does not need it as you can just have tiny or even 0 byte images | |
| 13:36:27 | gibi | just used --nic none | |
| 13:36:32 | sean-k-mooney | same with keystone and placment they just talk to a db | |
| 13:37:05 | sean-k-mooney | although they dont use eventlet so keystone and palcment are not really relevnet to testing nova at scale | |