| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-04 | |||
| 13:08:24 | sean-k-mooney | well we can get that info form libivrt too i think or there are other ways to do that hten calling the uptime command | |
| 13:08:41 | sean-k-mooney | that not what uptime is | |
| 13:08:49 | sean-k-mooney | the uptime is nto the uptime since the agent started | |
| 13:08:54 | sean-k-mooney | its the uptiem of the host | |
| 13:09:04 | sean-k-mooney | and if you restart nova-compute without a reboot those are not the same | |
| 13:09:13 | jkulik | yes. hence we add the agent-uptime to the initially-read host uptime | |
| 13:09:33 | sean-k-mooney | or we jsut dont provide this in the furue since you shoudl not be getting this form nova at all | |
| 13:09:42 | jkulik | fine by me :D | |
| 13:10:06 | sean-k-mooney | in wallaby we really wanted to remvoe this from nova but opt wanted ot keep it instead of pullling it ofmr a monitoring tool liek prometheus or collectd | |
| 13:10:41 | sean-k-mooney | the compromies is we can provide a cached value form the existing perodic which is basiclly free | |
| 13:10:55 | sean-k-mooney | for older microversion and delete this in a future one | |
| 13:12:09 | jkulik | sounded like bauzas wouldn't like calling `uptime` in the periodic on all hosts all the time. so we'd also need a cheaper replacement for that. | |
| 13:12:14 | sean-k-mooney | the imprant thing it so fix the perfoamnce reguression. we could also choose in a new microversion to make the retivial fo the uptime contoleable by a query arg and default to not but that fells overly complicaterd | |
| 13:12:28 | sean-k-mooney | jkulik: well we can get that form libvirt as well im pretty sure | |
| 13:12:41 | sean-k-mooney | but that is an entirly seperate issue | |
| 13:12:53 | sean-k-mooney | nova has alwasy done that | |
| 13:13:05 | sean-k-mooney | it was not a regression intorduced by 2.88 | |
| 13:13:28 | jkulik | periodically? only on request, right? and we're going to change how often Nova calls that, if we move it to the periodic task | |
| 13:13:50 | jkulik | it's probably™ not much overhead, so we could also fix that later if we see it being a problem. | |
| 13:14:37 | sean-k-mooney | jkulik: my suggestion is to only update it when the update aviable resouce perodic runs | |
| 13:14:44 | sean-k-mooney | which is every 5 mins | |
| 13:14:56 | sean-k-mooney | we coudl stash the start time of the agent in the stats instead | |
| 13:15:03 | sean-k-mooney | adn compute the uptime in the api | |
| 13:15:03 | bauzas | sean-k-mooney: maybe I'm not right, but https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L206 calls _view_hypervisor for each of the hypervisors lisrt | |
| 13:15:53 | bauzas | then in https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L97-L101 we unconditionnally call the compute API to get the uptime https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L97-L101 | |
| 13:16:20 | sean-k-mooney | bauzas: the rpc happens if you do a detail list or i belive hypervior show with 2.88+ | |
| 13:16:32 | bauzas | not only show right? | |
| 13:16:37 | bauzas | that's my point | |
| 13:16:48 | sean-k-mooney | right its in detail list as well | |
| 13:16:56 | sean-k-mooney | which is what was approvded in the spec | |
| 13:17:26 | bauzas | so for a detailed list of HVs using 2.88, we then call every compute for getting the uptime, right? | |
| 13:17:33 | sean-k-mooney | yep | |
| 13:18:00 | bauzas | each of those computes then eventually calls the driver, but in case of libvirt, it calls an OS process for getting uptime | |
| 13:18:24 | sean-k-mooney | yep | |
| 13:18:34 | sean-k-mooney | no one is saying this si good or scalable | |
| 13:18:40 | sean-k-mooney | but its not an api bug | |
| 13:18:43 | bauzas | so that means that if I have a list of 1000 computes all of them running libvirt, then I'd get 1000 RPC calls and each of them would access the OS before returning | |
| 13:18:59 | bauzas | that's what I call not scalable per say | |
| 13:19:10 | gibi | (I think we all agree that it is bad) | |
| 13:19:41 | sean-k-mooney | right we could also speed it up with sater gather but unless we want to break our api contract we cant fix it by just not including it | |
| 13:19:43 | gibi | we cannot just rip it out from 2.88 something that works for 10 computes but not for 1000 | |
| 13:19:55 | bauzas | previously we were not doing that in 2.87 (calling N RPC calls), that's why I can't say this is not a regression | |
| 13:20:03 | bauzas | if you have to opt in, for sure | |
| 13:21:24 | bauzas | MHO is that either we throw 2.88 as non usable by fixing it in another microversion, or we try somehow to fix 2.88 | |
| 13:21:45 | bauzas | my preference would be the latter for downstream reasons :) | |
| 13:21:54 | sean-k-mooney | bauzas: look im just going to go fix this becuase im -2 on just riping it ou8t | |
| 13:22:27 | bauzas | sean-k-mooney: I'm just awaiting gmaan and dansmith | |
| 13:22:30 | sean-k-mooney | ill file a bug and work on it this/next week | |
| 13:22:46 | dansmith | sorry, I'm on another call | |
| 13:22:59 | bauzas | there are multiple ways to adress that, but I'd rather prefer if we could, as a team, find a solution that we all agree | |
| 13:23:05 | bauzas | before even working on a fix | |
| 13:23:07 | jkulik | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/2122036 bug is filed already | |
| 13:23:24 | bauzas | dansmith: doh, I should attend that call too | |
| 13:23:25 | sean-k-mooney | jkulik: ah perfect thanks gibi | |
| 13:23:41 | sean-k-mooney | oh right | |
| 13:25:22 | sean-k-mooney | jkulik: we already depend on psutiil in nova so i can do time.time() - psutil.boot_time() adn cahce it on the first invocation | |
| 13:25:44 | sean-k-mooney | well i can cache the boot time part | |
| 13:25:53 | jkulik | nice | |
| 13:26:47 | gibi | sean-k-mooney: btw, devstack with fake driver works like a charm. I just rebuilt my two VM devstack with 5 fake compute in each VM | |
| 13:27:11 | sean-k-mooney | gibi: cool its been a very long time since i used it so im glad it still works | |
| 13:28:40 | opendevreview | ribaudr proposed openstack/nova stable/2025.1: Reproducer for bug 2114951 https://review.opendev.org/c/openstack/nova/+/959445 | |
| 13:28:41 | opendevreview | ribaudr proposed openstack/nova stable/2025.1: Fix bug 2114951 https://review.opendev.org/c/openstack/nova/+/959446 | |
| 13:29:36 | sean-k-mooney | apprently time.clock_gettime(time.CLOCK_BOOTTIME) os also a thing without the psutils dep on py 3.7+ | |
| 13:30:42 | sean-k-mooney | we dont have many deps on psutils so if we can aovid ading more that woudl eb good | |
| 13:34:21 | gibi | (and just booted 200 fake VMs successfully) | |
| 13:34:50 | sean-k-mooney | did hte fake networkign agent work by the way or did you do that without networking | |
| 13:35:19 | sean-k-mooney | ironic has a fake driver too but im not sure about cidner | |
| 13:36:13 | sean-k-mooney | glance kind of does not need it as you can just have tiny or even 0 byte images | |
| 13:36:27 | gibi | just used --nic none | |
| 13:36:32 | sean-k-mooney | same with keystone and placment they just talk to a db | |
| 13:37:05 | sean-k-mooney | although they dont use eventlet so keystone and palcment are not really relevnet to testing nova at scale | |
| 13:37:20 | sean-k-mooney | at least not in this context | |
| 13:43:24 | jkulik | regarding scale: we have a patch downstream that makes the server-group listing much faster with lots of server-groups around (e.g. when an admin lists them all). We changed the DB queries to actually do the pagination instead of doing it after fetching _all_ DB objects. | |
| 13:43:52 | jkulik | Would that be something I can propose just like that - with a bug-report - or does that need some special process? | |
| 13:45:03 | dansmith | bauzas: what do you want me to look at? | |
| 13:45:17 | sean-k-mooney | jkulik: yes for master, it would depend on the actual cahnge if it woudl be backprotable | |
| 13:45:41 | bauzas | not specifically looking at, just being there for accepting a solution | |
| 13:46:02 | jkulik | sean-k-mooney: I don't need backports, as we have it running. would just be nice to share if possible :) | |
| 13:46:08 | sean-k-mooney | jkulik: its alwasy valid to reprot performance bug and fix them on master. weither its backproabel will depend on if your using any feature form newer sqlachme ectra or if ti has any other upgrade impact | |
| 13:46:23 | bauzas | tl;dr: by 2.88, we now call every compute if you call hypervisor detailed list and we await the compute OS for returning us an uptime | |
| 13:46:47 | dansmith | bauzas: oh yeah I read the bug | |
| 13:52:00 | dansmith | bauzas: seems like the solution there _has_ to be to start reporting uptime somewhere we can persist it and fix that to query that location instead of making the (egads) rpc call | |
| 13:53:21 | bauzas | something that could then be backportable to Epoxy if we can | |
| 13:53:22 | dansmith | I agree we can't just stop doing it, and making it only present if you don't list, or don't list enough is also not great | |
| 13:53:51 | bauzas | if we can't backport it, then 2.88 will have the same issue | |
| 13:54:23 | sean-k-mooney | the hypervior api is already doing an sqlquery to the compute nodes db table and geting the stats | |
| 13:54:45 | sean-k-mooney | we can put it in that json bob which is entirly unversioned today | |
| 13:54:55 | sean-k-mooney | so if we want to backprot it we can | |
| 13:55:10 | dansmith | yeah was just going to say, ComputeNode has several places we could stash it I think | |
| 13:55:41 | dansmith | but, it will be absent during an upgrade so we probably need to fall back to the old behavior for nodes that aren't upgraded | |
| 13:55:59 | sean-k-mooney | the bit im currntly checking is it purly the time or are we includign the load info too | |
| 13:56:32 | sean-k-mooney | im hoping it just the former but some of our docs implies the latter | |
| 13:57:47 | dansmith | doesn't much matter | |
| 13:58:00 | sean-k-mooney | well its just changes what we need to stash | |
| 13:58:14 | sean-k-mooney | but ya it does not matther in the long run | |
| 13:58:20 | sean-k-mooney | the depercated api https://docs.openstack.org/api-ref/compute/#show-hypervisor-uptime-deprecated | |
| 13:58:22 | bauzas | if we can stuff the uptime value somewhere, sure | |
| 13:58:28 | sean-k-mooney | was the rwaw output of uptime | |
| 13:58:41 | sean-k-mooney | but i think in 2.88 its just the time in second | |
| 13:58:41 | bauzas | without needing to have a new DB field, cool | |