| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-04 | |||
| 13:15:03 | bauzas | sean-k-mooney: maybe I'm not right, but https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L206 calls _view_hypervisor for each of the hypervisors lisrt | |
| 13:15:53 | bauzas | then in https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L97-L101 we unconditionnally call the compute API to get the uptime https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L97-L101 | |
| 13:16:20 | sean-k-mooney | bauzas: the rpc happens if you do a detail list or i belive hypervior show with 2.88+ | |
| 13:16:32 | bauzas | not only show right? | |
| 13:16:37 | bauzas | that's my point | |
| 13:16:48 | sean-k-mooney | right its in detail list as well | |
| 13:16:56 | sean-k-mooney | which is what was approvded in the spec | |
| 13:17:26 | bauzas | so for a detailed list of HVs using 2.88, we then call every compute for getting the uptime, right? | |
| 13:17:33 | sean-k-mooney | yep | |
| 13:18:00 | bauzas | each of those computes then eventually calls the driver, but in case of libvirt, it calls an OS process for getting uptime | |
| 13:18:24 | sean-k-mooney | yep | |
| 13:18:34 | sean-k-mooney | no one is saying this si good or scalable | |
| 13:18:40 | sean-k-mooney | but its not an api bug | |
| 13:18:43 | bauzas | so that means that if I have a list of 1000 computes all of them running libvirt, then I'd get 1000 RPC calls and each of them would access the OS before returning | |
| 13:18:59 | bauzas | that's what I call not scalable per say | |
| 13:19:10 | gibi | (I think we all agree that it is bad) | |
| 13:19:41 | sean-k-mooney | right we could also speed it up with sater gather but unless we want to break our api contract we cant fix it by just not including it | |
| 13:19:43 | gibi | we cannot just rip it out from 2.88 something that works for 10 computes but not for 1000 | |
| 13:19:55 | bauzas | previously we were not doing that in 2.87 (calling N RPC calls), that's why I can't say this is not a regression | |
| 13:20:03 | bauzas | if you have to opt in, for sure | |
| 13:21:24 | bauzas | MHO is that either we throw 2.88 as non usable by fixing it in another microversion, or we try somehow to fix 2.88 | |
| 13:21:45 | bauzas | my preference would be the latter for downstream reasons :) | |
| 13:21:54 | sean-k-mooney | bauzas: look im just going to go fix this becuase im -2 on just riping it ou8t | |
| 13:22:27 | bauzas | sean-k-mooney: I'm just awaiting gmaan and dansmith | |
| 13:22:30 | sean-k-mooney | ill file a bug and work on it this/next week | |
| 13:22:46 | dansmith | sorry, I'm on another call | |
| 13:22:59 | bauzas | there are multiple ways to adress that, but I'd rather prefer if we could, as a team, find a solution that we all agree | |
| 13:23:05 | bauzas | before even working on a fix | |
| 13:23:07 | jkulik | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/2122036 bug is filed already | |
| 13:23:24 | bauzas | dansmith: doh, I should attend that call too | |
| 13:23:25 | sean-k-mooney | jkulik: ah perfect thanks gibi | |
| 13:23:41 | sean-k-mooney | oh right | |
| 13:25:22 | sean-k-mooney | jkulik: we already depend on psutiil in nova so i can do time.time() - psutil.boot_time() adn cahce it on the first invocation | |
| 13:25:44 | sean-k-mooney | well i can cache the boot time part | |
| 13:25:53 | jkulik | nice | |
| 13:26:47 | gibi | sean-k-mooney: btw, devstack with fake driver works like a charm. I just rebuilt my two VM devstack with 5 fake compute in each VM | |
| 13:27:11 | sean-k-mooney | gibi: cool its been a very long time since i used it so im glad it still works | |
| 13:28:40 | opendevreview | ribaudr proposed openstack/nova stable/2025.1: Reproducer for bug 2114951 https://review.opendev.org/c/openstack/nova/+/959445 | |
| 13:28:41 | opendevreview | ribaudr proposed openstack/nova stable/2025.1: Fix bug 2114951 https://review.opendev.org/c/openstack/nova/+/959446 | |
| 13:29:36 | sean-k-mooney | apprently time.clock_gettime(time.CLOCK_BOOTTIME) os also a thing without the psutils dep on py 3.7+ | |
| 13:30:42 | sean-k-mooney | we dont have many deps on psutils so if we can aovid ading more that woudl eb good | |
| 13:34:21 | gibi | (and just booted 200 fake VMs successfully) | |
| 13:34:50 | sean-k-mooney | did hte fake networkign agent work by the way or did you do that without networking | |
| 13:35:19 | sean-k-mooney | ironic has a fake driver too but im not sure about cidner | |
| 13:36:13 | sean-k-mooney | glance kind of does not need it as you can just have tiny or even 0 byte images | |
| 13:36:27 | gibi | just used --nic none | |
| 13:36:32 | sean-k-mooney | same with keystone and placment they just talk to a db | |
| 13:37:05 | sean-k-mooney | although they dont use eventlet so keystone and palcment are not really relevnet to testing nova at scale | |
| 13:37:20 | sean-k-mooney | at least not in this context | |
| 13:43:24 | jkulik | regarding scale: we have a patch downstream that makes the server-group listing much faster with lots of server-groups around (e.g. when an admin lists them all). We changed the DB queries to actually do the pagination instead of doing it after fetching _all_ DB objects. | |
| 13:43:52 | jkulik | Would that be something I can propose just like that - with a bug-report - or does that need some special process? | |
| 13:45:03 | dansmith | bauzas: what do you want me to look at? | |
| 13:45:17 | sean-k-mooney | jkulik: yes for master, it would depend on the actual cahnge if it woudl be backprotable | |
| 13:45:41 | bauzas | not specifically looking at, just being there for accepting a solution | |
| 13:46:02 | jkulik | sean-k-mooney: I don't need backports, as we have it running. would just be nice to share if possible :) | |
| 13:46:08 | sean-k-mooney | jkulik: its alwasy valid to reprot performance bug and fix them on master. weither its backproabel will depend on if your using any feature form newer sqlachme ectra or if ti has any other upgrade impact | |
| 13:46:23 | bauzas | tl;dr: by 2.88, we now call every compute if you call hypervisor detailed list and we await the compute OS for returning us an uptime | |
| 13:46:47 | dansmith | bauzas: oh yeah I read the bug | |
| 13:52:00 | dansmith | bauzas: seems like the solution there _has_ to be to start reporting uptime somewhere we can persist it and fix that to query that location instead of making the (egads) rpc call | |
| 13:53:21 | bauzas | something that could then be backportable to Epoxy if we can | |
| 13:53:22 | dansmith | I agree we can't just stop doing it, and making it only present if you don't list, or don't list enough is also not great | |
| 13:53:51 | bauzas | if we can't backport it, then 2.88 will have the same issue | |
| 13:54:23 | sean-k-mooney | the hypervior api is already doing an sqlquery to the compute nodes db table and geting the stats | |
| 13:54:45 | sean-k-mooney | we can put it in that json bob which is entirly unversioned today | |
| 13:54:55 | sean-k-mooney | so if we want to backprot it we can | |
| 13:55:10 | dansmith | yeah was just going to say, ComputeNode has several places we could stash it I think | |
| 13:55:41 | dansmith | but, it will be absent during an upgrade so we probably need to fall back to the old behavior for nodes that aren't upgraded | |
| 13:55:59 | sean-k-mooney | the bit im currntly checking is it purly the time or are we includign the load info too | |
| 13:56:32 | sean-k-mooney | im hoping it just the former but some of our docs implies the latter | |
| 13:57:47 | dansmith | doesn't much matter | |
| 13:58:00 | sean-k-mooney | well its just changes what we need to stash | |
| 13:58:14 | sean-k-mooney | but ya it does not matther in the long run | |
| 13:58:20 | sean-k-mooney | the depercated api https://docs.openstack.org/api-ref/compute/#show-hypervisor-uptime-deprecated | |
| 13:58:22 | bauzas | if we can stuff the uptime value somewhere, sure | |
| 13:58:28 | sean-k-mooney | was the rwaw output of uptime | |
| 13:58:41 | sean-k-mooney | but i think in 2.88 its just the time in second | |
| 13:58:41 | bauzas | without needing to have a new DB field, cool | |
| 13:59:01 | bauzas | sean-k-mooney: my concern is not with hypervisor show, again. | |
| 13:59:01 | sean-k-mooney | but i just need to check that on a master devstack to confirm | |
| 13:59:12 | sean-k-mooney | bauzas: yep i know | |
| 13:59:19 | bauzas | if we continue to call the OS for hypervisor show, meh | |
| 13:59:47 | bauzas | even if that's bad, that's not really an issue | |
| 14:00:18 | bauzas | but, if we are able to persist some uptime value for detailed-list, then of course we could modify hypervisor show | |
| 14:01:08 | jkulik | I don't see anything parsing the `uptime` command output, so it seems to be indeed the full one | |
| 14:01:17 | opendevreview | ribaudr proposed openstack/nova stable/2024.2: Reproducer for bug 2114951 https://review.opendev.org/c/openstack/nova/+/959525 | |
| 14:01:18 | opendevreview | ribaudr proposed openstack/nova stable/2024.2: Fix bug 2114951 https://review.opendev.org/c/openstack/nova/+/959526 | |
| 14:01:39 | sean-k-mooney | it is yes i just confirmed that | |
| 14:02:17 | sean-k-mooney | https://paste.opendev.org/show/bJ59n0G92NoP06hDsfAu/ | |
| 14:02:38 | jkulik | including a newline? :D | |
| 14:02:54 | sean-k-mooney | so i cant jsut retirn the tiem btu i can just stash the value in the stats and skip the rpc call if its there | |
| 14:03:06 | sean-k-mooney | and ya it also has a leadign space | |
| 14:03:43 | dansmith | can't what? | |
| 14:04:09 | sean-k-mooney | sorry, i cant just retive the time since boot and return that value in seconds | |
| 14:04:18 | bauzas | dansmith: given we return the whole uptime, we can't just calculate it | |
| 14:04:20 | sean-k-mooney | dansmith: jkulik suggeted doing that to aovid calling the uptime command | |
| 14:04:46 | sean-k-mooney | in a futre microverion we can jsut not in the backportabel fix | |
| 14:04:48 | dansmith | ack, sorry I just wasn't getting that meaning from the translation | |
| 14:05:02 | bauzas | this is honestly super weird to have that in our API | |
| 14:05:23 | sean-k-mooney | ya that was a pebkac, my keyboard was not aligent so i was typing off center | |
| 14:05:26 | bauzas | I don't disagree with sean-k-mooney when he said we should have stopped to provide it | |