| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-04 | |||
| 12:55:39 | gibi | yeah rally is also on my radar to look into, especially neutron's jobs | |
| 12:55:56 | sean-k-mooney | so rally and browbeat are the two scaling frameworks we have | |
| 12:56:09 | sean-k-mooney | at leats those are teh two i am vagley aware of | |
| 12:56:46 | gibi | yeah, I never saw a realy browbeat scenario but heard the name before | |
| 12:57:36 | sean-k-mooney | it might be a wrapper arount rally looking at the repo | |
| 12:57:43 | sean-k-mooney | https://opendev.org/x/browbeat/ | |
| 12:58:01 | sean-k-mooney | it has no ci jobs so that wont help us | |
| 12:58:07 | bauzas | gibi: sean-k-mooney: well, I think we should quickly remove the uptime call IMHO | |
| 12:58:09 | gibi | ohh brownbeat referst to "generate_tripleo_inventory" | |
| 12:58:18 | gibi | that turns me down :) | |
| 12:58:34 | gibi | bauzas: in a new microversion? in a non backportable way? | |
| 12:58:40 | bauzas | if you look how this works, then we call the driver for every compute, and then for example the driver directly calls the OS | |
| 12:59:00 | bauzas | so when you have 10000 hypervisors, then each of them calls the OS directly | |
| 12:59:12 | sean-k-mooney | looks like rally does not have any devstack jobs https://github.com/openstack/rally/blob/master/.zuul.d/zuul.yaml although im not sure wht rallty tox-functional does | |
| 12:59:42 | bauzas | gibi: I don't know how to fix this, I'd like to discuss that with gmaan and dansmith | |
| 12:59:54 | sean-k-mooney | bauzas: that would require an api micoversion | |
| 13:00:15 | bauzas | because when I see https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L12319-L12322 I'm pretty done | |
| 13:00:31 | bauzas | sean-k-mooney: well, if that's due to a bug, maybe not | |
| 13:00:41 | sean-k-mooney | bauzas: its not | |
| 13:00:49 | sean-k-mooney | stephen wanted to remove it in the spec | |
| 13:00:50 | bauzas | like, HTTP500 > HTTP200 doesn't need to have a new microversion | |
| 13:00:55 | sean-k-mooney | and we told them no | |
| 13:01:05 | sean-k-mooney | becuase ops asked for it to be kept | |
| 13:01:12 | bauzas | here, we don't have a HTTP500, but it can take more than one minute for getting a response | |
| 13:01:23 | sean-k-mooney | that does nto mean we cant make it better | |
| 13:02:00 | gibi | HTTP500 -> HTTP200 only accepted in a bugfix in my eyes if it still provides the data that the non HTP500 case provides | |
| 13:02:03 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/wallaby/implemented/modernize-os-hypervisors-api.html#problem-description | |
| 13:02:14 | bauzas | I saw it | |
| 13:02:25 | bauzas | already and I saw the implementation change too | |
| 13:02:40 | sean-k-mooney | yes htis was done yesar ago to be clear its not recent | |
| 13:02:47 | sean-k-mooney | it has been this way since wallaby | |
| 13:02:52 | gibi | I agree that we need to fix it but maybe we can fix it in a way that preserve the functionality without the performance hit | |
| 13:02:55 | bauzas | but the problem is now we call the OS for all of the hosts automatically | |
| 13:03:00 | bauzas | instead of a specific API | |
| 13:03:13 | sean-k-mooney | you opt into it with the micorversion | |
| 13:03:17 | sean-k-mooney | we dont if you use the old one | |
| 13:03:25 | bauzas | sure | |
| 13:03:29 | sean-k-mooney | but there is a way to achive both | |
| 13:03:46 | sean-k-mooney | first for any host that is not using the qemu virt_type | |
| 13:03:51 | sean-k-mooney | we canc skip the call entirly | |
| 13:03:55 | bauzas | but atm, 2.88 has a performance problem, right? | |
| 13:03:57 | sean-k-mooney | and return none | |
| 13:04:18 | sean-k-mooney | and then we can start adding the uptime to the compute node stats | |
| 13:04:31 | sean-k-mooney | jand if its in the stats we can also skip it even for libvirt | |
| 13:04:45 | gibi | (ahh the idea was from sean-k-mooney not from dansmith sorry for the misattribution above) | |
| 13:04:47 | sean-k-mooney | finelly if we want to remove it in the api entirly we can do that in a new microverison | |
| 13:05:47 | jkulik | instead of having nova-compute call out into the OS on every HV, we could add it once on startup and add a timestamp to it. the API call can then just add `time.time() - timestamp` to the originally reported uptime :) | |
| 13:05:48 | sean-k-mooney | so we have 2 backportable fixes (only do the rpc if the diver supprot it and cache the value in the state generate via update_aviable_resouces) and one non backportable fix removing it in a new micorverion | |
| 13:06:10 | sean-k-mooney | jkulik: well we can also do that too yes | |
| 13:06:18 | sean-k-mooney | but that woudl be diffent | |
| 13:06:20 | jkulik | then we don't need a regular task | |
| 13:06:30 | jkulik | and also no rpc call | |
| 13:06:31 | sean-k-mooney | we check the host so the value does not go backward if you restart the agent on the host | |
| 13:06:48 | sean-k-mooney | jkulik: we have a perodic that is updating the stats already | |
| 13:07:04 | sean-k-mooney | we woudl jus tbe making it add 1 more key to the json blob | |
| 13:07:08 | sean-k-mooney | so that pretty much free | |
| 13:07:10 | jkulik | but is it calling an process every time? sounded to me like someone didn't like that very much | |
| 13:07:37 | sean-k-mooney | if we want ot change the meaning to be the compute agent uptime instgead of the host uptime we coudl do the time.time()-timestamp change as well | |
| 13:07:54 | jkulik | out, err = processutils.execute('env', 'LANG=C', 'uptime') | |
| 13:08:24 | sean-k-mooney | well we can get that info form libivrt too i think or there are other ways to do that hten calling the uptime command | |
| 13:08:24 | jkulik | we call that on agent startup and add the time since agent startup -> meaning stays the same | |
| 13:08:41 | sean-k-mooney | that not what uptime is | |
| 13:08:49 | sean-k-mooney | the uptime is nto the uptime since the agent started | |
| 13:08:54 | sean-k-mooney | its the uptiem of the host | |
| 13:09:04 | sean-k-mooney | and if you restart nova-compute without a reboot those are not the same | |
| 13:09:13 | jkulik | yes. hence we add the agent-uptime to the initially-read host uptime | |
| 13:09:33 | sean-k-mooney | or we jsut dont provide this in the furue since you shoudl not be getting this form nova at all | |
| 13:09:42 | jkulik | fine by me :D | |
| 13:10:06 | sean-k-mooney | in wallaby we really wanted to remvoe this from nova but opt wanted ot keep it instead of pullling it ofmr a monitoring tool liek prometheus or collectd | |
| 13:10:41 | sean-k-mooney | the compromies is we can provide a cached value form the existing perodic which is basiclly free | |
| 13:10:55 | sean-k-mooney | for older microversion and delete this in a future one | |
| 13:12:09 | jkulik | sounded like bauzas wouldn't like calling `uptime` in the periodic on all hosts all the time. so we'd also need a cheaper replacement for that. | |
| 13:12:14 | sean-k-mooney | the imprant thing it so fix the perfoamnce reguression. we could also choose in a new microversion to make the retivial fo the uptime contoleable by a query arg and default to not but that fells overly complicaterd | |
| 13:12:28 | sean-k-mooney | jkulik: well we can get that form libvirt as well im pretty sure | |
| 13:12:41 | sean-k-mooney | but that is an entirly seperate issue | |
| 13:12:53 | sean-k-mooney | nova has alwasy done that | |
| 13:13:05 | sean-k-mooney | it was not a regression intorduced by 2.88 | |
| 13:13:28 | jkulik | periodically? only on request, right? and we're going to change how often Nova calls that, if we move it to the periodic task | |
| 13:13:50 | jkulik | it's probably™ not much overhead, so we could also fix that later if we see it being a problem. | |
| 13:14:37 | sean-k-mooney | jkulik: my suggestion is to only update it when the update aviable resouce perodic runs | |
| 13:14:44 | sean-k-mooney | which is every 5 mins | |
| 13:14:56 | sean-k-mooney | we coudl stash the start time of the agent in the stats instead | |
| 13:15:03 | bauzas | sean-k-mooney: maybe I'm not right, but https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/hypervisors.py#L206 calls _view_hypervisor for each of the hypervisors lisrt | |
| 13:15:03 | sean-k-mooney | adn compute the uptime in the api | |
| 13:15:53 | bauzas | then in https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L97-L101 we unconditionnally call the compute API to get the uptime https://github.com/openstack/nova/blob/9f156aa954fb57eeaf407cc13a0f782b8af1a5d3/nova/api/openstack/compute/hypervisors.py#L97-L101 | |
| 13:16:20 | sean-k-mooney | bauzas: the rpc happens if you do a detail list or i belive hypervior show with 2.88+ | |
| 13:16:32 | bauzas | not only show right? | |
| 13:16:37 | bauzas | that's my point | |
| 13:16:48 | sean-k-mooney | right its in detail list as well | |
| 13:16:56 | sean-k-mooney | which is what was approvded in the spec | |
| 13:17:26 | bauzas | so for a detailed list of HVs using 2.88, we then call every compute for getting the uptime, right? | |
| 13:17:33 | sean-k-mooney | yep | |
| 13:18:00 | bauzas | each of those computes then eventually calls the driver, but in case of libvirt, it calls an OS process for getting uptime | |
| 13:18:24 | sean-k-mooney | yep | |
| 13:18:34 | sean-k-mooney | no one is saying this si good or scalable | |
| 13:18:40 | sean-k-mooney | but its not an api bug | |
| 13:18:43 | bauzas | so that means that if I have a list of 1000 computes all of them running libvirt, then I'd get 1000 RPC calls and each of them would access the OS before returning | |
| 13:18:59 | bauzas | that's what I call not scalable per say | |
| 13:19:10 | gibi | (I think we all agree that it is bad) | |
| 13:19:41 | sean-k-mooney | right we could also speed it up with sater gather but unless we want to break our api contract we cant fix it by just not including it | |