| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-09-04 | |||
| 11:43:29 | gibi | so we might discussed this already upstream as well | |
| 11:48:22 | gibi | aand there is another semi related thread of sending that uptime RPC to computes that are down, but not yet marked as down in the DB is another source of slowness | |
| 11:50:09 | jkulik | I'd go with the "sending to the DB" - not sure how to solve that for computes being down, though. | |
| 11:51:05 | gibi | jkulik: we already have the periodic heartbeat to know when a compute is down, so we can use that info to avoid reporting back a stale uptime data on the API | |
| 12:07:09 | gibi | I did not find this uptime discussion in my IRC logs so this probably happened downstream. I will file a bug right away summarizing both problems and the solution suggestion | |
| 12:16:34 | gibi | aand I found the downstream report https://issues.redhat.com/browse/OSPRH-17563 so we probably have motivation to try to fix it :) | |
| 12:23:01 | gibi | field the bug https://bugs.launchpad.net/nova/+bug/2122036 | |
| 12:25:30 | gibi | mmessah: ^^ | |
| 12:39:21 | mmessah | Thanks for discussing the situation fastly :) I hope it will be resolved as soon as possible. We recently upgraded our test environment to Epoxy. We will continue to test all services. If we have any situation, we will share IRC. | |
| 12:41:29 | gibi | mmessah: thanks | |
| 12:42:55 | opendevreview | Rajesh Tailor proposed openstack/nova stable/2024.2: Fix case-sensitivity for metadata keys https://review.opendev.org/c/openstack/nova/+/959435 | |
| 12:49:04 | gibi | dansmith: sean-k-mooney: when you are referring to "large-ops" job from the past (in relation to scale testing eventlet removal) do you mean https://review.opendev.org/c/openstack/project-config/+/278978 ? | |
| 12:50:53 | sean-k-mooney | i was not refering to that because i did not know that existed | |
| 12:50:58 | sean-k-mooney | but maybe dan was | |
| 12:51:20 | gibi | sean-k-mooney: ahh sorry I did not remember where this input coming from :) | |
| 12:51:21 | sean-k-mooney | gibi: i have just mention that kolla and devstack can fake may compute nodes per phsyical host | |
| 12:51:41 | sean-k-mooney | my expeirnce of this was as part of the osci lab scale testing | |
| 12:52:05 | sean-k-mooney | where intel and rackspace gave access to i think 500? baremental host half x86 the rest ppc | |
| 12:52:13 | sean-k-mooney | for the comutnity to do some scale testing | |
| 12:54:02 | sean-k-mooney | gibi: i wonder if the rally jobs have any existing usage of the fake agents | |
| 12:54:24 | sean-k-mooney | or if the tempest jobs in rally ieslf are usign real agents | |
| 12:55:39 | gibi | yeah rally is also on my radar to look into, especially neutron's jobs | |
| 12:55:56 | sean-k-mooney | so rally and browbeat are the two scaling frameworks we have | |
| 12:56:09 | sean-k-mooney | at leats those are teh two i am vagley aware of | |
| 12:56:46 | gibi | yeah, I never saw a realy browbeat scenario but heard the name before | |
| 12:57:36 | sean-k-mooney | it might be a wrapper arount rally looking at the repo | |
| 12:57:43 | sean-k-mooney | https://opendev.org/x/browbeat/ | |
| 12:58:01 | sean-k-mooney | it has no ci jobs so that wont help us | |
| 12:58:07 | bauzas | gibi: sean-k-mooney: well, I think we should quickly remove the uptime call IMHO | |
| 12:58:09 | gibi | ohh brownbeat referst to "generate_tripleo_inventory" | |
| 12:58:18 | gibi | that turns me down :) | |
| 12:58:34 | gibi | bauzas: in a new microversion? in a non backportable way? | |
| 12:58:40 | bauzas | if you look how this works, then we call the driver for every compute, and then for example the driver directly calls the OS | |
| 12:59:00 | bauzas | so when you have 10000 hypervisors, then each of them calls the OS directly | |
| 12:59:12 | sean-k-mooney | looks like rally does not have any devstack jobs https://github.com/openstack/rally/blob/master/.zuul.d/zuul.yaml although im not sure wht rallty tox-functional does | |
| 12:59:42 | bauzas | gibi: I don't know how to fix this, I'd like to discuss that with gmaan and dansmith | |
| 12:59:54 | sean-k-mooney | bauzas: that would require an api micoversion | |
| 13:00:15 | bauzas | because when I see https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L12319-L12322 I'm pretty done | |
| 13:00:31 | bauzas | sean-k-mooney: well, if that's due to a bug, maybe not | |
| 13:00:41 | sean-k-mooney | bauzas: its not | |
| 13:00:49 | sean-k-mooney | stephen wanted to remove it in the spec | |
| 13:00:50 | bauzas | like, HTTP500 > HTTP200 doesn't need to have a new microversion | |
| 13:00:55 | sean-k-mooney | and we told them no | |
| 13:01:05 | sean-k-mooney | becuase ops asked for it to be kept | |
| 13:01:12 | bauzas | here, we don't have a HTTP500, but it can take more than one minute for getting a response | |
| 13:01:23 | sean-k-mooney | that does nto mean we cant make it better | |
| 13:02:00 | gibi | HTTP500 -> HTTP200 only accepted in a bugfix in my eyes if it still provides the data that the non HTP500 case provides | |
| 13:02:03 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/wallaby/implemented/modernize-os-hypervisors-api.html#problem-description | |
| 13:02:14 | bauzas | I saw it | |
| 13:02:25 | bauzas | already and I saw the implementation change too | |
| 13:02:40 | sean-k-mooney | yes htis was done yesar ago to be clear its not recent | |
| 13:02:47 | sean-k-mooney | it has been this way since wallaby | |
| 13:02:52 | gibi | I agree that we need to fix it but maybe we can fix it in a way that preserve the functionality without the performance hit | |
| 13:02:55 | bauzas | but the problem is now we call the OS for all of the hosts automatically | |
| 13:03:00 | bauzas | instead of a specific API | |
| 13:03:13 | sean-k-mooney | you opt into it with the micorversion | |
| 13:03:17 | sean-k-mooney | we dont if you use the old one | |
| 13:03:25 | bauzas | sure | |
| 13:03:29 | sean-k-mooney | but there is a way to achive both | |
| 13:03:46 | sean-k-mooney | first for any host that is not using the qemu virt_type | |
| 13:03:51 | sean-k-mooney | we canc skip the call entirly | |
| 13:03:55 | bauzas | but atm, 2.88 has a performance problem, right? | |
| 13:03:57 | sean-k-mooney | and return none | |
| 13:04:18 | sean-k-mooney | and then we can start adding the uptime to the compute node stats | |
| 13:04:31 | sean-k-mooney | jand if its in the stats we can also skip it even for libvirt | |
| 13:04:45 | gibi | (ahh the idea was from sean-k-mooney not from dansmith sorry for the misattribution above) | |
| 13:04:47 | sean-k-mooney | finelly if we want to remove it in the api entirly we can do that in a new microverison | |
| 13:05:47 | jkulik | instead of having nova-compute call out into the OS on every HV, we could add it once on startup and add a timestamp to it. the API call can then just add `time.time() - timestamp` to the originally reported uptime :) | |
| 13:05:48 | sean-k-mooney | so we have 2 backportable fixes (only do the rpc if the diver supprot it and cache the value in the state generate via update_aviable_resouces) and one non backportable fix removing it in a new micorverion | |
| 13:06:10 | sean-k-mooney | jkulik: well we can also do that too yes | |
| 13:06:18 | sean-k-mooney | but that woudl be diffent | |
| 13:06:20 | jkulik | then we don't need a regular task | |
| 13:06:30 | jkulik | and also no rpc call | |
| 13:06:31 | sean-k-mooney | we check the host so the value does not go backward if you restart the agent on the host | |
| 13:06:48 | sean-k-mooney | jkulik: we have a perodic that is updating the stats already | |
| 13:07:04 | sean-k-mooney | we woudl jus tbe making it add 1 more key to the json blob | |
| 13:07:08 | sean-k-mooney | so that pretty much free | |
| 13:07:10 | jkulik | but is it calling an process every time? sounded to me like someone didn't like that very much | |
| 13:07:37 | sean-k-mooney | if we want ot change the meaning to be the compute agent uptime instgead of the host uptime we coudl do the time.time()-timestamp change as well | |
| 13:07:54 | jkulik | out, err = processutils.execute('env', 'LANG=C', 'uptime') | |
| 13:08:24 | sean-k-mooney | well we can get that info form libivrt too i think or there are other ways to do that hten calling the uptime command | |
| 13:08:24 | jkulik | we call that on agent startup and add the time since agent startup -> meaning stays the same | |
| 13:08:41 | sean-k-mooney | that not what uptime is | |
| 13:08:49 | sean-k-mooney | the uptime is nto the uptime since the agent started | |
| 13:08:54 | sean-k-mooney | its the uptiem of the host | |
| 13:09:04 | sean-k-mooney | and if you restart nova-compute without a reboot those are not the same | |
| 13:09:13 | jkulik | yes. hence we add the agent-uptime to the initially-read host uptime | |
| 13:09:33 | sean-k-mooney | or we jsut dont provide this in the furue since you shoudl not be getting this form nova at all | |
| 13:09:42 | jkulik | fine by me :D | |
| 13:10:06 | sean-k-mooney | in wallaby we really wanted to remvoe this from nova but opt wanted ot keep it instead of pullling it ofmr a monitoring tool liek prometheus or collectd | |
| 13:10:41 | sean-k-mooney | the compromies is we can provide a cached value form the existing perodic which is basiclly free | |
| 13:10:55 | sean-k-mooney | for older microversion and delete this in a future one | |
| 13:12:09 | jkulik | sounded like bauzas wouldn't like calling `uptime` in the periodic on all hosts all the time. so we'd also need a cheaper replacement for that. | |
| 13:12:14 | sean-k-mooney | the imprant thing it so fix the perfoamnce reguression. we could also choose in a new microversion to make the retivial fo the uptime contoleable by a query arg and default to not but that fells overly complicaterd | |
| 13:12:28 | sean-k-mooney | jkulik: well we can get that form libvirt as well im pretty sure | |
| 13:12:41 | sean-k-mooney | but that is an entirly seperate issue | |
| 13:12:53 | sean-k-mooney | nova has alwasy done that | |
| 13:13:05 | sean-k-mooney | it was not a regression intorduced by 2.88 | |
| 13:13:28 | jkulik | periodically? only on request, right? and we're going to change how often Nova calls that, if we move it to the periodic task | |
| 13:13:50 | jkulik | it's probably™ not much overhead, so we could also fix that later if we see it being a problem. | |