| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-04-29 | |||
| 14:06:25 | sean-k-mooney | oh good point | |
| 14:06:32 | sean-k-mooney | are you usign https://docs.openstack.org/futurist/latest/reference/index.html#futurist.periodics.Watcher | |
| 14:06:34 | gibi | futurist creates cpu * 5 threads per executor by default https://github.com/openstack/futurist/blob/master/futurist/_utils.py#L126 | |
| 14:06:57 | sean-k-mooney | gibi: ack thats proably more then we need | |
| 14:08:07 | gibi | I don't use periodics, I log the stats when new work is submitted and if the last report was older than 60 sec | |
| 14:08:22 | gibi | we can change it to periodic | |
| 14:08:33 | gibi | and or we can have a config option for the age | |
| 14:09:12 | gibi | yeah, 20 worker for scatter gather is too much, we can aim for 4 ish, but it depeds on the amount of parallel scheduling we need to serve | |
| 14:11:02 | sean-k-mooney | right but we have 2 ways to scale workers and treads in each workser thread pool. and we can make it configurable | |
| 14:11:43 | sean-k-mooney | well technially 3 you can just run more schdulers | |
| 14:12:36 | sean-k-mooney | i think tying the count to the cpu cores will bit ues down the line espcially if this is deployed on k8s as that could change based on where the pod runs so that why i would prefer to have a small integer default | |
| 14:16:55 | gibi | yeah I will expose a config option with a small default | |
| 14:17:11 | gibi | and document that you can ask for more worker processes as well as a scaling mechanism | |
| 14:19:47 | dansmith | I think it will take some time to figure out what the real default should be | |
| 14:20:16 | dansmith | greenthreads are nice, but they also kinda avoid the resource exhaustion question such that we don't really have a good idea of what it needs to be, IMHO | |
| 14:54:45 | gibi | yepp | |
| 14:54:51 | sean-k-mooney | gibi: https://zuul.opendev.org/t/openstack/buildset/f495c3661ca346a198c80a01387d6ba4 so that not terible for a first run | |
| 14:55:06 | gibi | nova-next is green that is enough for me to declare success | |
| 14:55:19 | sean-k-mooney | im not sure why grenade is unhappy | |
| 14:55:28 | sean-k-mooney | maybe the devstack change | |
| 14:55:32 | gibi | some tempest jobs does not have oslo.service as required project hence the oslo.service depends on is ignored | |
| 14:55:45 | sean-k-mooney | ah ya that too | |
| 14:56:49 | gibi | also nova-multi-cell is green | |
| 14:57:02 | sean-k-mooney | yep most of them are | |
| 14:57:20 | opendevreview | Dan Smith proposed openstack/nova master: WIP Example NVMe cleaning script for one-time-use https://review.opendev.org/c/openstack/nova/+/948446 | |
| 14:57:52 | sean-k-mooney | gibi: the cyborg job is broken because they do not have pyproject.toml supprot so there wsgi script are not gerneated | |
| 14:57:54 | gibi | nova-multi-cell executed the most scatter-gathers and the nova-scheduler RSS is still around 122MB | |
| 14:58:24 | gibi | sean-k-mooney: yeah I ended up at the same conclusion this morning, as well as barbican and designate | |
| 14:58:30 | sean-k-mooney | im fixing that on watcher but that will impact other proejct shortly | |
| 14:58:36 | gibi | I filed bugs for barbican and cyborg, ralonsoh has bug for designate | |
| 14:58:54 | sean-k-mooney | ack there was a pip release 3 days ago | |
| 14:59:11 | gibi | yeah frickler pointed out that setupstools=80 is the root cause | |
| 14:59:30 | sean-k-mooney | well kind of | |
| 14:59:47 | ralonsoh | just for information: https://review.opendev.org/c/openstack/designate/+/902846 | |
| 14:59:59 | ralonsoh | the designate fix (along with another fix for the CI) | |
| 15:00:00 | sean-k-mooney | the actual root causae is we have been relying on a compatibality fallbac that was added 2 years ago and then deprecated for removal a year ago | |
| 15:00:57 | sean-k-mooney | ralonsoh: this is the watcher fix but im respining it now https://review.opendev.org/c/openstack/watcher/+/948438 to fix the mistakes | |
| 15:01:52 | sean-k-mooney | ralonsoh: i was going to defer the removal of the wsgi_script until next cycle | |
| 15:02:14 | ralonsoh | well, now is mandatory I think | |
| 15:02:14 | sean-k-mooney | but i guess if we backported this to epoxy we coudl remove it this cycle maybe | |
| 15:02:24 | sean-k-mooney | no it will still work with older pip | |
| 15:02:33 | sean-k-mooney | on editbale installs | |
| 15:02:40 | sean-k-mooney | ralonsoh: nova still has supprot for them | |
| 15:03:11 | johnsom | I worry about backporting these patches given there is package/deployer impact. I was considering pinning setuptools in 2025.1 | |
| 15:03:17 | sean-k-mooney | if you build the whelie in non editable mode the extenion we have also work i belive | |
| 15:03:35 | sean-k-mooney | johnsom: ya im trying to avoid backporting it too | |
| 15:04:13 | sean-k-mooney | johnsom: that why im intentially not removing the old way fo usign the console scrit in the intial patch | |
| 15:05:14 | opendevreview | Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340 | |
| 15:05:14 | opendevreview | Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311 | |
| 15:05:15 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Run nova-next with n-sch in threading mode https://review.opendev.org/c/openstack/nova/+/948450 | |
| 15:05:21 | johnsom | Well, if we don't pin, it's still likely to break people. That is why I was pushing to do it all at once and get the pain overwith | |
| 15:06:40 | sean-k-mooney | ya its unfortuent we dint get this done in carical liek we orgianlly wanted to do | |
| 15:06:50 | johnsom | +1 | |
| 15:07:30 | sean-k-mooney | in theory if you use an old enough pip/setuptoosl nova still supprot setup.py directly too but nothing tests that | |
| 15:07:37 | johnsom | We plan to land the designate patch today (there is a typo to fix) | |
| 15:40:52 | bauzas | nova meeting in 20 mins here | |
| 15:45:58 | haleyb | we noticed a failure in the neutron periodic jobs today with the oslo.utils master branch, but seems more a nova issue - https://bugs.launchpad.net/nova/+bug/2109592 | |
| 15:46:19 | haleyb | i'm assuming that would eventually get caught in an integration job somewhere | |
| 15:46:38 | dansmith | that looks more like an oslo, requirements, etc thing | |
| 15:46:51 | dansmith | (could be nova requirements.txt) | |
| 15:47:00 | sean-k-mooney | ya i was going ot say the same | |
| 15:47:19 | sean-k-mooney | what branch was this on "master" i assume | |
| 15:47:20 | haleyb | dansmith: could be just didn't seem neutron-specific | |
| 15:47:29 | dansmith | haleyb: yes :D | |
| 15:47:41 | frickler | from the timing when this started failing, I was assuming setuptools==80 to be related | |
| 15:47:42 | haleyb | yes, we have a job that uses most (all) of the oslo master branches | |
| 15:47:45 | sean-k-mooney | actully thi si from a neutorn job that is testing with oslo master | |
| 15:48:03 | haleyb | it's a canary job | |
| 15:48:08 | sean-k-mooney | so that may also be a regression in oslo | |
| 15:48:16 | sean-k-mooney | i.e. they moved the module | |
| 15:48:23 | frickler | because from sunday to monday no changes were merged in oslo, neutron or nova | |
| 15:48:53 | gmaan | yeah it is not yet released so will not be visible in other jobs but oslo master one | |
| 15:49:05 | sean-k-mooney | no its where we expect | |
| 15:49:17 | haleyb | we noticed today, started seeing yesterday according to build logs | |
| 15:49:35 | haleyb | https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-release-with-oslo-master&job_name=neutron-ovs-tempest-with-oslo-master&skip=0 | |
| 15:51:15 | sean-k-mooney | the module is still in the same place https://github.com/openstack/oslo.utils/blob/master/oslo_utils/imageutils/format_inspector.py | |
| 15:51:29 | sean-k-mooney | so its a little od | |
| 15:51:49 | sean-k-mooney | oh | |
| 15:52:00 | sean-k-mooney | /usr/lib/python3/dist-packages/oslo_utils/imageutils.py | |
| 15:52:12 | sean-k-mooney | that look wrong | |
| 15:53:42 | sean-k-mooney | it is using setup tools 80 Requirement already satisfied: setuptools in /opt/stack/data/venv/lib/python3.12/site-packages (from pbr>=6.1.0->oslo.utils==8.3.0.dev7) (80.0.0) | |
| 16:00:22 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Run nova-next with n-sch in threading mode https://review.opendev.org/c/openstack/nova/+/948450 | |
| 16:01:04 | bauzas | #startmeeting nova | |
| 16:01:04 | opendevmeet | Meeting started Tue Apr 29 16:01:04 2025 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:01:04 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:01:04 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:01:08 | gibi | o/ | |
| 16:01:12 | bauzas | guys, we have a meeting now | |
| 16:01:13 | sean-k-mooney | o/ | |
| 16:01:13 | elodilles | o/ | |
| 16:01:18 | bauzas | do you want to discuss ? | |
| 16:01:30 | gmaan | o/ | |
| 16:01:30 | bauzas | I can chair it for the next 20/30 mins | |
| 16:01:41 | fwiesel | o | |
| 16:01:44 | gibi | I have one gate topic | |
| 16:01:47 | bauzas | and then I'll need to go offline | |
| 16:01:51 | bauzas | okay, let's start | |
| 16:02:11 | bauzas | #topic Bugs (stuck/critical) | |
| 16:02:21 | bauzas | #info No Critical bug | |
| 16:02:29 | bauzas | #topic Gate status | |