Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-04-29
14:03:43 gibi futurist decides today
14:03:48 gibi but we can have a config
14:04:03 sean-k-mooney i belvie it default to 2.5*nproc
14:04:15 gibi I can run some tests to see what is the runtime stats there
14:04:32 sean-k-mooney i reused one fo the existing config option but i would proably default this to 4
14:04:49 sean-k-mooney and allow peopel to overried if needed
14:04:50 gibi https://docs.openstack.org/futurist/latest/reference/index.html#futurist.ExecutorStatistics.runtime doc says total runtime of the pool
14:05:14 sean-k-mooney hum well it went down form 69 to 70
14:06:15 gibi we have two worker processes so you need to check the pid to see if this is the same worker
14:06:25 sean-k-mooney oh good point
14:06:32 sean-k-mooney are you usign https://docs.openstack.org/futurist/latest/reference/index.html#futurist.periodics.Watcher
14:06:34 gibi futurist creates cpu * 5 threads per executor by default https://github.com/openstack/futurist/blob/master/futurist/_utils.py#L126
14:06:57 sean-k-mooney gibi: ack thats proably more then we need
14:08:07 gibi I don't use periodics, I log the stats when new work is submitted and if the last report was older than 60 sec
14:08:22 gibi we can change it to periodic
14:08:33 gibi and or we can have a config option for the age
14:09:12 gibi yeah, 20 worker for scatter gather is too much, we can aim for 4 ish, but it depeds on the amount of parallel scheduling we need to serve
14:11:02 sean-k-mooney right but we have 2 ways to scale workers and treads in each workser thread pool. and we can make it configurable
14:11:43 sean-k-mooney well technially 3 you can just run more schdulers
14:12:36 sean-k-mooney i think tying the count to the cpu cores will bit ues down the line espcially if this is deployed on k8s as that could change based on where the pod runs so that why i would prefer to have a small integer default
14:16:55 gibi yeah I will expose a config option with a small default
14:17:11 gibi and document that you can ask for more worker processes as well as a scaling mechanism
14:19:47 dansmith I think it will take some time to figure out what the real default should be
14:20:16 dansmith greenthreads are nice, but they also kinda avoid the resource exhaustion question such that we don't really have a good idea of what it needs to be, IMHO
14:54:45 gibi yepp
14:54:51 sean-k-mooney gibi: https://zuul.opendev.org/t/openstack/buildset/f495c3661ca346a198c80a01387d6ba4 so that not terible for a first run
14:55:06 gibi nova-next is green that is enough for me to declare success
14:55:19 sean-k-mooney im not sure why grenade is unhappy
14:55:28 sean-k-mooney maybe the devstack change
14:55:32 gibi some tempest jobs does not have oslo.service as required project hence the oslo.service depends on is ignored
14:55:45 sean-k-mooney ah ya that too
14:56:49 gibi also nova-multi-cell is green
14:57:02 sean-k-mooney yep most of them are
14:57:20 opendevreview Dan Smith proposed openstack/nova master: WIP Example NVMe cleaning script for one-time-use https://review.opendev.org/c/openstack/nova/+/948446
14:57:52 sean-k-mooney gibi: the cyborg job is broken because they do not have pyproject.toml supprot so there wsgi script are not gerneated
14:57:54 gibi nova-multi-cell executed the most scatter-gathers and the nova-scheduler RSS is still around 122MB
14:58:24 gibi sean-k-mooney: yeah I ended up at the same conclusion this morning, as well as barbican and designate
14:58:30 sean-k-mooney im fixing that on watcher but that will impact other proejct shortly
14:58:36 gibi I filed bugs for barbican and cyborg, ralonsoh has bug for designate
14:58:54 sean-k-mooney ack there was a pip release 3 days ago
14:59:11 gibi yeah frickler pointed out that setupstools=80 is the root cause
14:59:30 sean-k-mooney well kind of
14:59:47 ralonsoh just for information: https://review.opendev.org/c/openstack/designate/+/902846
14:59:59 ralonsoh the designate fix (along with another fix for the CI)
15:00:00 sean-k-mooney the actual root causae is we have been relying on a compatibality fallbac that was added 2 years ago and then deprecated for removal a year ago
15:00:57 sean-k-mooney ralonsoh: this is the watcher fix but im respining it now https://review.opendev.org/c/openstack/watcher/+/948438 to fix the mistakes
15:01:52 sean-k-mooney ralonsoh: i was going to defer the removal of the wsgi_script until next cycle
15:02:14 sean-k-mooney but i guess if we backported this to epoxy we coudl remove it this cycle maybe
15:02:14 ralonsoh well, now is mandatory I think
15:02:24 sean-k-mooney no it will still work with older pip
15:02:33 sean-k-mooney on editbale installs
15:02:40 sean-k-mooney ralonsoh: nova still has supprot for them
15:03:11 johnsom I worry about backporting these patches given there is package/deployer impact. I was considering pinning setuptools in 2025.1
15:03:17 sean-k-mooney if you build the whelie in non editable mode the extenion we have also work i belive
15:03:35 sean-k-mooney johnsom: ya im trying to avoid backporting it too
15:04:13 sean-k-mooney johnsom: that why im intentially not removing the old way fo usign the console scrit in the intial patch
15:05:14 opendevreview Balazs Gibizer proposed openstack/nova master: WIP: allow service to start with threading https://review.opendev.org/c/openstack/nova/+/948311
15:05:14 opendevreview Balazs Gibizer proposed openstack/nova master: Print ThreadPool statistics https://review.opendev.org/c/openstack/nova/+/948340
15:05:15 opendevreview Balazs Gibizer proposed openstack/nova master: DNM:Run nova-next with n-sch in threading mode https://review.opendev.org/c/openstack/nova/+/948450
15:05:21 johnsom Well, if we don't pin, it's still likely to break people. That is why I was pushing to do it all at once and get the pain overwith
15:06:40 sean-k-mooney ya its unfortuent we dint get this done in carical liek we orgianlly wanted to do
15:06:50 johnsom +1
15:07:30 sean-k-mooney in theory if you use an old enough pip/setuptoosl nova still supprot setup.py directly too but nothing tests that
15:07:37 johnsom We plan to land the designate patch today (there is a typo to fix)
15:40:52 bauzas nova meeting in 20 mins here
15:45:58 haleyb we noticed a failure in the neutron periodic jobs today with the oslo.utils master branch, but seems more a nova issue - https://bugs.launchpad.net/nova/+bug/2109592
15:46:19 haleyb i'm assuming that would eventually get caught in an integration job somewhere
15:46:38 dansmith that looks more like an oslo, requirements, etc thing
15:46:51 dansmith (could be nova requirements.txt)
15:47:00 sean-k-mooney ya i was going ot say the same
15:47:19 sean-k-mooney what branch was this on "master" i assume
15:47:20 haleyb dansmith: could be just didn't seem neutron-specific
15:47:29 dansmith haleyb: yes :D
15:47:41 frickler from the timing when this started failing, I was assuming setuptools==80 to be related
15:47:42 haleyb yes, we have a job that uses most (all) of the oslo master branches
15:47:45 sean-k-mooney actully thi si from a neutorn job that is testing with oslo master
15:48:03 haleyb it's a canary job
15:48:08 sean-k-mooney so that may also be a regression in oslo
15:48:16 sean-k-mooney i.e. they moved the module
15:48:23 frickler because from sunday to monday no changes were merged in oslo, neutron or nova
15:48:53 gmaan yeah it is not yet released so will not be visible in other jobs but oslo master one
15:49:05 sean-k-mooney no its where we expect
15:49:17 haleyb we noticed today, started seeing yesterday according to build logs
15:49:35 haleyb https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-release-with-oslo-master&job_name=neutron-ovs-tempest-with-oslo-master&skip=0
15:51:15 sean-k-mooney the module is still in the same place https://github.com/openstack/oslo.utils/blob/master/oslo_utils/imageutils/format_inspector.py
15:51:29 sean-k-mooney so its a little od
15:51:49 sean-k-mooney oh
15:52:00 sean-k-mooney /usr/lib/python3/dist-packages/oslo_utils/imageutils.py
15:52:12 sean-k-mooney that look wrong
15:53:42 sean-k-mooney it is using setup tools 80 Requirement already satisfied: setuptools in /opt/stack/data/venv/lib/python3.12/site-packages (from pbr>=6.1.0->oslo.utils==8.3.0.dev7) (80.0.0)
16:00:22 opendevreview Balazs Gibizer proposed openstack/nova master: DNM:Run nova-next with n-sch in threading mode https://review.opendev.org/c/openstack/nova/+/948450
16:01:04 opendevmeet The meeting name has been set to 'nova'
16:01:04 opendevmeet Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:04 opendevmeet Meeting started Tue Apr 29 16:01:04 2025 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:04 bauzas #startmeeting nova
16:01:08 gibi o/
16:01:12 bauzas guys, we have a meeting now
16:01:13 elodilles o/
16:01:13 sean-k-mooney o/
16:01:18 bauzas do you want to discuss ?

Earlier   Later