Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-09-15
23:34:02 opendevreview Merged openstack/placement master: Drop unused bandit target https://review.opendev.org/c/openstack/placement/+/976906
#openstack-nova - 2026-09-16
00:00:32 opendevreview Merged openstack/placement master: Skip functional tests for git config update https://review.opendev.org/c/openstack/placement/+/976907
02:01:29 mw_ Hi folks, could someone please take a look at this Nova change when you have a chance? https://review.opendev.org/c/openstack/nova/+/999841
02:01:41 mw_ It adds bandwidth limiting for attached volume retypes and migrations to make their network impact more predictable.
04:17:05 opendevreview Merged openstack/placement master: Commonize code for updating and creating allocations https://review.opendev.org/c/openstack/placement/+/808535
06:25:53 zigo With Hibiscus, it looks like the systemd notify doesn't work. My systemd nova-compute.service stays in "Activating" mode until systemd believes it's crashed, and shut it down.
06:25:53 zigo Hi team!
06:26:02 zigo Is it possibly broken ?!?
06:26:07 zigo How can I check?
08:38:43 gibi zigo: I'm spinning up a devstack from master to see if I can replicate that
08:43:25 gibi can I see your systemd service file easily somewhere?
08:46:14 gibi zigo: also a quick check, have you tried switching the nova-compute back to eventlet mode (if you can) to see if that resolves the systemd problem?
08:50:28 gibi zigo: I see this https://github.com/openstack/oslo.service/blob/364aafec725c9b98d23426450bc247d838cba611/oslo_service/backend/_eventlet/service.py#L297 but so far I haven't seen the _threading equivalent of it
08:51:08 gibi https://github.com/search?q=repo%3Aopenstack%2Foslo.service+systemd.notify&type=code yeah it feels like that systemd support is missing from oslo.service threading backend
08:51:23 gibi once I confirm it locally in devstack I will file a bug for oslo.service
08:52:32 gibi stephenfin: tkajinam: ^^
09:10:48 gibi adding Type=notify to the devstack's service file reproduces the problem and switching nova-compute to eventlet mode fixes it
09:11:52 gibi so the reason this is seen now is because we switched the default to threading mode for nova-compute in Hibiscus
09:12:11 gibi but I guess this oslo.service bug affects other services running in threading mode as well
09:24:26 zigo gibi: Thanks for checking for this, I'll switch nova-compute to Type=simple at least temporarily.
09:25:22 zigo apt-get update
09:25:22 zigo extrepo openstack_hibiscus
09:25:22 zigo apt-get install extrepo
09:25:22 zigo If you need to check for my packaging, next time, simply do:
09:25:23 zigo that setup everything. Otherwise it's in Debian Experimental.
09:26:42 gibi zigo: thanks I found the service file
09:43:37 gibi filed https://bugs.launchpad.net/nova/+bug/2167448
10:01:22 opendevreview Merged openstack/os-vif stable/2026.1: ovs: disable IPv6 on taps created by create_tap before bringing them up https://review.opendev.org/c/openstack/os-vif/+/1005663
11:00:49 gibi zigo: am I sensing right that you are OK to fix this later, after Hibiscus GA?
11:09:03 zigo gibi: If only Nova is affected, that's ok, yes.
11:09:57 gibi zigo: thanks. I guess you will see in you testing if other services are affected or not
11:10:48 zigo I beI'm currently checking.
11:11:44 zigo gibi: nova-conductor seems ok, in my CI.
11:12:11 zigo It does have Type=notify and system see it as up.
11:12:22 gibi thanks that is a good data point
11:13:58 zigo I'm asking GLM-5 if he knows why... :P
11:14:19 gibi yepp I see that locally too (super conductor is not affected even though running in threading mode)
11:26:24 gibi zigo: it is due to cotyledon + nova-compute using no-fork=True
11:26:32 zigo Ok, the difference, according to my AI, is because nova-conductor calls server(server, workers=N) with default to no_fork=False, but nova-compute calls it with the added no_fork=True, added in https://review.opendev.org/c/openstack/nova/+/965466 (Do not fork compute workers in native threading mode) that actually, you're the patch owner.
11:26:39 zigo Yeah... :P
11:26:44 gibi our LLMs are agreeing :)
11:26:52 gibi I will draft a fix
11:27:26 zigo gibi: Do you think we should fix this is olso.service rather than in Nova ?
11:27:30 zigo (I do...)
11:29:10 gibi I do too
11:29:18 gibi as olso.service give the support for no-fork
11:29:33 gibi but this probably means no fix in Hibiscus GA
11:29:54 gibi or at least it become more complicated
11:29:59 gibi due to req freeze
11:30:19 gibi but it also means this bug only affects services that are not forking
11:30:59 gibi (this is not first bug due to nova-compute being special, so we should also think about working there a single worker and basically load the nova-compute service there)
11:31:12 gibi s/working/forking/
11:36:36 zigo Well, I can carry the patch in my Debian package. It's really only 2 lines, as much as I can see.
11:39:08 zigo https://review.opendev.org/c/openstack/oslo.service/+/1005864
11:39:10 zigo :)
11:39:57 gibi you type faster than me
11:39:58 gibi :)
11:40:10 gibi but I tested exactly the same thing now locally
11:40:15 gibi and it resolve the prolbme
11:41:14 zigo I'll try to patch my oslo.service Debian package and see if nova-compute is happy. :P
11:41:36 gibi it is happy for me :)
11:42:17 gibi please add a Closes-Bug: #2167448 line to the commit message otherwise it looks good to me
11:44:00 opendevreview Ashish Gupta proposed openstack/nova master: Fix init_host crash migration test under native threading https://review.opendev.org/c/openstack/nova/+/1001369
11:45:01 zigo Right.
11:51:04 zigo Yes, works for me as well ! :)
11:51:45 zigo I can see "Creating RPC server for service: nova-compute on topic: compute start" in my logs, what's that for?!?
11:53:18 zigo Should I worry?
11:53:18 zigo https://paste.opendev.org/show/b6Z3DQTexWsKxupCuLit/
11:53:18 zigo Also, I'm seeing this in my log:
11:54:02 zigo Missing sentence in my above paste: A dynamic interval looping call should supply either an interval or periodic_interval_max
11:54:59 gibi do you see that during shutdown?
11:55:04 gibi i.e. seeing Skipping periodic tasks during graceful shutdown.
11:55:14 zigo Ah, yes.
11:55:44 zigo Can't see anything now, because I'm restarting my CI from scratch, but yes, it was during shutdown.
11:56:38 gibi then I would not worry much especially if nova-compute exited
11:56:51 gibi which I assume did to be able to start up again
11:56:59 gibi I will look into it
11:57:25 gibi btw it seems only nova-compute and ironic vncproxy service are using no_fork=True https://codesearch.opendev.org/?q=no_fork%3DTrue&i=nope&literal=nope&files=&excludeFiles=&repos=
11:58:05 gibi yepp this is happening to me too during shutdown https://paste.openstack.org/show/bG4BK2BV3lcOqBKtC5ry/
11:58:50 gibi https://paste.openstack.org/show/bEAmK5c6VltFQ1OmI9JD/ more complete paste
11:59:04 gibi so it is cosmetic, the shutdown happens
12:05:50 opendevreview Merged openstack/placement stable/2025.1: Reproduce bug 2160721 https://review.opendev.org/c/openstack/placement/+/998877
12:32:00 gibi zigo: we should return a number here https://github.com/openstack/nova/blob/94de0576ca02e0668233bedfbe7e18b87316c708/nova/manager.py#L131 to fulfill the expected proptocol of oslo and to avoid the stack trace, purely cosmetic
12:32:33 zigo Oh ok.
12:32:41 zigo Would be nice though ... :P
12:32:48 gibi yeah I will make a patch
12:33:08 zigo Any stack trace in our production triggers some alerts. :P
12:33:20 gibi understandable
13:16:26 opendevreview Merged openstack/placement stable/2025.1: Check overall capacity before generating candidates https://review.opendev.org/c/openstack/placement/+/998878
13:36:46 opendevreview Balazs Gibizer proposed openstack/nova master: Stop periodic tasks during shutdown https://review.opendev.org/c/openstack/nova/+/1005880
13:36:46 opendevreview Balazs Gibizer proposed openstack/nova master: Reproduce bug/2167481 https://review.opendev.org/c/openstack/nova/+/1005879
13:36:51 gibi zigo: ^^
13:36:58 gibi this removes the stack trace
13:37:14 gibi cc gmaan as this is a shutdown improvement
13:39:25 opendevreview Eduardo Henrique Lima de Morais proposed openstack/nova master: Enforce nullable=False on instance_uuid https://review.opendev.org/c/openstack/nova/+/1005881
13:46:06 eduardomorais[m] I'm a new contributor and I'm looking for simpler tasks to get familiar with the codebase. I found a very simple TODO and implemented what it asked for. I'd really appreciate it if you could review it! https://review.opendev.org/c/openstack/nova/+/1005881
13:46:55 opendevreview Stephen Finucane proposed openstack/placement master: Fix tests on Python 3.14 https://review.opendev.org/c/openstack/placement/+/1005883
13:52:23 zigo Thanks, added to my Debian package.
13:52:55 gibi zigo: thanks for reporting the issues
13:54:49 gibi Uggla: I could not join today's upstream bug triage. Just to summarize the above. We have a regression in systemd service status reporting in nova-compute but as this is limited to that service (and maybe ironic vncproxy) zigo is OK to carry a patch in debian until we can land the fix and backport to H. So this is not an RC2 trigger in my eyes (also it is an oslo.service fix)

Earlier   Later