Posted	Nick	Remark
#openstack-nova - 2019-12-19
15:33:21	mriedem	stephenfin: maybe just hitting faster nodes
15:33:27	stephenfin	Could be
15:33:30	mriedem	when it fails it's on some rax nodes
15:33:38	mriedem	fewer noisy neighbors right now
15:33:44	stephenfin	+W'd anyway
15:33:46	stephenfin	nice find
15:41:02	stephenfin	mriedem_afk: Should I figure out some way to break up https://review.opendev.org/#/c/696514/ to get it over the line?
16:33:23	openstackgerrit	Shilpa Devharakar proposed openstack/nova master: Handle new is_volume_backend join column query https://review.opendev.org/694462
16:33:23	openstackgerrit	Shilpa Devharakar proposed openstack/nova master: Instance object changes for the new 'is_volume_backed' expected_attr https://review.opendev.org/694463
16:33:24	openstackgerrit	Shilpa Devharakar proposed openstack/nova master: Ignore root_gb if instance is booted from volume https://review.opendev.org/612626
16:35:12	openstackgerrit	Stephen Finucane proposed openstack/nova stable/train: Block rebuild when NUMA topology changed https://review.opendev.org/698530
16:35:13	openstackgerrit	Stephen Finucane proposed openstack/nova stable/train: Disable NUMATopologyFilter on rebuild https://review.opendev.org/698532
16:35:13	openstackgerrit	Stephen Finucane proposed openstack/nova stable/train: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/700021
16:35:35	stephenfin	artom: Respun that to address your comments. Could you take another look? ^
16:36:12	stephenfin	lyarwood: You'll need to hit that too, if that's okay? I backported an additional change (https://review.opendev.org/700021) to avoid a merge conflict
16:36:57	artom	stephenfin, thank you! Lemme finish a thing, then I'll do my due diligence and +1
16:37:14	stephenfin	once mriedem_afk is back, ima bug him to hit the series too since I'm not sure what other stable core can review that stuff
16:40:14	aarents	lyarwood: if you may have a look on that https://review.opendev.org/#/c/696084/ Kashyap also, it will be helpfull !
16:45:20	artom	stephenfin, I think you got the wrong changeID on https://review.opendev.org/#/c/698530/3
16:45:55	stephenfin	gdi
16:47:54	openstackgerrit	Stephen Finucane proposed openstack/nova stable/train: Block rebuild when NUMA topology changed https://review.opendev.org/698530
16:47:54	openstackgerrit	Stephen Finucane proposed openstack/nova stable/train: Disable NUMATopologyFilter on rebuild https://review.opendev.org/698532
17:45:21	openstackgerrit	Mykola Yakovliev proposed openstack/nova master: Fix boot_roles in InstanceSystemMetadata https://review.opendev.org/698040
18:33:54	openstackgerrit	Merged openstack/nova master: Fix race in test_create_servers_with_vpmem https://review.opendev.org/699997
19:22:27	gmann	stephenfin ping
20:09:22	openstackgerrit	Matt Riedemann proposed openstack/nova master: Ensure source service is up before resizing/migrating https://review.opendev.org/700062
20:25:18	openstackgerrit	Matt Riedemann proposed openstack/nova master: Ensure source service is up before resizing/migrating https://review.opendev.org/700062
20:55:29	mriedem	artom: stephenfin: since you summoned me https://review.opendev.org/#/c/698530/
21:02:13	mriedem	and https://review.opendev.org/#/c/698532/
21:02:35	mriedem	do we want to wait for https://review.opendev.org/#/c/698260/ to get cleaned up and backported on top of the train backports? because if so, someone is likely going to have to clean up https://review.opendev.org/#/c/698260/ for sean
21:14:28	openstackgerrit	Matt Riedemann proposed openstack/nova master: FUP for in-place numa rebuild https://review.opendev.org/698260
21:14:44	mriedem	done ^ let's get that approved on master and stacked on top of the backports
21:36:42	openstackgerrit	Matt Riedemann proposed openstack/nova stable/pike: pike-only: remove broken non-voting ceph jobs https://review.opendev.org/700072
21:45:15	TheJulia	Are there any known issues with nova-compute on the master branch right now?
21:45:37	TheJulia	like... ones that might show up with grenade running
21:46:09	mriedem	that's pretty broad
21:46:32	mriedem	you have a grenade job that rebalances ironic nodes between 2 computes right?
21:46:51	mriedem	there is a known race issue with that https://review.opendev.org/#/c/694802/
21:47:05	mriedem	https://bugs.launchpad.net/nova/+bug/1841481
21:47:05	openstack	Launchpad bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
21:49:23	TheJulia	mriedem: single node stack where nova-compute seems to die after being upgraded and started up for a little while
21:50:09	TheJulia	mriedem: but, I'm only asking broadly because I've only glanced at the logs enough to have a feeling it is nova-compute. I've not gone back and gone through everything comparing out the smoke test times yet
21:50:28	TheJulia	hence, why I asked if there is anything known before I go dig for a few hours
21:50:55	openstackgerrit	Matt Riedemann proposed openstack/nova stable/ocata: ocata-only: drop non-voting ceph job https://review.opendev.org/700077
21:51:02	mriedem	then not that i know of
21:51:29	TheJulia	mriedem: for context, two grenade jobs, one does a full upgrade, the other does a partial upgrade for historical reasons and that breaks most cycles leading to me wanting to drink.
21:51:38	TheJulia	Okay, yippie!
21:51:39	TheJulia	:)
22:26:57	TheJulia	hmm, in rescheduling for post upgrade smoke testing
#openstack-nova - 2019-12-20
01:12:27	openstackgerrit	Merged openstack/nova stable/stein: Zuul v3: use devstack-plugin-nfs-tempest-full https://review.opendev.org/699924
06:41:36	openstackgerrit	Merged openstack/nova master: Add functional cross-cell revert test with detached volume https://review.opendev.org/695335
06:41:43	openstackgerrit	Merged openstack/nova master: Refresh target cell instance after finish_snapshot_based_resize_at_dest https://review.opendev.org/698787
08:23:47	openstackgerrit	Luigi Toscano proposed openstack/nova stable/rocky: Zuul v3: use devstack-plugin-nfs-tempest-full https://review.opendev.org/700121
08:26:53	openstackgerrit	XiaojueGuan proposed openstack/nova master: Fix error out with an UnboundException when in build_and_run_instance https://review.opendev.org/698140
08:50:03	openstackgerrit	Brin Zhang proposed openstack/nova master: Add api for instance action details https://review.opendev.org/700123
09:04:45	openstackgerrit	XiaojueGuan proposed openstack/nova master: Fix error out with an UnboundException when in build_and_run_instance https://review.opendev.org/698140
09:05:00	openstackgerrit	Brin Zhang proposed openstack/nova-specs master: Add composable flavor properties https://review.opendev.org/663563
09:10:15	openstackgerrit	Chason Chan proposed openstack/nova master: Add description of live_migration_timeout_action option https://review.opendev.org/700124
09:42:43	openstackgerrit	Stephen Finucane proposed openstack/nova stable/train: FUP for in-place numa rebuild https://review.opendev.org/700127
10:50:02	openstackgerrit	Lee Yarwood proposed openstack/nova master: docs: Add note about an image signature validation limitation when using rbd https://review.opendev.org/700152
10:50:22	lyarwood	stephenfin: ^ simple doc wrinkle if you have time today
10:51:45	stephenfin	looks sane. +2
10:59:06	openstackgerrit	Stephen Finucane proposed openstack/nova master: FakeDriver: adding and removing instances on live migration. https://review.opendev.org/243613
11:41:02	openstackgerrit	Stephen Finucane proposed openstack/nova master: Print help if nova-manage subcommand is not specified https://review.opendev.org/694806
11:41:57	openstackgerrit	Merged openstack/nova master: FUP for in-place numa rebuild https://review.opendev.org/698260
11:47:48	openstackgerrit	Shilpa Devharakar proposed openstack/nova master: Handle new is_volume_backend join column query https://review.opendev.org/694462
11:47:48	openstackgerrit	Shilpa Devharakar proposed openstack/nova master: Instance object changes for the new 'is_volume_backed' expected_attr https://review.opendev.org/694463
11:47:49	openstackgerrit	Shilpa Devharakar proposed openstack/nova master: Ignore root_gb if instance is booted from volume https://review.opendev.org/612626
12:09:07	openstackgerrit	Stephen Finucane proposed openstack/nova stable/train: FUP for in-place numa rebuild https://review.opendev.org/700127
12:10:54	stephenfin	lyarwood: fixed https://review.opendev.org/#/c/700127/
12:15:35	lyarwood	stephenfin: ack, grabbing lunch and then I'll queue that series up
12:28:17	stephenfin	bauzas: You bored? I have a load of patches with single +2 that could do with a little push
12:28:40	stephenfin	And I think you're the only other core still around. Lucky you ;)
13:59:56	bauzas	stephenfin: heh, sure
14:00:04	bauzas	stephenfin: sorry, I was on gym :)
14:09:13	mriedem	lyarwood: i'm +2 on these train numa rebuild backports now if you want to send them on https://review.opendev.org/#/q/topic:bug/1763766+status:open
14:10:01	lyarwood	mriedem: ack, have them open now, looking.
14:11:36	TheJulia	Can anyone provide background on the "forbidden trait COMPUTE_STATUS_DISABLED" ?
14:12:21	mriedem	TheJulia: https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#compute-disabled-status-support
14:14:50	TheJulia	hmm... then the question becomes why does nova think the nova-compute for ironic is no longer working when it just deleted an instance :\
14:15:04	TheJulia	well, minutes before in the same running process :\
14:15:10	TheJulia	logs
14:18:28	TheJulia	mriedem: is it the conductor that is making that declaration ?
14:20:58	mriedem	no,
14:21:22	mriedem	if the compute service is enabled/disabled in the rest api then the trait added to or removed from the associated nodes that service is managing
14:21:37	mriedem	some virt drivers (i think only libvirt) will auto-enable/disable if the hypervisor is down/up
14:21:48	mriedem	so it could also be the compute via the driver that sets it
14:22:14	mriedem	you can disable a compute service and still delete instances from it
14:22:17	TheJulia	I was thought that there was some automagical thing close to the compute driver... sifting through the n-cpu logs very carefully at the moment
14:22:20	mriedem	disable != down
14:22:29	TheJulia	disable from scheduling, that makes sense
14:22:38	mriedem	correct
14:27:39	mriedem	are you seeing the logging from here? https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L546-L553
14:28:44	TheJulia	mriedem: nope :\
14:29:41	mriedem	does the ironic grenade multinode job disable one compute before upgrading it which triggers a rebalance of nodes from one compute to another?
14:31:29	TheJulia	this is a single stack job so everything get stopped and restarted
14:32:30	TheJulia	I see a get to os-service early on, but no puts
14:32:32	TheJulia	err, posts
14:32:33	mriedem	and the ironic virt driver doesn't auto enable/disable the compute service does it?
14:32:41	mriedem	the disable/enable is a PUT

Earlier   Later