Posted	Nick	Remark
#openstack-nova - 2020-07-22
13:12:37	openstackgerrit	Lee Yarwood proposed openstack/nova stable/queens: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices https://review.opendev.org/742424
13:12:37	openstackgerrit	Lee Yarwood proposed openstack/nova stable/queens: sync_guest_time: use the proper errno https://review.opendev.org/742423
13:21:04	lyarwood	bauzas: https://review.opendev.org/#/c/729595/ - would you mind hitting this again?
13:21:14	bauzas	ack
13:21:56	lyarwood	https://review.opendev.org/#/c/729261/ could also use some additional core review if anyone has bandwidth
13:22:51	lyarwood	and https://review.opendev.org/#/c/724842/ both are pretty simple
13:27:35	gibi	lyarwood: added to my queue but I will focus on the sriov attach this afternoon so I can get to your patches tomorrow
13:29:21	bauzas	lyarwood: stephenfin: gibi: others, honestly, I'd be super happy if we could just provide a commit tag for py3-only changes
13:29:30	bauzas	something like "Py3-only"
13:29:41	bauzas	so, stable cores looking at it would see it
13:29:56	bauzas	I know how this can be difficult to do it every time
13:30:20	bauzas	but if people don't think about py3 styling, then I'm pretty sure we will get bugs in say one year
13:30:38	bauzas	context https://review.opendev.org/#/c/729595/6/nova/virt/libvirt/guest.py@868
13:30:51	bauzas	I totally understand that people want to use py3 styling
13:31:17	bauzas	but honestly, if you want to do it, think about backports
13:31:35	gibi	bauzas: I would expect that such thing is caught by the unit test on the backported patch
13:31:36	lyarwood	stable cores would see this, py2 jobs would bork due to this.
13:32:12	lyarwood	also, the py3 style boat has sailed
13:32:20	bauzas	gibi: well, for https://review.opendev.org/#/c/729595/6/nova/virt/libvirt/guest.py@868 we don't test it, right?
13:32:23	lyarwood	stephenfin has been landing stuff for most of the cycle already
13:32:51	bauzas	lyarwood: I don't see how CI can see it : https://review.opendev.org/#/c/729595/6/nova/virt/libvirt/guest.py@868
13:32:53	lyarwood	bauzas: that's a DEBUG log in a codepath we hit loads in our tempest tests so that would definitley catch it
13:32:54	bauzas	with py3
13:33:03	bauzas	py2 rather
13:33:12	lyarwood	we log at DEBUG in CI
13:33:19	lyarwood	we always run tests that hit that codepath
13:33:22	bauzas	again, there will be conditions where some code wouldn't be tested by CI and then we would get bugs
13:34:18	bauzas	this is not that I hate py3 styling, I'm just conservative with any change that can be backported, that's it
13:34:26	bauzas	for a new feature ? sure.
13:34:57	gibi	I would rather improve our test coverage than rely on a manual commit tag
13:36:28	bauzas	meh, ok, you're right
13:39:00	bauzas	lyarwood: fwiw, given you just told that stein jobs would hit this, I upvoted
13:39:45	bauzas	my main concern is, please be explicit when you write py3-styling and be clear whether you already thought about backports
13:39:55	bauzas	the better commit msg you write, the easier it would be for reviewers
13:40:09	bauzas	that's it, I don't want to rathole about f-strings anymore
13:41:13	lyarwood	bauzas: FWIW I have no intention of backporting this, it was more to help debug some failures in master CI that look performance related
13:45:07	gibi	bauzas: I agree that better commit message helps
13:45:23	bauzas	lyarwood: ack, gtk
13:45:25	bauzas	thanks
13:50:30	gibi	stephenfin: when the reno patch merges for the use-pcpu-and-vcpu-in-one-instance does it mean that the feature is done?
13:52:19	gibi	I'm asking this as there are other patches open but they seem to be good to have not must hav
13:52:21	stephenfin	gibi: Yep. The only thing that we don't support after that is requesting PCPU + VCPU via 'resources:' extra specs, which we're deferring since it's (a) low priority and (b) interferes with the NUMA in placement work
13:52:37	gibi	OK
13:52:58	gibi	then I will mark the bp complete (with a note about the deferred part) after the reno merge
13:53:01	gibi	s
13:53:23	gibi	thanks for checking it
13:57:00	artom	gibi, replied on https://review.opendev.org/#/c/741436/, but you seem to have found the right track on your own :)
13:58:11	gibi	artom: thanks. I will check if I can check for vlan
13:58:46	gibi	artom: thanks for pointing me to the code that was supposed to clean up the metadata I missed that orignally
13:59:02	artom	gibi, it was well hidden. Blame mriedem ;)
13:59:19	gibi	:)
13:59:31	artom	(For real, it was his idea - performance-wise, it's the good choice. Code-understandability-wise, clearly not so great ;)
14:01:50	gibi	artom: as far as I see if I simply remove the tag in vif conditional then the code still don't do any expensive things, jut iterate through instance.device_metadata to find the metadata by vif.address
14:02:18	artom	gibi, maybe the easiest way to do it is to do a reverse lookup kind of thing: just loop through the device metadata regardless, and if the mac matches the vif mac, delete that entry.
14:02:27	artom	gibi, hah, jiinx
14:02:49	gibi	that the baseline code does today
14:02:52	artom	IOW, as you said, remove the conditional
14:02:58	gibi	https://opendev.org/openstack/nova/src/branch/master/nova/network/neutron.py#L1788
14:03:02	gibi	Ok
14:06:24	openstack	Launchpad bug 1775947 in tempest "tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest failing" [Medium,Confirmed]
14:06:24	artom	Which reminds me, I really need to get https://bugs.launchpad.net/tempest/+bug/1775947 figured out
14:06:26	artom	And fixed
14:21:36	openstackgerrit	Merged openstack/nova master: libvirt: Remove workaround for really old QEMU https://review.opendev.org/740335
14:34:37	openstackgerrit	Artom Lifshitz proposed openstack/nova master: DNM: extra logging for tagged device detach https://review.opendev.org/742450
14:39:02	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Delete non tag related device metadata during detach https://review.opendev.org/742451
14:42:41	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Delete non tag related device metadata during detach https://review.opendev.org/741436
16:03:10	openstackgerrit	Balazs Gibizer proposed openstack/nova master: [WIP] Support SRIOV interface attach and detach https://review.opendev.org/740995
16:09:56	openstackgerrit	Merged openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485
18:22:59	openstackgerrit	Gonéri Le Bouder proposed openstack/nova master: QEMU/KVM: accept vmxnet3 NIC https://review.opendev.org/738695
18:41:20	openstackgerrit	Gonéri Le Bouder proposed openstack/nova master: QEMU/KVM: accept vmxnet3 NIC https://review.opendev.org/738695
18:48:45	artom	Random thought - someone should probably remove efried and mriedem from https://review.opendev.org/#/admin/groups/25,members, no?
18:48:53	artom	Nothing against you fellas, obviously :)
19:00:32	mriedem	i'm removed myseflf
19:00:38	mriedem	*myself even
19:01:14	mriedem	and https://review.opendev.org/#/admin/groups/1936,members
19:07:58	artom	mriedem, RIP in pieces :(
19:08:21	artom	I was randomly linking that page to someone who needed core review
19:08:34	artom	This isn't a coup, is what I'm saying :)
19:21:03	mriedem	yup, np
20:22:57	jralbert	in 'nova hypervisor-stats', and have no entries in the allocations table in the nova_api database. Running nova-manage placement heal_allocations for affected instances returns an error that "Compute host <compute_node> could not be found", but all the compute nodes are visible in nova-manage cell_v2 list_hosts. Has anyone encountered this? Any
20:22:57	jralbert	I've raised this question in #openstack and so far received no response; anybody here have any advice? I've just upgraded from Rocky to Stein, and Placement seems to have gotten very confused about resources in-use vs available. A number of compute nodes with running instances which are known to 'openstack server show' are reporting 0 running_vms
20:22:58	jralbert	suggestions on how to troubleshoot it further?
20:39:15	melwitt	jralbert: it sounds like your nova.instances records instance.host and instance.node aren't matching any records in nova.compute_nodes 'host' and 'hypervisor_hostname', as for why you're getting Compute host could not be found
20:40:30	melwitt	did you replace any nova-compute services during the upgrade with different hostnames or something like that?
20:41:36	jralbert	melwitt: Thanks, I'll have a look at those tables. We didn't change any hostnames during the upgrade, although because it was being orchestrated by OSA there may have been steps taken we weren't fully aware of
20:43:49	melwitt	ok. I can't tell you how things got into that state but just for the sake of understanding why heal_allocations won't work: it will look up the compute node for an instance by querying for the instance.host and instance.node in the nova.compute_nodes table and if it finds no match, you'll get the Compute host could not be found error
20:45:32	melwitt	I'm not aware of any change from rocky to stein that could cause something like this. are you running with integrated placement (in nova) or did you split out to running extracted placement I wonder?
20:46:38	jralbert	I'm not 100% sure what integrated/extracted placement means, but I can find out - we're relying mostly on OSA defaults with some overrides, but that's not one I'm aware of.
20:46:44	melwitt	if you've moved to extracted placement, there are additional steps you need to follow for an upgrade, linked from here: https://docs.openstack.org/releasenotes/nova/stein.html#prelude
20:47:02	jralbert	Interestingly, there is a bit of a hint in the tables: http://paste.openstack.org/show/796222/
20:47:05	melwitt	https://docs.openstack.org/placement/latest/admin/upgrade-to-stein.html
20:47:29	jralbert	The compute nodes get '.openstack.local' hostnames in instances, but a genuine FQDN in compute_nodes
20:48:33	melwitt	the reason I ask about extracted placement is because you said your nova_api.allocations table is empty. did you already move to a separate placement db? do you have placement.allocations records?
20:49:13	jralbert	Sorry, the allocations table isn't completely empty - it's just missing *some* instances on *some* compute nodes
20:49:30	melwitt	ok, I think that difference in the nodename is causing the failure to lookup the compute nodes for the heal_allocations command
20:49:46	jralbert	we became aware of this because our instance/vcpu metrics suddenly dropped by about half during the upgrade, but all the instances were still running that had been
20:50:19	melwitt	this is where I see the lookup by host and node for heal_allocations https://github.com/openstack/nova/blob/stable/stein/nova/cmd/manage.py#L1829
20:50:42	jralbert	So would I safely be able to update those definitions directly in the database, or is there a better-bounded API/CLI approach?
20:52:32	jralbert	I guess I would have hoped that compute nodes would be identified by uuids since names (especially hostnames) often produce collisions like this, but I'm guessing there's a legacy reason why this is the way it is.
20:55:09	melwitt	yeah, they are not linked by uuid. I don't know the history behind it. probably something like since you can delete/create services via the API, if we didn't link by name, you'd lose the association for all your instances even if the hostname is the same (service create makes a new uuid)

Earlier   Later