Posted	Nick	Remark
#openstack-nova - 2020-10-01
13:26:12	lyarwood	would explain why we only saw it on overloaded CI hosts
13:27:47	kashyap	lyarwood: Hey, was out for a small walk; reading your comments in the change
13:28:37	kashyap	That "unable to execute ... device_del' rings a bell
13:34:22	kashyap	lyarwood: I think we _are_ hitting the bug the check was introduced to avoid, no? The QEMU commit skips repeated QMP 'device_del' calls
13:37:19	lyarwood	kashyap: right and with >=v5.0.0 that raises an error, I was wondering above if prior to that on focal we are actually hitting the original issue and hot unplugs are being cancelled in QEMU
13:37:45	kashyap	lyarwood: Ah, hence your comment on "why we only saw it on overloaded CI hosts"
13:38:01	lyarwood	kashyap: either way the fix here seems to be to use libvirt events to wait until a detach either succeeds or fails before we retry
13:38:04	sean-k-mooney	kashyap: right when we race with a slow detach due to load
13:38:45	sean-k-mooney	lyarwood: ya so we should do that in either case and then we can use the ci to see if it fixes the ci issue with a DNM patch to swap the job to focal form bionic
13:39:16	sean-k-mooney	lyarwood: we do sometime get random detach failure on bionic too
13:39:21	sean-k-mooney	which could be caused by this
13:40:01	sean-k-mooney	its much less common but i have seen many jobs fail for seminly unrelated volume detach issues
13:40:36	kashyap	lyarwood: Yeah, it's a long-time TODO to wait for proper events, instead of polling :-( There should also be a documented TODO in driver.py/guest.py
14:23:01	openstackgerrit	Lee Yarwood proposed openstack/nova master: DNM libvirt: Remove loopingcall when detaching device https://review.opendev.org/755526
14:49:28	openstackgerrit	sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642
14:49:37	openstackgerrit	sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432
14:56:50	sean-k-mooney	those are just rebases by the way ^ i want to see if the tempest failure is real or not
14:57:21	sean-k-mooney	i dont think they are but a rebase never hurts
15:45:14	gibi	nova weekly meeting starts in 15 minutes in #openstack-meeting-3
16:30:35	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
16:31:34	gibi	lyarwood, stephenfin : could you hit it again? ^
16:32:32	lyarwood	gibi: will do once it passes
16:32:46	gibi	OK :)
16:33:05	gibi	thanks
16:33:26	gibi	I will leave for today in 30 minutes so I guess I won't see it pass
16:35:40	stephenfin	#
16:35:55	stephenfin	Done. If it doesn't pass, it simply won't merge :)
16:38:09	gibi	thanks
21:09:26	openstackgerrit	melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444
21:09:27	openstackgerrit	melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055
21:09:28	openstackgerrit	melanie witt proposed openstack/nova stable/pike: Add resource_class to fields in ironic node cache https://review.opendev.org/755653
#openstack-nova - 2020-10-02
07:41:51	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
08:41:18	openstackgerrit	Stephen Finucane proposed openstack/nova master: api-ref: Move 'os-agents' API to obsolete section https://review.opendev.org/755729
13:44:15	lyarwood	stephenfin: https://review.opendev.org/#/c/755498/ can you hit this agian please
13:46:13	stephenfin	done
13:51:14	lyarwood	ta
14:14:37	openstackgerrit	Lee Yarwood proposed openstack/nova master: WIP libvirt: Increase incremental and max sleep time during detach https://review.opendev.org/755799
14:32:22	openstackgerrit	Lee Yarwood proposed openstack/nova master: WIP libvirt: Increase incremental and max sleep time during detach https://review.opendev.org/755799
16:49:40	openstackgerrit	Merged openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
17:01:58	openstackgerrit	Lee Yarwood proposed openstack/nova master: nova-evacuate: Disable libvirtd service and sockets during negative tests https://review.opendev.org/754684
17:02:26	openstackgerrit	Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985
17:02:51	openstackgerrit	Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986
17:17:09	openstackgerrit	Stephen Finucane proposed openstack/nova master: functional: Add tests for mixed CPU policy https://review.opendev.org/755852
17:17:47	stephenfin	sean-k-mooney, artom: Should be an easy fix if either of you are bored :-P I'm off for a run though so will take a look next week if not https://bugs.launchpad.net/nova/+bug/1898272
17:17:47	openstack	Launchpad bug 1898272 in OpenStack Compute (nova) ""mixed" policy calculations don't account for host cells with no free shared CPUs" [Medium,Confirmed] - Assigned to Stephen Finucane (stephenfinucane)
17:19:15	artom	stephenfin, fuuuun
17:19:26	stephenfin	you're telling me :(
17:54:19	sean-k-mooney	hum ya that a bug
17:54:46	sean-k-mooney	you could have moved it to triaged too
17:55:39	sean-k-mooney	stephenfin: does mixed cpus support numa at all
17:56:45	sean-k-mooney	we shoudl be able to test this with whitebox by the way
17:56:57	sean-k-mooney	we can test it with a functional test too
17:58:01	sean-k-mooney	stephenfin: thats failing because we are not tracking the numa affinity fo the shared cores yes
18:04:38	sean-k-mooney	ah i see the fixt thats subtle
18:11:39	sean-k-mooney	stephenfin: there is more legecy debt there
18:11:52	sean-k-mooney	stephenfin: the cpu toplogy is not per numa node
18:11:54	sean-k-mooney	its per vm
18:12:07	sean-k-mooney	so we should not have a toplogy filed in the instance cell at all
18:13:13	sean-k-mooney	so like the pagesize we should be pulling that field out of the instance numa cell object
18:14:39	sean-k-mooney	both of these https://github.com/openstack/nova/blob/master/nova/objects/instance_numa.py#L79-L81 should be removed
18:15:08	sean-k-mooney	and the cpu_thread_policy
18:16:11	sean-k-mooney	they coudl be moved to the InstanceNUMATopology object i guess
18:16:31	sean-k-mooney	although they are not really realted to numa but it would be more correct
18:16:55	sean-k-mooney	stephenfin: can you add a todo to remove them
18:24:03	sean-k-mooney	stephenfin: for what its worth i dont think we need the siblings field on the instance_cell object either
18:24:26	sean-k-mooney	i think we only ever use teh siblings proerty on the host cell object not the property ihn the instnace cell
18:24:34	sean-k-mooney	at least based on http://codesearch.openstack.org/?q=siblings&i=nope&files=&repos=openstack/nova
18:24:39	sean-k-mooney	its not obvious if we do
18:29:07	sean-k-mooney	the inital implemenation was quite bad at stashing things in the instnace cell object for convenince hten actully ensuign we only had one souce or truth
#openstack-nova - 2020-10-03
02:41:45	openstackgerrit	melanie witt proposed openstack/nova master: Initialize global data separately and run_once in WSGI app init https://review.opendev.org/733627
04:53:07	openstackgerrit	melanie witt proposed openstack/nova master: Initialize global data separately and run_once in WSGI app init https://review.opendev.org/733627
06:03:27	openstackgerrit	Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175
12:18:52	openstackgerrit	Lee Yarwood proposed openstack/nova master: bindep: Install python3 and python3-devel on CentOS 8 and Fedora https://review.opendev.org/755928
12:21:01	openstackgerrit	Lee Yarwood proposed openstack/nova master: bindep: Install python3 and python3-devel on CentOS 8 and Fedora https://review.opendev.org/755928
#openstack-nova - 2020-10-04
18:10:49	openstackgerrit	Merged openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985
18:14:57	openstackgerrit	Merged openstack/nova master: nova-evacuate: Disable libvirtd service and sockets during negative tests https://review.opendev.org/754684
#openstack-nova - 2020-10-05
03:47:23	openstackgerrit	Wenping Song proposed openstack/nova master: Add instance project_id for cyborg arq https://review.opendev.org/738428
07:02:02	openstackgerrit	Balazs Gibizer proposed openstack/nova master: Revert "Temporarily disable parts of heal port allocation test" https://review.opendev.org/755995
08:37:22	hemanth_n	gibi: thanks for the review on https://review.opendev.org/#/c/749175/ last thursday, sean have clarified on your comments and i added them as code comments to avoid confusion..
08:46:17	openstackgerrit	Lee Yarwood proposed openstack/nova master: zuul: Merge nova-evacuate into nova-multinode-live-migration https://review.opendev.org/755309
08:58:29	gibi	hemanth_n: I will go back to that review today
08:59:22	hemanth_n	gibi: thanks
09:47:53	jiriprox	Hi, I would need your support. We are using kolla images for train release from docker.io and last week on Tuesday images stopped working in our environment. PCI passthrough devices a are properly detected on compute node but they are not stored to DB. When we check corresponding table in nova db on controller it is empty :-(
10:23:48	openstackgerrit	Wenping Song proposed openstack/nova master: Add instance project_id for cyborg arq https://review.opendev.org/738428
10:33:01	openstackgerrit	Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701
10:39:47	gibi	jiriprox: have you traced back the kolla image changes to a certain nova version change in those imageS?
10:44:13	jiriprox	@gibi No. I don't know how to track which source code version was used in docker image pulled from docker.io with tag train
10:44:41	gibi	jiriprox: do you know which was the last docker image version that worked for you?
10:47:31	jiriprox	@gibi The last one working was pulled 25.9. It stopped working with images pulled 29.9.
10:48:11	gibi	jiriprox: could you check with the kolla developers if they can map those dates to nova versions?
10:48:27	gibi	sean-k-mooney: Do you happen to know such mapping ? ^^
10:50:42	jiriprox	@gibi I don't know whom to ask. Who is responsible for building kolla images and uploading them to docker.io?
10:51:33	gibi	jiriprox: I suggest to ask around in #openstack-kolla channel
10:55:04	sean-k-mooney	gibi: i dont think that exists
10:55:29	sean-k-mooney	but you could possibley add it via build metadata at least in the source build
10:56:11	sean-k-mooney	the base file has https://github.com/openstack/kolla/blob/master/docker/nova/nova-base/Dockerfile.j2#L3
10:56:29	sean-k-mooney	{% block labels %}
10:56:32	sean-k-mooney	LABEL maintainer="{{ maintainer }}" name="{{ image_name }}" build-date="{{ build_date }}"
10:56:33	sean-k-mooney	{% endblock %}
10:56:57	sean-k-mooney	you could add other labels with the nova version

Earlier   Later