Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-10-01
11:30:03 lyarwood still would be nice if a neutron job could execute the test somewhere in their gate
11:30:17 lyarwood if we could break the test out into a seperate script somewhere
11:30:37 gibi yeah, I agree with that
11:37:02 gibi lyarwood: I guess the whole nova gate is broken now due to this
11:43:14 gibi lyarwood: I'll try to disable this part of the nova-next job as it will take time to get a new neutron-lib release
11:55:29 gibi lyarwood: also stable/victoria could have the same issue if https://review.opendev.org/#/c/755180 merges
12:05:46 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
12:05:52 gibi lyarwood: ^^
12:07:49 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
12:15:39 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
12:15:52 lyarwood gibi: ack thanks
12:16:18 gibi slaweq and lajoskatona are aware of the issue and working on a solution
12:17:03 lyarwood awesome thanks
12:17:27 lyarwood stephenfin: ^ the above should unblock the gate on master btw
12:23:02 openstackgerrit Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245
12:23:13 openstackgerrit Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245
12:31:07 openstackgerrit Elod Illes proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245
13:09:08 openstackgerrit Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175
13:14:04 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701
13:14:16 lyarwood kashyap: ^ think I found a n-cpu bug with QEMU v5.0.0, looks like we will need to rewrite our detach logic
13:14:44 lyarwood I'll write up a bug once the vanilla fedora-latest-virt-preview job has also hit it without q35
13:20:04 sean-k-mooney lyarwood: is this related the the qemu issue
13:20:08 sean-k-mooney on ubuntu
13:20:26 sean-k-mooney i.e. would the same rewite help
13:21:02 lyarwood sean-k-mooney: it's unrelated
13:22:50 lyarwood sean-k-mooney: https://github.com/qemu/qemu/blob/cbba3dc6ea3fc9aa66e9f9eb41051536e3ad7cd0/qdev-monitor.c#L886-L898 - n-cpu is tripping up over this check introduced in QEMU v5.0.0 as we fire off detach requests without waiting for libvirt to emit events.
13:23:20 lyarwood hmm unless
13:23:44 sean-k-mooney if (dev->pending_deleted_event) {
13:23:45 lyarwood are we hitting the bug this check was introduced to avoid?
13:23:47 sean-k-mooney that check
13:24:03 sean-k-mooney perhaps
13:24:22 sean-k-mooney https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b
13:24:26 lyarwood so the commit references PCI devices
13:24:37 lyarwood oh I wondeer
13:24:42 lyarwood interesting
13:25:01 sean-k-mooney On PCIe devices, this cancels the hot-unplug process.
13:25:13 lyarwood I don't recall seeing another request from libvirt to QEMU in the focal examples
13:25:15 lyarwood but maybe
13:25:31 sean-k-mooney if we retry it it would right
13:25:37 sean-k-mooney we just loop
13:25:48 lyarwood right but I can't recall seeing the requests overlap
13:25:59 lyarwood but the loop is pretty tight
13:26:12 lyarwood would explain why we only saw it on overloaded CI hosts
13:27:47 kashyap lyarwood: Hey, was out for a small walk; reading your comments in the change
13:28:37 kashyap That "unable to execute ... device_del' rings a bell
13:34:22 kashyap lyarwood: I think we _are_ hitting the bug the check was introduced to avoid, no? The QEMU commit skips repeated QMP 'device_del' calls
13:37:19 lyarwood kashyap: right and with >=v5.0.0 that raises an error, I was wondering above if prior to that on focal we are actually hitting the original issue and hot unplugs are being cancelled in QEMU
13:37:45 kashyap lyarwood: Ah, hence your comment on "why we only saw it on overloaded CI hosts"
13:38:01 lyarwood kashyap: either way the fix here seems to be to use libvirt events to wait until a detach either succeeds or fails before we retry
13:38:04 sean-k-mooney kashyap: right when we race with a slow detach due to load
13:38:45 sean-k-mooney lyarwood: ya so we should do that in either case and then we can use the ci to see if it fixes the ci issue with a DNM patch to swap the job to focal form bionic
13:39:16 sean-k-mooney lyarwood: we do sometime get random detach failure on bionic too
13:39:21 sean-k-mooney which could be caused by this
13:40:01 sean-k-mooney its much less common but i have seen many jobs fail for seminly unrelated volume detach issues
13:40:36 kashyap lyarwood: Yeah, it's a long-time TODO to wait for proper events, instead of polling :-( There should also be a documented TODO in driver.py/guest.py
14:23:01 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM libvirt: Remove loopingcall when detaching device https://review.opendev.org/755526
14:49:28 openstackgerrit sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642
14:49:37 openstackgerrit sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432
14:56:50 sean-k-mooney those are just rebases by the way ^ i want to see if the tempest failure is real or not
14:57:21 sean-k-mooney i dont think they are but a rebase never hurts
15:45:14 gibi nova weekly meeting starts in 15 minutes in #openstack-meeting-3
16:30:35 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
16:31:34 gibi lyarwood, stephenfin : could you hit it again? ^
16:32:32 lyarwood gibi: will do once it passes
16:32:46 gibi OK :)
16:33:05 gibi thanks
16:33:26 gibi I will leave for today in 30 minutes so I guess I won't see it pass
16:35:40 stephenfin #
16:35:55 stephenfin Done. If it doesn't pass, it simply won't merge :)
16:38:09 gibi thanks
21:09:26 openstackgerrit melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444
21:09:27 openstackgerrit melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055
21:09:28 openstackgerrit melanie witt proposed openstack/nova stable/pike: Add resource_class to fields in ironic node cache https://review.opendev.org/755653
#openstack-nova - 2020-10-02
07:41:51 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
08:41:18 openstackgerrit Stephen Finucane proposed openstack/nova master: api-ref: Move 'os-agents' API to obsolete section https://review.opendev.org/755729
13:44:15 lyarwood stephenfin: https://review.opendev.org/#/c/755498/ can you hit this agian please
13:46:13 stephenfin done
13:51:14 lyarwood ta
14:14:37 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Increase incremental and max sleep time during detach https://review.opendev.org/755799
14:32:22 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Increase incremental and max sleep time during detach https://review.opendev.org/755799
16:49:40 openstackgerrit Merged openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
17:01:58 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-evacuate: Disable libvirtd service and sockets during negative tests https://review.opendev.org/754684
17:02:26 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985
17:02:51 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986
17:17:09 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add tests for mixed CPU policy https://review.opendev.org/755852
17:17:47 stephenfin sean-k-mooney, artom: Should be an easy fix if either of you are bored :-P I'm off for a run though so will take a look next week if not https://bugs.launchpad.net/nova/+bug/1898272
17:17:47 openstack Launchpad bug 1898272 in OpenStack Compute (nova) ""mixed" policy calculations don't account for host cells with no free shared CPUs" [Medium,Confirmed] - Assigned to Stephen Finucane (stephenfinucane)
17:19:15 artom stephenfin, fuuuun
17:19:26 stephenfin you're telling me :(
17:54:19 sean-k-mooney hum ya that a bug
17:54:46 sean-k-mooney you could have moved it to triaged too
17:55:39 sean-k-mooney stephenfin: does mixed cpus support numa at all
17:56:45 sean-k-mooney we shoudl be able to test this with whitebox by the way
17:56:57 sean-k-mooney we can test it with a functional test too
17:58:01 sean-k-mooney stephenfin: thats failing because we are not tracking the numa affinity fo the shared cores yes
18:04:38 sean-k-mooney ah i see the fixt thats subtle
18:11:39 sean-k-mooney stephenfin: there is more legecy debt there
18:11:52 sean-k-mooney stephenfin: the cpu toplogy is not per numa node
18:11:54 sean-k-mooney its per vm
18:12:07 sean-k-mooney so we should not have a toplogy filed in the instance cell at all

Earlier   Later