Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-10-01
08:33:28 lyarwood awesome thanks
08:35:12 openstackgerrit Lee Yarwood proposed openstack/nova stable/victoria: [stable-only] nova-live-migration: Disable libvirtd during negative tests https://review.opendev.org/754688
09:10:07 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701
10:17:09 stephenfin gibi: Think you could spin through https://review.opendev.org/#/q/topic:remove-xenapi+status:open before the week is out?
10:24:53 gibi stephenfin: I'm not comfortable making big changes to master during release time, can we take this after the release?
10:25:06 stephenfin oh, of course
10:30:34 openstackgerrit Balazs Gibizer proposed openstack/nova-specs master: Support interface attach with qos ports https://review.opendev.org/755477
11:13:34 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Add modernize-os-hypervisors-api spec https://review.opendev.org/755109
11:18:10 gibi lyarwood: it seem neutron-lib needs a new release to contain this: https://review.opendev.org/#/c/753230/
11:18:56 lyarwood gibi: ack thanks for looking into that!
11:19:18 lyarwood gibi: ./me wonders if neutron have nova-next in their check/gate?
11:19:28 gibi they don't I guess
11:19:44 lyarwood yeah might be something to suggest given the QoS coverage it has
11:22:02 gibi good point
11:22:35 gibi to be fair the normal qos coverage is in tempest. the nova-next only has the coverage for the qos part of the nova-manage placement heal_allocation CLI
11:23:20 lyarwood ah so we could add something to tempest to also cover this?
11:26:45 gibi I think it was added to the nova-next job as it has to create an invalid situation first then heal it
11:27:02 gibi in a real word missing port allocation only happen during upgrade from older openstack version
11:27:13 gibi we cannot really simulate that
11:27:26 gibi so we instead deleteing the allocation manually
11:27:29 gibi then try to heal it
11:29:20 lyarwood ack
11:30:03 lyarwood still would be nice if a neutron job could execute the test somewhere in their gate
11:30:17 lyarwood if we could break the test out into a seperate script somewhere
11:30:37 gibi yeah, I agree with that
11:37:02 gibi lyarwood: I guess the whole nova gate is broken now due to this
11:43:14 gibi lyarwood: I'll try to disable this part of the nova-next job as it will take time to get a new neutron-lib release
11:55:29 gibi lyarwood: also stable/victoria could have the same issue if https://review.opendev.org/#/c/755180 merges
12:05:46 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
12:05:52 gibi lyarwood: ^^
12:07:49 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
12:15:39 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
12:15:52 lyarwood gibi: ack thanks
12:16:18 gibi slaweq and lajoskatona are aware of the issue and working on a solution
12:17:03 lyarwood awesome thanks
12:17:27 lyarwood stephenfin: ^ the above should unblock the gate on master btw
12:23:02 openstackgerrit Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245
12:23:13 openstackgerrit Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245
12:31:07 openstackgerrit Elod Illes proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245
13:09:08 openstackgerrit Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175
13:14:04 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701
13:14:16 lyarwood kashyap: ^ think I found a n-cpu bug with QEMU v5.0.0, looks like we will need to rewrite our detach logic
13:14:44 lyarwood I'll write up a bug once the vanilla fedora-latest-virt-preview job has also hit it without q35
13:20:04 sean-k-mooney lyarwood: is this related the the qemu issue
13:20:08 sean-k-mooney on ubuntu
13:20:26 sean-k-mooney i.e. would the same rewite help
13:21:02 lyarwood sean-k-mooney: it's unrelated
13:22:50 lyarwood sean-k-mooney: https://github.com/qemu/qemu/blob/cbba3dc6ea3fc9aa66e9f9eb41051536e3ad7cd0/qdev-monitor.c#L886-L898 - n-cpu is tripping up over this check introduced in QEMU v5.0.0 as we fire off detach requests without waiting for libvirt to emit events.
13:23:20 lyarwood hmm unless
13:23:44 sean-k-mooney if (dev->pending_deleted_event) {
13:23:45 lyarwood are we hitting the bug this check was introduced to avoid?
13:23:47 sean-k-mooney that check
13:24:03 sean-k-mooney perhaps
13:24:22 sean-k-mooney https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b
13:24:26 lyarwood so the commit references PCI devices
13:24:37 lyarwood oh I wondeer
13:24:42 lyarwood interesting
13:25:01 sean-k-mooney On PCIe devices, this cancels the hot-unplug process.
13:25:13 lyarwood I don't recall seeing another request from libvirt to QEMU in the focal examples
13:25:15 lyarwood but maybe
13:25:31 sean-k-mooney if we retry it it would right
13:25:37 sean-k-mooney we just loop
13:25:48 lyarwood right but I can't recall seeing the requests overlap
13:25:59 lyarwood but the loop is pretty tight
13:26:12 lyarwood would explain why we only saw it on overloaded CI hosts
13:27:47 kashyap lyarwood: Hey, was out for a small walk; reading your comments in the change
13:28:37 kashyap That "unable to execute ... device_del' rings a bell
13:34:22 kashyap lyarwood: I think we _are_ hitting the bug the check was introduced to avoid, no? The QEMU commit skips repeated QMP 'device_del' calls
13:37:19 lyarwood kashyap: right and with >=v5.0.0 that raises an error, I was wondering above if prior to that on focal we are actually hitting the original issue and hot unplugs are being cancelled in QEMU
13:37:45 kashyap lyarwood: Ah, hence your comment on "why we only saw it on overloaded CI hosts"
13:38:01 lyarwood kashyap: either way the fix here seems to be to use libvirt events to wait until a detach either succeeds or fails before we retry
13:38:04 sean-k-mooney kashyap: right when we race with a slow detach due to load
13:38:45 sean-k-mooney lyarwood: ya so we should do that in either case and then we can use the ci to see if it fixes the ci issue with a DNM patch to swap the job to focal form bionic
13:39:16 sean-k-mooney lyarwood: we do sometime get random detach failure on bionic too
13:39:21 sean-k-mooney which could be caused by this
13:40:01 sean-k-mooney its much less common but i have seen many jobs fail for seminly unrelated volume detach issues
13:40:36 kashyap lyarwood: Yeah, it's a long-time TODO to wait for proper events, instead of polling :-( There should also be a documented TODO in driver.py/guest.py
14:23:01 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM libvirt: Remove loopingcall when detaching device https://review.opendev.org/755526
14:49:28 openstackgerrit sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642
14:49:37 openstackgerrit sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432
14:56:50 sean-k-mooney those are just rebases by the way ^ i want to see if the tempest failure is real or not
14:57:21 sean-k-mooney i dont think they are but a rebase never hurts
15:45:14 gibi nova weekly meeting starts in 15 minutes in #openstack-meeting-3
16:30:35 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
16:31:34 gibi lyarwood, stephenfin : could you hit it again? ^
16:32:32 lyarwood gibi: will do once it passes
16:32:46 gibi OK :)
16:33:05 gibi thanks
16:33:26 gibi I will leave for today in 30 minutes so I guess I won't see it pass
16:35:40 stephenfin #
16:35:55 stephenfin Done. If it doesn't pass, it simply won't merge :)
16:38:09 gibi thanks
21:09:26 openstackgerrit melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444
21:09:27 openstackgerrit melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055
21:09:28 openstackgerrit melanie witt proposed openstack/nova stable/pike: Add resource_class to fields in ironic node cache https://review.opendev.org/755653
#openstack-nova - 2020-10-02
07:41:51 openstackgerrit Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498
08:41:18 openstackgerrit Stephen Finucane proposed openstack/nova master: api-ref: Move 'os-agents' API to obsolete section https://review.opendev.org/755729
13:44:15 lyarwood stephenfin: https://review.opendev.org/#/c/755498/ can you hit this agian please
13:46:13 stephenfin done
13:51:14 lyarwood ta

Earlier   Later