| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-01 | |||
| 08:13:27 | lyarwood | np | |
| 08:13:52 | gibi | hemanth_n: that patch is on my review list for long | |
| 08:14:57 | gibi | hemanth_n: I see that sean-k-mooney1 is +1 on it already so I will try to read it today | |
| 08:15:08 | gibi | hemanth_n: is there any special aspect I should be aware of? | |
| 08:15:53 | hemanth_n | gibi: cool thanks .. nope except its a specific scenario which is mentioned in the commit message and bug description | |
| 08:17:07 | gibi | hemanth_n: I saw that you managed to add a functional test for it. thanks that helps a lot | |
| 08:17:32 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/victoria: [stable-only] nova-live-migration: Disable libvirtd during negative tests https://review.opendev.org/754688 | |
| 08:25:09 | frickler | lyarwood: the right solution would be to move that job away from using ds-gate, I'm not sure if someone has started doing that yet. even if we keep ds-gate working for victoria, that should be considered unstable and very much deprecated | |
| 08:25:56 | lyarwood | frickler: yeah I'm working on that in master, just need sort out a multinode ceph job and we should be good to backport to stable/victoria | |
| 08:26:09 | lyarwood | frickler: and agreed, this is just for a few days hopefully | |
| 08:27:26 | lyarwood | gibi: are you aware of https://zuul.opendev.org/t/openstack/builds?job_name=nova-next - looks like gate/post_test_hook.sh is failing when it calls `openstack port unset --binding-profile allocation port-normal-qos` | |
| 08:27:41 | lyarwood | on master at least | |
| 08:28:02 | gibi | lyarwood: thanks for the ping | |
| 08:28:10 | gibi | I haven't seen it yet | |
| 08:29:50 | lyarwood | gibi: np, can't see anything obvious in nova master that could be causing it, I assume something in Neutron has changed? | |
| 08:32:22 | gibi | lyarwood: I will ping lajoskatona on Neutron side, he did some recent changes around qos | |
| 08:33:28 | lyarwood | awesome thanks | |
| 08:35:12 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/victoria: [stable-only] nova-live-migration: Disable libvirtd during negative tests https://review.opendev.org/754688 | |
| 09:10:07 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701 | |
| 10:17:09 | stephenfin | gibi: Think you could spin through https://review.opendev.org/#/q/topic:remove-xenapi+status:open before the week is out? | |
| 10:24:53 | gibi | stephenfin: I'm not comfortable making big changes to master during release time, can we take this after the release? | |
| 10:25:06 | stephenfin | oh, of course | |
| 10:30:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Support interface attach with qos ports https://review.opendev.org/755477 | |
| 11:13:34 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Add modernize-os-hypervisors-api spec https://review.opendev.org/755109 | |
| 11:18:10 | gibi | lyarwood: it seem neutron-lib needs a new release to contain this: https://review.opendev.org/#/c/753230/ | |
| 11:18:56 | lyarwood | gibi: ack thanks for looking into that! | |
| 11:19:18 | lyarwood | gibi: ./me wonders if neutron have nova-next in their check/gate? | |
| 11:19:28 | gibi | they don't I guess | |
| 11:19:44 | lyarwood | yeah might be something to suggest given the QoS coverage it has | |
| 11:22:02 | gibi | good point | |
| 11:22:35 | gibi | to be fair the normal qos coverage is in tempest. the nova-next only has the coverage for the qos part of the nova-manage placement heal_allocation CLI | |
| 11:23:20 | lyarwood | ah so we could add something to tempest to also cover this? | |
| 11:26:45 | gibi | I think it was added to the nova-next job as it has to create an invalid situation first then heal it | |
| 11:27:02 | gibi | in a real word missing port allocation only happen during upgrade from older openstack version | |
| 11:27:13 | gibi | we cannot really simulate that | |
| 11:27:26 | gibi | so we instead deleteing the allocation manually | |
| 11:27:29 | gibi | then try to heal it | |
| 11:29:20 | lyarwood | ack | |
| 11:30:03 | lyarwood | still would be nice if a neutron job could execute the test somewhere in their gate | |
| 11:30:17 | lyarwood | if we could break the test out into a seperate script somewhere | |
| 11:30:37 | gibi | yeah, I agree with that | |
| 11:37:02 | gibi | lyarwood: I guess the whole nova gate is broken now due to this | |
| 11:43:14 | gibi | lyarwood: I'll try to disable this part of the nova-next job as it will take time to get a new neutron-lib release | |
| 11:55:29 | gibi | lyarwood: also stable/victoria could have the same issue if https://review.opendev.org/#/c/755180 merges | |
| 12:05:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 12:05:52 | gibi | lyarwood: ^^ | |
| 12:07:49 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 12:15:39 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 12:15:52 | lyarwood | gibi: ack thanks | |
| 12:16:18 | gibi | slaweq and lajoskatona are aware of the issue and working on a solution | |
| 12:17:03 | lyarwood | awesome thanks | |
| 12:17:27 | lyarwood | stephenfin: ^ the above should unblock the gate on master btw | |
| 12:23:02 | openstackgerrit | Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245 | |
| 12:23:13 | openstackgerrit | Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245 | |
| 12:31:07 | openstackgerrit | Elod Illes proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245 | |
| 13:09:08 | openstackgerrit | Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175 | |
| 13:14:04 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701 | |
| 13:14:16 | lyarwood | kashyap: ^ think I found a n-cpu bug with QEMU v5.0.0, looks like we will need to rewrite our detach logic | |
| 13:14:44 | lyarwood | I'll write up a bug once the vanilla fedora-latest-virt-preview job has also hit it without q35 | |
| 13:20:04 | sean-k-mooney | lyarwood: is this related the the qemu issue | |
| 13:20:08 | sean-k-mooney | on ubuntu | |
| 13:20:26 | sean-k-mooney | i.e. would the same rewite help | |
| 13:21:02 | lyarwood | sean-k-mooney: it's unrelated | |
| 13:22:50 | lyarwood | sean-k-mooney: https://github.com/qemu/qemu/blob/cbba3dc6ea3fc9aa66e9f9eb41051536e3ad7cd0/qdev-monitor.c#L886-L898 - n-cpu is tripping up over this check introduced in QEMU v5.0.0 as we fire off detach requests without waiting for libvirt to emit events. | |
| 13:23:20 | lyarwood | hmm unless | |
| 13:23:44 | sean-k-mooney | if (dev->pending_deleted_event) { | |
| 13:23:45 | lyarwood | are we hitting the bug this check was introduced to avoid? | |
| 13:23:47 | sean-k-mooney | that check | |
| 13:24:03 | sean-k-mooney | perhaps | |
| 13:24:22 | sean-k-mooney | https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b | |
| 13:24:26 | lyarwood | so the commit references PCI devices | |
| 13:24:37 | lyarwood | oh I wondeer | |
| 13:24:42 | lyarwood | interesting | |
| 13:25:01 | sean-k-mooney | On PCIe devices, this cancels the hot-unplug process. | |
| 13:25:13 | lyarwood | I don't recall seeing another request from libvirt to QEMU in the focal examples | |
| 13:25:15 | lyarwood | but maybe | |
| 13:25:31 | sean-k-mooney | if we retry it it would right | |
| 13:25:37 | sean-k-mooney | we just loop | |
| 13:25:48 | lyarwood | right but I can't recall seeing the requests overlap | |
| 13:25:59 | lyarwood | but the loop is pretty tight | |
| 13:26:12 | lyarwood | would explain why we only saw it on overloaded CI hosts | |
| 13:27:47 | kashyap | lyarwood: Hey, was out for a small walk; reading your comments in the change | |
| 13:28:37 | kashyap | That "unable to execute ... device_del' rings a bell | |
| 13:34:22 | kashyap | lyarwood: I think we _are_ hitting the bug the check was introduced to avoid, no? The QEMU commit skips repeated QMP 'device_del' calls | |
| 13:37:19 | lyarwood | kashyap: right and with >=v5.0.0 that raises an error, I was wondering above if prior to that on focal we are actually hitting the original issue and hot unplugs are being cancelled in QEMU | |
| 13:37:45 | kashyap | lyarwood: Ah, hence your comment on "why we only saw it on overloaded CI hosts" | |
| 13:38:01 | lyarwood | kashyap: either way the fix here seems to be to use libvirt events to wait until a detach either succeeds or fails before we retry | |
| 13:38:04 | sean-k-mooney | kashyap: right when we race with a slow detach due to load | |
| 13:38:45 | sean-k-mooney | lyarwood: ya so we should do that in either case and then we can use the ci to see if it fixes the ci issue with a DNM patch to swap the job to focal form bionic | |
| 13:39:16 | sean-k-mooney | lyarwood: we do sometime get random detach failure on bionic too | |
| 13:39:21 | sean-k-mooney | which could be caused by this | |
| 13:40:01 | sean-k-mooney | its much less common but i have seen many jobs fail for seminly unrelated volume detach issues | |
| 13:40:36 | kashyap | lyarwood: Yeah, it's a long-time TODO to wait for proper events, instead of polling :-( There should also be a documented TODO in driver.py/guest.py | |
| 14:23:01 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM libvirt: Remove loopingcall when detaching device https://review.opendev.org/755526 | |
| 14:49:28 | openstackgerrit | sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642 | |
| 14:49:37 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432 | |
| 14:56:50 | sean-k-mooney | those are just rebases by the way ^ i want to see if the tempest failure is real or not | |
| 14:57:21 | sean-k-mooney | i dont think they are but a rebase never hurts | |
| 15:45:14 | gibi | nova weekly meeting starts in 15 minutes in #openstack-meeting-3 | |
| 16:30:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |