| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-01 | |||
| 08:28:02 | gibi | lyarwood: thanks for the ping | |
| 08:28:10 | gibi | I haven't seen it yet | |
| 08:29:50 | lyarwood | gibi: np, can't see anything obvious in nova master that could be causing it, I assume something in Neutron has changed? | |
| 08:32:22 | gibi | lyarwood: I will ping lajoskatona on Neutron side, he did some recent changes around qos | |
| 08:33:28 | lyarwood | awesome thanks | |
| 08:35:12 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/victoria: [stable-only] nova-live-migration: Disable libvirtd during negative tests https://review.opendev.org/754688 | |
| 09:10:07 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701 | |
| 10:17:09 | stephenfin | gibi: Think you could spin through https://review.opendev.org/#/q/topic:remove-xenapi+status:open before the week is out? | |
| 10:24:53 | gibi | stephenfin: I'm not comfortable making big changes to master during release time, can we take this after the release? | |
| 10:25:06 | stephenfin | oh, of course | |
| 10:30:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Support interface attach with qos ports https://review.opendev.org/755477 | |
| 11:13:34 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Add modernize-os-hypervisors-api spec https://review.opendev.org/755109 | |
| 11:18:10 | gibi | lyarwood: it seem neutron-lib needs a new release to contain this: https://review.opendev.org/#/c/753230/ | |
| 11:18:56 | lyarwood | gibi: ack thanks for looking into that! | |
| 11:19:18 | lyarwood | gibi: ./me wonders if neutron have nova-next in their check/gate? | |
| 11:19:28 | gibi | they don't I guess | |
| 11:19:44 | lyarwood | yeah might be something to suggest given the QoS coverage it has | |
| 11:22:02 | gibi | good point | |
| 11:22:35 | gibi | to be fair the normal qos coverage is in tempest. the nova-next only has the coverage for the qos part of the nova-manage placement heal_allocation CLI | |
| 11:23:20 | lyarwood | ah so we could add something to tempest to also cover this? | |
| 11:26:45 | gibi | I think it was added to the nova-next job as it has to create an invalid situation first then heal it | |
| 11:27:02 | gibi | in a real word missing port allocation only happen during upgrade from older openstack version | |
| 11:27:13 | gibi | we cannot really simulate that | |
| 11:27:26 | gibi | so we instead deleteing the allocation manually | |
| 11:27:29 | gibi | then try to heal it | |
| 11:29:20 | lyarwood | ack | |
| 11:30:03 | lyarwood | still would be nice if a neutron job could execute the test somewhere in their gate | |
| 11:30:17 | lyarwood | if we could break the test out into a seperate script somewhere | |
| 11:30:37 | gibi | yeah, I agree with that | |
| 11:37:02 | gibi | lyarwood: I guess the whole nova gate is broken now due to this | |
| 11:43:14 | gibi | lyarwood: I'll try to disable this part of the nova-next job as it will take time to get a new neutron-lib release | |
| 11:55:29 | gibi | lyarwood: also stable/victoria could have the same issue if https://review.opendev.org/#/c/755180 merges | |
| 12:05:46 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 12:05:52 | gibi | lyarwood: ^^ | |
| 12:07:49 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 12:15:39 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 12:15:52 | lyarwood | gibi: ack thanks | |
| 12:16:18 | gibi | slaweq and lajoskatona are aware of the issue and working on a solution | |
| 12:17:03 | lyarwood | awesome thanks | |
| 12:17:27 | lyarwood | stephenfin: ^ the above should unblock the gate on master btw | |
| 12:23:02 | openstackgerrit | Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245 | |
| 12:23:13 | openstackgerrit | Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245 | |
| 12:31:07 | openstackgerrit | Elod Illes proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245 | |
| 13:09:08 | openstackgerrit | Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175 | |
| 13:14:04 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701 | |
| 13:14:16 | lyarwood | kashyap: ^ think I found a n-cpu bug with QEMU v5.0.0, looks like we will need to rewrite our detach logic | |
| 13:14:44 | lyarwood | I'll write up a bug once the vanilla fedora-latest-virt-preview job has also hit it without q35 | |
| 13:20:04 | sean-k-mooney | lyarwood: is this related the the qemu issue | |
| 13:20:08 | sean-k-mooney | on ubuntu | |
| 13:20:26 | sean-k-mooney | i.e. would the same rewite help | |
| 13:21:02 | lyarwood | sean-k-mooney: it's unrelated | |
| 13:22:50 | lyarwood | sean-k-mooney: https://github.com/qemu/qemu/blob/cbba3dc6ea3fc9aa66e9f9eb41051536e3ad7cd0/qdev-monitor.c#L886-L898 - n-cpu is tripping up over this check introduced in QEMU v5.0.0 as we fire off detach requests without waiting for libvirt to emit events. | |
| 13:23:20 | lyarwood | hmm unless | |
| 13:23:44 | sean-k-mooney | if (dev->pending_deleted_event) { | |
| 13:23:45 | lyarwood | are we hitting the bug this check was introduced to avoid? | |
| 13:23:47 | sean-k-mooney | that check | |
| 13:24:03 | sean-k-mooney | perhaps | |
| 13:24:22 | sean-k-mooney | https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b | |
| 13:24:26 | lyarwood | so the commit references PCI devices | |
| 13:24:37 | lyarwood | oh I wondeer | |
| 13:24:42 | lyarwood | interesting | |
| 13:25:01 | sean-k-mooney | On PCIe devices, this cancels the hot-unplug process. | |
| 13:25:13 | lyarwood | I don't recall seeing another request from libvirt to QEMU in the focal examples | |
| 13:25:15 | lyarwood | but maybe | |
| 13:25:31 | sean-k-mooney | if we retry it it would right | |
| 13:25:37 | sean-k-mooney | we just loop | |
| 13:25:48 | lyarwood | right but I can't recall seeing the requests overlap | |
| 13:25:59 | lyarwood | but the loop is pretty tight | |
| 13:26:12 | lyarwood | would explain why we only saw it on overloaded CI hosts | |
| 13:27:47 | kashyap | lyarwood: Hey, was out for a small walk; reading your comments in the change | |
| 13:28:37 | kashyap | That "unable to execute ... device_del' rings a bell | |
| 13:34:22 | kashyap | lyarwood: I think we _are_ hitting the bug the check was introduced to avoid, no? The QEMU commit skips repeated QMP 'device_del' calls | |
| 13:37:19 | lyarwood | kashyap: right and with >=v5.0.0 that raises an error, I was wondering above if prior to that on focal we are actually hitting the original issue and hot unplugs are being cancelled in QEMU | |
| 13:37:45 | kashyap | lyarwood: Ah, hence your comment on "why we only saw it on overloaded CI hosts" | |
| 13:38:01 | lyarwood | kashyap: either way the fix here seems to be to use libvirt events to wait until a detach either succeeds or fails before we retry | |
| 13:38:04 | sean-k-mooney | kashyap: right when we race with a slow detach due to load | |
| 13:38:45 | sean-k-mooney | lyarwood: ya so we should do that in either case and then we can use the ci to see if it fixes the ci issue with a DNM patch to swap the job to focal form bionic | |
| 13:39:16 | sean-k-mooney | lyarwood: we do sometime get random detach failure on bionic too | |
| 13:39:21 | sean-k-mooney | which could be caused by this | |
| 13:40:01 | sean-k-mooney | its much less common but i have seen many jobs fail for seminly unrelated volume detach issues | |
| 13:40:36 | kashyap | lyarwood: Yeah, it's a long-time TODO to wait for proper events, instead of polling :-( There should also be a documented TODO in driver.py/guest.py | |
| 14:23:01 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM libvirt: Remove loopingcall when detaching device https://review.opendev.org/755526 | |
| 14:49:28 | openstackgerrit | sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642 | |
| 14:49:37 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432 | |
| 14:56:50 | sean-k-mooney | those are just rebases by the way ^ i want to see if the tempest failure is real or not | |
| 14:57:21 | sean-k-mooney | i dont think they are but a rebase never hurts | |
| 15:45:14 | gibi | nova weekly meeting starts in 15 minutes in #openstack-meeting-3 | |
| 16:30:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 16:31:34 | gibi | lyarwood, stephenfin : could you hit it again? ^ | |
| 16:32:32 | lyarwood | gibi: will do once it passes | |
| 16:32:46 | gibi | OK :) | |
| 16:33:05 | gibi | thanks | |
| 16:33:26 | gibi | I will leave for today in 30 minutes so I guess I won't see it pass | |
| 16:35:40 | stephenfin | # | |
| 16:35:55 | stephenfin | Done. If it doesn't pass, it simply won't merge :) | |
| 16:38:09 | gibi | thanks | |
| 21:09:26 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444 | |
| 21:09:27 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055 | |
| 21:09:28 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Add resource_class to fields in ironic node cache https://review.opendev.org/755653 | |
| #openstack-nova - 2020-10-02 | |||
| 07:41:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |