| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-01 | |||
| 12:16:18 | gibi | slaweq and lajoskatona are aware of the issue and working on a solution | |
| 12:17:03 | lyarwood | awesome thanks | |
| 12:17:27 | lyarwood | stephenfin: ^ the above should unblock the gate on master btw | |
| 12:23:02 | openstackgerrit | Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245 | |
| 12:23:13 | openstackgerrit | Rodrigo Barbieri proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245 | |
| 12:31:07 | openstackgerrit | Elod Illes proposed openstack/nova stable/ussuri: Allow tap interface with multiqueue https://review.opendev.org/752245 | |
| 13:09:08 | openstackgerrit | Hemanth Nakkina proposed openstack/nova master: Update pci stat pools based on PCI device changes https://review.opendev.org/749175 | |
| 13:14:04 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type https://review.opendev.org/708701 | |
| 13:14:16 | lyarwood | kashyap: ^ think I found a n-cpu bug with QEMU v5.0.0, looks like we will need to rewrite our detach logic | |
| 13:14:44 | lyarwood | I'll write up a bug once the vanilla fedora-latest-virt-preview job has also hit it without q35 | |
| 13:20:04 | sean-k-mooney | lyarwood: is this related the the qemu issue | |
| 13:20:08 | sean-k-mooney | on ubuntu | |
| 13:20:26 | sean-k-mooney | i.e. would the same rewite help | |
| 13:21:02 | lyarwood | sean-k-mooney: it's unrelated | |
| 13:22:50 | lyarwood | sean-k-mooney: https://github.com/qemu/qemu/blob/cbba3dc6ea3fc9aa66e9f9eb41051536e3ad7cd0/qdev-monitor.c#L886-L898 - n-cpu is tripping up over this check introduced in QEMU v5.0.0 as we fire off detach requests without waiting for libvirt to emit events. | |
| 13:23:20 | lyarwood | hmm unless | |
| 13:23:44 | sean-k-mooney | if (dev->pending_deleted_event) { | |
| 13:23:45 | lyarwood | are we hitting the bug this check was introduced to avoid? | |
| 13:23:47 | sean-k-mooney | that check | |
| 13:24:03 | sean-k-mooney | perhaps | |
| 13:24:22 | sean-k-mooney | https://github.com/qemu/qemu/commit/cce8944cc9efab47d4bf29cfffb3470371c3541b | |
| 13:24:26 | lyarwood | so the commit references PCI devices | |
| 13:24:37 | lyarwood | oh I wondeer | |
| 13:24:42 | lyarwood | interesting | |
| 13:25:01 | sean-k-mooney | On PCIe devices, this cancels the hot-unplug process. | |
| 13:25:13 | lyarwood | I don't recall seeing another request from libvirt to QEMU in the focal examples | |
| 13:25:15 | lyarwood | but maybe | |
| 13:25:31 | sean-k-mooney | if we retry it it would right | |
| 13:25:37 | sean-k-mooney | we just loop | |
| 13:25:48 | lyarwood | right but I can't recall seeing the requests overlap | |
| 13:25:59 | lyarwood | but the loop is pretty tight | |
| 13:26:12 | lyarwood | would explain why we only saw it on overloaded CI hosts | |
| 13:27:47 | kashyap | lyarwood: Hey, was out for a small walk; reading your comments in the change | |
| 13:28:37 | kashyap | That "unable to execute ... device_del' rings a bell | |
| 13:34:22 | kashyap | lyarwood: I think we _are_ hitting the bug the check was introduced to avoid, no? The QEMU commit skips repeated QMP 'device_del' calls | |
| 13:37:19 | lyarwood | kashyap: right and with >=v5.0.0 that raises an error, I was wondering above if prior to that on focal we are actually hitting the original issue and hot unplugs are being cancelled in QEMU | |
| 13:37:45 | kashyap | lyarwood: Ah, hence your comment on "why we only saw it on overloaded CI hosts" | |
| 13:38:01 | lyarwood | kashyap: either way the fix here seems to be to use libvirt events to wait until a detach either succeeds or fails before we retry | |
| 13:38:04 | sean-k-mooney | kashyap: right when we race with a slow detach due to load | |
| 13:38:45 | sean-k-mooney | lyarwood: ya so we should do that in either case and then we can use the ci to see if it fixes the ci issue with a DNM patch to swap the job to focal form bionic | |
| 13:39:16 | sean-k-mooney | lyarwood: we do sometime get random detach failure on bionic too | |
| 13:39:21 | sean-k-mooney | which could be caused by this | |
| 13:40:01 | sean-k-mooney | its much less common but i have seen many jobs fail for seminly unrelated volume detach issues | |
| 13:40:36 | kashyap | lyarwood: Yeah, it's a long-time TODO to wait for proper events, instead of polling :-( There should also be a documented TODO in driver.py/guest.py | |
| 14:23:01 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM libvirt: Remove loopingcall when detaching device https://review.opendev.org/755526 | |
| 14:49:28 | openstackgerrit | sean mooney proposed openstack/nova master: Fix unplugging VIF when migrate/resize VM https://review.opendev.org/751642 | |
| 14:49:37 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432 | |
| 14:56:50 | sean-k-mooney | those are just rebases by the way ^ i want to see if the tempest failure is real or not | |
| 14:57:21 | sean-k-mooney | i dont think they are but a rebase never hurts | |
| 15:45:14 | gibi | nova weekly meeting starts in 15 minutes in #openstack-meeting-3 | |
| 16:30:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 16:31:34 | gibi | lyarwood, stephenfin : could you hit it again? ^ | |
| 16:32:32 | lyarwood | gibi: will do once it passes | |
| 16:32:46 | gibi | OK :) | |
| 16:33:05 | gibi | thanks | |
| 16:33:26 | gibi | I will leave for today in 30 minutes so I guess I won't see it pass | |
| 16:35:40 | stephenfin | # | |
| 16:35:55 | stephenfin | Done. If it doesn't pass, it simply won't merge :) | |
| 16:38:09 | gibi | thanks | |
| 21:09:26 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Include only required fields in ironic node cache https://review.opendev.org/754444 | |
| 21:09:27 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/723055 | |
| 21:09:28 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Add resource_class to fields in ironic node cache https://review.opendev.org/755653 | |
| #openstack-nova - 2020-10-02 | |||
| 07:41:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 08:41:18 | openstackgerrit | Stephen Finucane proposed openstack/nova master: api-ref: Move 'os-agents' API to obsolete section https://review.opendev.org/755729 | |
| 13:44:15 | lyarwood | stephenfin: https://review.opendev.org/#/c/755498/ can you hit this agian please | |
| 13:46:13 | stephenfin | done | |
| 13:51:14 | lyarwood | ta | |
| 14:14:37 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Increase incremental and max sleep time during detach https://review.opendev.org/755799 | |
| 14:32:22 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Increase incremental and max sleep time during detach https://review.opendev.org/755799 | |
| 16:49:40 | openstackgerrit | Merged openstack/nova master: Temporarily disable parts of heal port allocation test https://review.opendev.org/755498 | |
| 17:01:58 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-evacuate: Disable libvirtd service and sockets during negative tests https://review.opendev.org/754684 | |
| 17:02:26 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985 | |
| 17:02:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986 | |
| 17:17:09 | openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Add tests for mixed CPU policy https://review.opendev.org/755852 | |
| 17:17:47 | stephenfin | sean-k-mooney, artom: Should be an easy fix if either of you are bored :-P I'm off for a run though so will take a look next week if not https://bugs.launchpad.net/nova/+bug/1898272 | |
| 17:17:47 | openstack | Launchpad bug 1898272 in OpenStack Compute (nova) ""mixed" policy calculations don't account for host cells with no free shared CPUs" [Medium,Confirmed] - Assigned to Stephen Finucane (stephenfinucane) | |
| 17:19:15 | artom | stephenfin, fuuuun | |
| 17:19:26 | stephenfin | you're telling me :( | |
| 17:54:19 | sean-k-mooney | hum ya that a bug | |
| 17:54:46 | sean-k-mooney | you could have moved it to triaged too | |
| 17:55:39 | sean-k-mooney | stephenfin: does mixed cpus support numa at all | |
| 17:56:45 | sean-k-mooney | we shoudl be able to test this with whitebox by the way | |
| 17:56:57 | sean-k-mooney | we can test it with a functional test too | |
| 17:58:01 | sean-k-mooney | stephenfin: thats failing because we are not tracking the numa affinity fo the shared cores yes | |
| 18:04:38 | sean-k-mooney | ah i see the fixt thats subtle | |
| 18:11:39 | sean-k-mooney | stephenfin: there is more legecy debt there | |
| 18:11:52 | sean-k-mooney | stephenfin: the cpu toplogy is not per numa node | |
| 18:11:54 | sean-k-mooney | its per vm | |
| 18:12:07 | sean-k-mooney | so we should not have a toplogy filed in the instance cell at all | |
| 18:13:13 | sean-k-mooney | so like the pagesize we should be pulling that field out of the instance numa cell object | |
| 18:14:39 | sean-k-mooney | both of these https://github.com/openstack/nova/blob/master/nova/objects/instance_numa.py#L79-L81 should be removed | |
| 18:15:08 | sean-k-mooney | and the cpu_thread_policy | |
| 18:16:11 | sean-k-mooney | they coudl be moved to the InstanceNUMATopology object i guess | |
| 18:16:31 | sean-k-mooney | although they are not really realted to numa but it would be more correct | |
| 18:16:55 | sean-k-mooney | stephenfin: can you add a todo to remove them | |
| 18:24:03 | sean-k-mooney | stephenfin: for what its worth i dont think we need the siblings field on the instance_cell object either | |
| 18:24:26 | sean-k-mooney | i think we only ever use teh siblings proerty on the host cell object not the property ihn the instnace cell | |
| 18:24:34 | sean-k-mooney | at least based on http://codesearch.openstack.org/?q=siblings&i=nope&files=&repos=openstack/nova | |
| 18:24:39 | sean-k-mooney | its not obvious if we do | |
| 18:29:07 | sean-k-mooney | the inital implemenation was quite bad at stashing things in the instnace cell object for convenince hten actully ensuign we only had one souce or truth | |