| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-25 | |||
| 08:04:20 | gibi | elod: fixed the commit message accordingly ^^ | |
| 08:06:26 | openstackgerrit | Merged openstack/os-vif master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728612 | |
| 08:52:50 | bauzas | good morning Nova | |
| 08:52:58 | bauzas | gibi: sorry, saw your ping, was on PTO | |
| 09:02:56 | elod | gibi: thx, I'll look into it soon | |
| 09:14:35 | openstackgerrit | Wenping Song proposed openstack/nova master: WIP: Cyborg suspend/resume support https://review.opendev.org/729945 | |
| 09:29:16 | gibi | bauzas: no problem | |
| 09:30:00 | bauzas | gibi: also, I'll try to help for the routed networks | |
| 09:31:31 | gibi | bauzas: there I have started something but I don't think I will have time to continue in V cycle | |
| 09:31:57 | bauzas | gibi: I provided an email for it | |
| 09:32:03 | gibi | ahh OK, I will read | |
| 09:32:14 | bauzas | gibi: sean-k-mooney will work on NUMA topology in Placement instead of me | |
| 09:32:21 | bauzas | so I could have time for this | |
| 09:33:40 | bauzas | gibi: also, I'll volunteer for working on https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/reshape-provider-tree.html#offline-upgrade-script | |
| 09:34:02 | gibi | bauzas: cool, thanks | |
| 09:35:13 | sean-k-mooney | well thats the plan at least. ill try to work on the numa stuff as the primary thing im aiming to get done this cycle and hopefully allowing bauzas to complete other work | |
| 09:35:54 | bauzas | sure | |
| 09:35:55 | sean-k-mooney | if i manage to get the numa stuff done early in the cycle ill move on to something else of if not ill move on to bug fixes and focus on get it done | |
| 09:36:10 | bauzas | anyway, it's a plan :) | |
| 09:36:16 | bauzas | like for me | |
| 09:36:24 | bauzas | (and like any other) | |
| 09:36:47 | bauzas | if anyone has other priority, then of course it can change | |
| 09:39:04 | bauzas | gibi: btw. do you think we could have a spec for the routed networks ? | |
| 09:39:50 | sean-k-mooney | ok i woke up around 4:45 this morning so im actully starting to get hungery. im going to go grab something for a deli and see if i can pick up a few bits and pieces. ill be back on irc in an hour or so | |
| 09:40:36 | gibi | bauzas: we could have a doc that described the scenarios we need to support | |
| 09:40:36 | sean-k-mooney | bauzas: we should have a spec for routed networks if for nothing else to define the scope. | |
| 09:40:44 | gibi | +1 ^^ | |
| 09:41:16 | bauzas | ok, I'll work on this then | |
| 09:41:27 | sean-k-mooney | i would like to see use eventually get to full network aware schduleing but if you want to target basic routed network support for this cycle that would be fine with me. just define the rest as out of scope | |
| 09:47:48 | bauzas | sean-k-mooney: the latter for Victoria honestly | |
| 09:49:43 | openstackgerrit | Merged openstack/os-vif master: Use unittest.mock instead of third party mock https://review.opendev.org/724965 | |
| 09:51:04 | gibi | bauzas: thanks. | |
| 09:51:32 | bauzas | gibi: I need to get more context (since I'm not a network expert) before providing the spec | |
| 09:51:51 | bauzas | but what I like is that I'll try to understand more about networks :) | |
| 09:53:00 | gibi | bauzas: the functional test in this patch has a lot of comments that I think helps understanding the goal of the feature https://review.opendev.org/#/c/656885/7/nova/tests/functional/test_servers.py | |
| 09:53:20 | bauzas | ahah thanks ! | |
| 09:53:45 | gibi | also there was a mail from me during U http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012846.html | |
| 09:54:17 | bauzas | excellent, ta :) | |
| 09:54:38 | gibi | :) | |
| 10:03:21 | kashyap | sean-k-mooney: When you get a min, I'm trying to see what are the "real Q35" failures, but I don't see any that is directly related to Q35 here - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_725/708701/8/check/devstack-platform-fedora-latest-virt-preview-q35/7259849/testr_results.html | |
| 10:03:31 | kashyap | I only see "Request timed out"s | |
| 10:34:33 | openstackgerrit | Wenping Song proposed openstack/nova master: WIP: Cyborg suspend/resume support https://review.opendev.org/729945 | |
| 10:36:51 | sean-k-mooney | kashyap in theory there should be no "real q35" failure but there might be some test configuration issues | |
| 10:37:32 | sean-k-mooney | kashyap: we have been allowing customer to use q35 in production since osp 13 if not eairler so i would hope they would have reported any real issues long before now | |
| 10:39:39 | kashyap | sean-k-mooney: For test configuration, right: wherever 'ide' is used, that needs to be replaced (or better: conditionalized). E.g. https://review.opendev.org/#/c/721555/ | |
| 10:39:53 | sean-k-mooney | ERROR cinder.volume.api [None req-a5e3e7e0-f8ce-4a58-bd74-53517edc5fd5 tempest-VolumeQuotasNegativeTestJSON-777903610 tempest-VolumeQuotasNegativeTestJSON-777903610] Quota exceeded for a769409cccff4ea48d0fe2efea551c8a, tried to extend volume by 1G, (1G of 1G already consumed).: cinder.exception.OverQuota: Quota exceeded for resources: ['gigabytes'] | |
| 10:40:01 | sean-k-mooney | i think the issue are mostly down to that | |
| 10:40:41 | sean-k-mooney | kashyap: i dont think ide will be used anyhwere in the test and that is not the reason for the failures as far as i can tell | |
| 10:41:16 | kashyap | sean-k-mooney: IDE _is_ used in some places, see the link I posted :-) | |
| 10:41:32 | kashyap | [But be back in a few...] | |
| 10:41:37 | sean-k-mooney | well most of the test failures are down to exceeding quota i belive | |
| 10:42:15 | sean-k-mooney | why did lee us ide for hw_rescue_bus? | |
| 10:42:43 | sean-k-mooney | that specificly was only added last cycle so that should have used ide form the start | |
| 10:48:42 | stephenfin | yo, bauzas - care to look at https://review.opendev.org/#/c/728968/ and surrounding patches? | |
| 11:04:25 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: remove get_quota_classes() driver method https://review.opendev.org/615615 | |
| 11:04:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: remove Context.quota_class https://review.opendev.org/615616 | |
| 11:04:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: remove FakeContext from quota unit tests https://review.opendev.org/615617 | |
| 11:04:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: remove _no_class tests https://review.opendev.org/615618 | |
| 11:04:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: clean up DbQuotaDriver unit tests https://review.opendev.org/615619 | |
| 11:04:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: rename arguments to clarify they are limits https://review.opendev.org/615633 | |
| 11:27:23 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/train: Add admin doc information about image cache resource accounting https://review.opendev.org/730326 | |
| 11:54:06 | sean-k-mooney | stephenfin: i forget tox -e py3 worked | |
| 12:03:36 | gibi | sean-k-mooney: I know that you have a better view than me regarding network outage during live migration. there is a new bug 1880389 reported about it. Could you check it if you have time? | |
| 12:03:36 | openstack | bug 1880389 in OpenStack Compute (nova) "lost net connection when live migration" [Undecided,New] https://launchpad.net/bugs/1880389 | |
| 12:09:41 | sean-k-mooney | sure i can take a look shortly just grabing coffee but i have it open for when i get back | |
| 12:10:23 | sean-k-mooney | also without looking is sure bad things proably happen if there is a network outage while live migrating | |
| 12:16:18 | sean-k-mooney | ok so the issue is with the vm losing network connectivyt not a network partion at the host level causeing the migration to fail in weird ways which is less scary | |
| 12:18:19 | sean-k-mooney | gibi: i need to dig deaper but its proably a duplicate of https://bugs.launchpad.net/neutron/+bug/1815989 which is cause by the root cause of https://bugs.launchpad.net/neutron/+bug/1734320 | |
| 12:18:19 | openstack | Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] | |
| 12:18:20 | openstack | Launchpad bug 1734320 in OpenStack Compute (nova) "Eavesdropping private traffic" [Undecided,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 12:19:05 | bauzas | stephenfin: ack will look | |
| 12:19:19 | sean-k-mooney | gibi: https://review.opendev.org/#/c/724386/ is a potential workaround which some have said fixes the issue | |
| 12:19:48 | sean-k-mooney | gibi: this is how i actully wanted to fix it https://review.opendev.org/#/q/status:open++branch:master+topic:bug/1734320 | |
| 12:20:14 | sean-k-mooney | that said they are using linux bridge so it could be something else | |
| 12:22:00 | sean-k-mooney | "the ping was broken down. and the ping package from the physic switch was still set to the source." | |
| 12:22:28 | sean-k-mooney | so they were observing that pings are switched to the souce host which implies the mac learning frames were lost | |
| 12:22:55 | sean-k-mooney | which is bug: #1815989 | |
| 12:22:55 | openstack | bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] https://launchpad.net/bugs/1815989 | |
| 12:23:14 | sean-k-mooney | although in this case they are reportign the same behavior for linux bridge | |
| 12:23:53 | sean-k-mooney | ah last paragraph "And I found the time of 5 RARP from VM is before the time of the vlan sub interface insert to the linuxbridge bridge when live migration. maybe this will help us to make sure the problem I found." | |
| 12:24:19 | sean-k-mooney | so yes its the same underlying issue | |
| 12:25:19 | sean-k-mooney | the issue happens in the ovs case when libvirt recreated the port that os-vif has allready created resulting in a race with the neutron l2 agent | |
| 12:25:38 | sean-k-mooney | i did not think we had the same race for linux bridge but ill take a look | |
| 12:26:04 | gibi | sean-k-mooney: thanks | |
| 12:48:15 | sean-k-mooney | so we use interface type bridge for linux bridge which is not very surpising | |
| 12:48:18 | sean-k-mooney | <interface type='bridge'> | |
| 12:48:20 | sean-k-mooney | <source bridge='br1'/> | |
| 12:48:22 | sean-k-mooney | <target dev='vnet7'/> | |
| 12:48:24 | sean-k-mooney | <mac address="00:11:22:33:44:55"/> | |
| 12:48:26 | sean-k-mooney | </interface> | |
| 12:48:48 | sean-k-mooney | and se the target, mac and source to the values provided by neutron | |
| 12:49:41 | sean-k-mooney | before the vm is spawned on the destination host we invoke os-vif to ensure that the bridge exists | |
| 12:50:29 | sean-k-mooney | so when we define the libvirt domian there should be nothing for libvirt to do it should be alredy configured | |
| 12:51:06 | sean-k-mooney | this is all os-vif does https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_bridge.py#L97-L112 | |
| 12:52:45 | openstackgerrit | Merged openstack/os-vif master: Remove .testr.conf https://review.opendev.org/729156 | |
| 12:55:53 | sean-k-mooney | i should proably go clean up that code since nova net is now dead and we should not really be doing any ip filtering in os-vif even in neutron is not anymore since that was really for nova net | |
| 13:00:01 | gibi | sean-k-mooney: thanks for triageing that bug | |
| 13:05:28 | sean-k-mooney | im debating if im goint to try and repoduce it or not | |
| 13:06:18 | sean-k-mooney | i would need to check the libvirt sorce code but i dont think it should have the same race | |
| 13:06:32 | sean-k-mooney | it might but i did not think it should | |