| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-25 | |||
| 09:35:55 | sean-k-mooney | if i manage to get the numa stuff done early in the cycle ill move on to something else of if not ill move on to bug fixes and focus on get it done | |
| 09:36:10 | bauzas | anyway, it's a plan :) | |
| 09:36:16 | bauzas | like for me | |
| 09:36:24 | bauzas | (and like any other) | |
| 09:36:47 | bauzas | if anyone has other priority, then of course it can change | |
| 09:39:04 | bauzas | gibi: btw. do you think we could have a spec for the routed networks ? | |
| 09:39:50 | sean-k-mooney | ok i woke up around 4:45 this morning so im actully starting to get hungery. im going to go grab something for a deli and see if i can pick up a few bits and pieces. ill be back on irc in an hour or so | |
| 09:40:36 | sean-k-mooney | bauzas: we should have a spec for routed networks if for nothing else to define the scope. | |
| 09:40:36 | gibi | bauzas: we could have a doc that described the scenarios we need to support | |
| 09:40:44 | gibi | +1 ^^ | |
| 09:41:16 | bauzas | ok, I'll work on this then | |
| 09:41:27 | sean-k-mooney | i would like to see use eventually get to full network aware schduleing but if you want to target basic routed network support for this cycle that would be fine with me. just define the rest as out of scope | |
| 09:47:48 | bauzas | sean-k-mooney: the latter for Victoria honestly | |
| 09:49:43 | openstackgerrit | Merged openstack/os-vif master: Use unittest.mock instead of third party mock https://review.opendev.org/724965 | |
| 09:51:04 | gibi | bauzas: thanks. | |
| 09:51:32 | bauzas | gibi: I need to get more context (since I'm not a network expert) before providing the spec | |
| 09:51:51 | bauzas | but what I like is that I'll try to understand more about networks :) | |
| 09:53:00 | gibi | bauzas: the functional test in this patch has a lot of comments that I think helps understanding the goal of the feature https://review.opendev.org/#/c/656885/7/nova/tests/functional/test_servers.py | |
| 09:53:20 | bauzas | ahah thanks ! | |
| 09:53:45 | gibi | also there was a mail from me during U http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012846.html | |
| 09:54:17 | bauzas | excellent, ta :) | |
| 09:54:38 | gibi | :) | |
| 10:03:21 | kashyap | sean-k-mooney: When you get a min, I'm trying to see what are the "real Q35" failures, but I don't see any that is directly related to Q35 here - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_725/708701/8/check/devstack-platform-fedora-latest-virt-preview-q35/7259849/testr_results.html | |
| 10:03:31 | kashyap | I only see "Request timed out"s | |
| 10:34:33 | openstackgerrit | Wenping Song proposed openstack/nova master: WIP: Cyborg suspend/resume support https://review.opendev.org/729945 | |
| 10:36:51 | sean-k-mooney | kashyap in theory there should be no "real q35" failure but there might be some test configuration issues | |
| 10:37:32 | sean-k-mooney | kashyap: we have been allowing customer to use q35 in production since osp 13 if not eairler so i would hope they would have reported any real issues long before now | |
| 10:39:39 | kashyap | sean-k-mooney: For test configuration, right: wherever 'ide' is used, that needs to be replaced (or better: conditionalized). E.g. https://review.opendev.org/#/c/721555/ | |
| 10:39:53 | sean-k-mooney | ERROR cinder.volume.api [None req-a5e3e7e0-f8ce-4a58-bd74-53517edc5fd5 tempest-VolumeQuotasNegativeTestJSON-777903610 tempest-VolumeQuotasNegativeTestJSON-777903610] Quota exceeded for a769409cccff4ea48d0fe2efea551c8a, tried to extend volume by 1G, (1G of 1G already consumed).: cinder.exception.OverQuota: Quota exceeded for resources: ['gigabytes'] | |
| 10:40:01 | sean-k-mooney | i think the issue are mostly down to that | |
| 10:40:41 | sean-k-mooney | kashyap: i dont think ide will be used anyhwere in the test and that is not the reason for the failures as far as i can tell | |
| 10:41:16 | kashyap | sean-k-mooney: IDE _is_ used in some places, see the link I posted :-) | |
| 10:41:32 | kashyap | [But be back in a few...] | |
| 10:41:37 | sean-k-mooney | well most of the test failures are down to exceeding quota i belive | |
| 10:42:15 | sean-k-mooney | why did lee us ide for hw_rescue_bus? | |
| 10:42:43 | sean-k-mooney | that specificly was only added last cycle so that should have used ide form the start | |
| 10:48:42 | stephenfin | yo, bauzas - care to look at https://review.opendev.org/#/c/728968/ and surrounding patches? | |
| 11:04:25 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: remove get_quota_classes() driver method https://review.opendev.org/615615 | |
| 11:04:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: remove FakeContext from quota unit tests https://review.opendev.org/615617 | |
| 11:04:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: remove Context.quota_class https://review.opendev.org/615616 | |
| 11:04:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: clean up DbQuotaDriver unit tests https://review.opendev.org/615619 | |
| 11:04:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: remove _no_class tests https://review.opendev.org/615618 | |
| 11:04:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: quota: rename arguments to clarify they are limits https://review.opendev.org/615633 | |
| 11:27:23 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/train: Add admin doc information about image cache resource accounting https://review.opendev.org/730326 | |
| 11:54:06 | sean-k-mooney | stephenfin: i forget tox -e py3 worked | |
| 12:03:36 | openstack | bug 1880389 in OpenStack Compute (nova) "lost net connection when live migration" [Undecided,New] https://launchpad.net/bugs/1880389 | |
| 12:03:36 | gibi | sean-k-mooney: I know that you have a better view than me regarding network outage during live migration. there is a new bug 1880389 reported about it. Could you check it if you have time? | |
| 12:09:41 | sean-k-mooney | sure i can take a look shortly just grabing coffee but i have it open for when i get back | |
| 12:10:23 | sean-k-mooney | also without looking is sure bad things proably happen if there is a network outage while live migrating | |
| 12:16:18 | sean-k-mooney | ok so the issue is with the vm losing network connectivyt not a network partion at the host level causeing the migration to fail in weird ways which is less scary | |
| 12:18:19 | openstack | Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] | |
| 12:18:19 | sean-k-mooney | gibi: i need to dig deaper but its proably a duplicate of https://bugs.launchpad.net/neutron/+bug/1815989 which is cause by the root cause of https://bugs.launchpad.net/neutron/+bug/1734320 | |
| 12:18:20 | openstack | Launchpad bug 1734320 in OpenStack Compute (nova) "Eavesdropping private traffic" [Undecided,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 12:19:05 | bauzas | stephenfin: ack will look | |
| 12:19:19 | sean-k-mooney | gibi: https://review.opendev.org/#/c/724386/ is a potential workaround which some have said fixes the issue | |
| 12:19:48 | sean-k-mooney | gibi: this is how i actully wanted to fix it https://review.opendev.org/#/q/status:open++branch:master+topic:bug/1734320 | |
| 12:20:14 | sean-k-mooney | that said they are using linux bridge so it could be something else | |
| 12:22:00 | sean-k-mooney | "the ping was broken down. and the ping package from the physic switch was still set to the source." | |
| 12:22:28 | sean-k-mooney | so they were observing that pings are switched to the souce host which implies the mac learning frames were lost | |
| 12:22:55 | openstack | bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] https://launchpad.net/bugs/1815989 | |
| 12:22:55 | sean-k-mooney | which is bug: #1815989 | |
| 12:23:14 | sean-k-mooney | although in this case they are reportign the same behavior for linux bridge | |
| 12:23:53 | sean-k-mooney | ah last paragraph "And I found the time of 5 RARP from VM is before the time of the vlan sub interface insert to the linuxbridge bridge when live migration. maybe this will help us to make sure the problem I found." | |
| 12:24:19 | sean-k-mooney | so yes its the same underlying issue | |
| 12:25:19 | sean-k-mooney | the issue happens in the ovs case when libvirt recreated the port that os-vif has allready created resulting in a race with the neutron l2 agent | |
| 12:25:38 | sean-k-mooney | i did not think we had the same race for linux bridge but ill take a look | |
| 12:26:04 | gibi | sean-k-mooney: thanks | |
| 12:48:15 | sean-k-mooney | so we use interface type bridge for linux bridge which is not very surpising | |
| 12:48:18 | sean-k-mooney | <interface type='bridge'> | |
| 12:48:20 | sean-k-mooney | <source bridge='br1'/> | |
| 12:48:22 | sean-k-mooney | <target dev='vnet7'/> | |
| 12:48:24 | sean-k-mooney | <mac address="00:11:22:33:44:55"/> | |
| 12:48:26 | sean-k-mooney | </interface> | |
| 12:48:48 | sean-k-mooney | and se the target, mac and source to the values provided by neutron | |
| 12:49:41 | sean-k-mooney | before the vm is spawned on the destination host we invoke os-vif to ensure that the bridge exists | |
| 12:50:29 | sean-k-mooney | so when we define the libvirt domian there should be nothing for libvirt to do it should be alredy configured | |
| 12:51:06 | sean-k-mooney | this is all os-vif does https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_bridge.py#L97-L112 | |
| 12:52:45 | openstackgerrit | Merged openstack/os-vif master: Remove .testr.conf https://review.opendev.org/729156 | |
| 12:55:53 | sean-k-mooney | i should proably go clean up that code since nova net is now dead and we should not really be doing any ip filtering in os-vif even in neutron is not anymore since that was really for nova net | |
| 13:00:01 | gibi | sean-k-mooney: thanks for triageing that bug | |
| 13:05:28 | sean-k-mooney | im debating if im goint to try and repoduce it or not | |
| 13:06:18 | sean-k-mooney | i would need to check the libvirt sorce code but i dont think it should have the same race | |
| 13:06:32 | sean-k-mooney | it might but i did not think it should | |
| 13:32:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: virt: Add 'context', drop 'network_info' parameters for 'unrescue' https://review.opendev.org/730382 | |
| 13:32:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 13:32:17 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add emulated TPM support to Nova https://review.opendev.org/631363 | |
| 13:51:11 | openstackgerrit | sean mooney proposed openstack/os-vif master: update tox envs and support pdf docs https://review.opendev.org/728037 | |
| 14:02:11 | sean-k-mooney | stephenfin: hopefully you are more happy with ^ | |
| 14:02:49 | sean-k-mooney | but also that is more a merge when convenient patch. as long as we merge it before m3 (perferably m1) its fine | |
| 14:02:53 | sean-k-mooney | so no rush | |
| 14:03:22 | stephenfin | better but still not correct :) | |
| 14:04:26 | sean-k-mooney | hehe ok im goint to update my osc patch for evacuate ill take a look when that is done | |
| 14:09:02 | sean-k-mooney | just looked. ya ill do some of the chagne in a follow up and adress some in that patch. ill push both patches togeter this evening or tommorow | |
| 14:09:25 | sean-k-mooney | that patch is getting a bit long but the change you suggest make sense | |
| 14:18:52 | sean-k-mooney | 'Wait for instance evacuation to complete' sound less worng then 'Wait for evacuation to complete' right? | |
| 14:20:01 | stephenfin | The operation affects multiple instances, right? | |
| 14:20:10 | sean-k-mooney | no | |
| 14:20:13 | sean-k-mooney | only one instance | |
| 14:20:21 | stephenfin | then the first one is better, yes | |
| 14:20:28 | sean-k-mooney | im porting evacuate not host-evacuate | |