Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-08-05
10:18:30 openstackgerrit Tony Su proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522
10:18:31 openstackgerrit Tony Su proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460
10:19:00 openstackgerrit Lee Yarwood proposed openstack/nova master: zuul: Start to migrate nova-live-migration to zuulv3 https://review.opendev.org/711604
10:21:10 tony_su stephenfin: gibi: the last 3 provider-config-file patches are ready to go. change log: 1) all comments reflected in code 2) add three more test case to cover negative coe logic 3) unified coding style.
10:27:24 gibi tony_su: thanks. I added that to the queue for runway slot
10:31:07 tony_su gibi: thanks gibi. I will continue working on these patches in the next two weeks...
10:43:26 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP zuul: nova-multinode-evacuate https://review.opendev.org/744883
10:44:00 lyarwood hmm I thought the grenade issues had been resolved?
11:01:14 brinzhang stephenfin: hi, do I need fixed your point comments in https://review.opendev.org/#/c/742863/2/nova/compute/manager.py@2615
11:02:03 stephenfin brinzhang: Not at all. I was saying that your fix was correct and mine was not :)
11:02:04 brinzhang change the LOG.exception(exc.format_message()) to LOG.exception(exc), and use Exception instead of exception.NovaException
11:02:35 brinzhang an I will report a bug for this change
11:04:19 brinzhang stephenfin: oh sorry, your patch are working now, I will just fixed mine ^
11:07:31 openstackgerrit Brin Zhang proposed openstack/nova master: [Trivial] Remove wrong format_message() conversion https://review.opendev.org/744280
11:09:02 brinzhang stephenfin: updated, thanks
11:10:15 brinzhang stephenfin, sean-k-mooney, gibi: the cyborg evacuate support patch was updated, pls review again while you have time, thanks.
11:10:15 brinzhang https://review.opendev.org/#/c/715326/
11:10:47 brinzhang and I was rebased on the optimize patch, pls see https://review.opendev.org/#/c/726564/6
11:27:56 artom stephenfin, sorry :( it was a drive-by thing as well, I didn't actually look at anything else
11:28:17 artom Hence no vote
11:30:51 lyarwood artom: how dare you provide useful reviews
11:30:58 lyarwood I mean really
11:31:10 lyarwood you're making the rest of us look bad ^_^
11:31:25 lyarwood or maybe I'm doing that on my own
11:31:59 artom lyarwood, "useful" would be a stretch :P
11:33:09 gibi sean-k-mooney: thanks for the repeated test of the sriov attach. I responded in the review. Unfortunately I cannot reproduce your failures in my env.
11:33:17 artom (Yes, I'm knowingly leaving your last sentence unanswered >;)
11:33:27 gibi brinzhang: ack
11:38:51 openstackgerrit Merged openstack/python-novaclient master: Remove unused code https://review.opendev.org/744136
11:41:11 sean-k-mooney gibi: so in your env removal of direct physical or macvtap devices worked?
11:41:19 gibi sean-k-mooney: yes
11:41:37 gibi the only thing that I can see is the remaining MAC on the VF after macvtap removal
11:41:52 gibi the rest of your failure does not appear to me
11:42:08 sean-k-mooney i used a single vm to do all the testing
11:42:21 sean-k-mooney ill try it again with multiple vms and see it it makes any difference
11:42:53 sean-k-mooney what os and libvirt version are you using?
11:43:15 gibi ubuntu 18.04, libvirt 6.0.0 qemu 4.2
11:43:39 sean-k-mooney the libvirt behavior may have changed? i was using centos 8.1 maybe 8.2 i might be using older libvirt
11:43:49 sean-k-mooney i need to boot up the server to check
11:44:07 gibi the silent failure of macvtap and direct physical removel feels like a problem with findind the device that needs to be removed. I can try to add extra LOGs around that logic to trace the matching in your nev
11:44:11 gibi env
11:45:23 gibi two weeks ago I upgraded the libvirt from 4.0.0 to 6.0.0 and qemu from 2.11 to 4.2 due to a different failure in the "simple" direct case
11:46:38 sean-k-mooney if we need libvirt 6.0.0 we could not that as a min verion i guess.
11:46:42 sean-k-mooney but yes
11:46:53 sean-k-mooney both feel like we just did not find it in the xml and remove it
11:47:53 gibi regarding the leaking MAC addess after macvtap removal, who should do the removal of the MAC from the VF? is it libvirt?
11:49:12 sean-k-mooney gibi: libvirt should altrhough we also have code in nova to clear it for old libvirts
11:49:25 sean-k-mooney its also not reseting the programed vlan on the vf
11:49:33 sean-k-mooney but that is likely the same issue
11:50:14 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L774-L784
11:50:27 sean-k-mooney we seam not to be calling unplug
11:50:51 sean-k-mooney in the direct case it proably is not clearing the trused vf status
11:51:08 sean-k-mooney i didnt actuly test that
11:51:37 sean-k-mooney we should be calling https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L833 as part of detach
11:52:03 gibi sean-k-mooney: thanks I will trace this missing unplug in my env
11:52:06 sean-k-mooney we are apprently https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2201
11:53:14 gibi I even have a log in the dmesg about the MAC removal during macvtap
11:53:15 gibi [1911821.990047] ixgbe 0000:81:00.0: removing MAC on VF 2
11:53:30 gibi but right after it
11:53:31 gibi [1911822.241163] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 2
11:53:42 gibi it seems the vlan is set back to it
11:53:57 sean-k-mooney oh god damit....
11:54:18 sean-k-mooney the neutron sriov nic agent is proably racing with unplug
11:54:51 sean-k-mooney although i guess it could be libvirt?
11:55:14 sean-k-mooney we might want to do unplug after detach?
11:55:28 gibi good points
11:55:45 gibi I will gather logs from libvirt and the neutron agent to see if this is a race
11:57:02 sean-k-mooney [centos@sriov-1 ~]$ libvirtd --version
11:57:04 sean-k-mooney libvirtd (libvirt) 6.0.0
11:57:09 sean-k-mooney so same libvirt version
11:57:46 sean-k-mooney im using differnet nics then you you have nicantic 10G nics and im using 1G e1000 nics but that should not matter
11:58:28 sean-k-mooney @chat:button1[centos@sriov-1 ~]$ /usr/libexec/qemu-kvm --version
11:58:31 sean-k-mooney QEMU emulator version 4.2.0 (qemu-kvm-4.2.0-19.el8)
11:58:41 sean-k-mooney i think that is the same qemu too?
11:58:49 sean-k-mooney yep
11:58:58 sean-k-mooney so ya likely not related to the versions
11:59:16 gibi cool, on set of possible differences is ruled out
11:59:19 gibi one
12:09:28 sean-k-mooney its proably libvirt
12:10:26 stephenfin gibi, lyarwood: Could you folks stick https://review.opendev.org/#/c/744021/ on your respective review queues, please? Feel free to chuck something my way too
12:12:07 lyarwood stephenfin: I was looking at that yesterday
12:12:30 lyarwood stephenfin: I *think* I get it, I just wanted to grep around a little more before voting
12:13:14 stephenfin nw. Lots of context needed for it, unfortunately
12:14:24 lyarwood yup indeed, func tests helped however so thanks for that at least
12:24:03 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Drop support for Xen https://review.opendev.org/743231
12:24:04 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Remove 'hypervisor_version' from 'libvirt_info' https://review.opendev.org/744199
12:35:41 gibi sean-k-mooney: yeah, it is a race between libvirt detaching the device and nova unpluging the vif (and reseting the MAC), If I move the unplug after the detach in the nova code then the VF MAC and VLAN is reset properly after the macvtap port is detached
12:36:05 sean-k-mooney i have been wondering if we should do unplug twice
12:36:30 sean-k-mooney we generally wantto disconnect the device form the network backend before removing it form the vm
12:36:40 sean-k-mooney but just doing it once might be fine too
12:36:43 lyarwood stephenfin: LGTM btw
12:36:51 sean-k-mooney at the end
12:37:25 sean-k-mooney gibi: did you confirm it was libvirt by stoping the sriov nic agent?
12:38:22 sean-k-mooney gibi: or did you manage to find a log message
12:42:47 gibi sean-k-mooney: stopping the neutron nic agent did not solved the race so I assumed it is libvirt
12:42:59 sean-k-mooney yep makes sense to me
12:43:34 sean-k-mooney i saw in the libvirtd log that it does set the mac a number of time sbut i dont have the devstack logs to corralte the timestamps
12:43:42 alex_xu efried: do you know what is the usecase for this https://review.opendev.org/#/c/693414/3/specs/ussuri/approved/provider-config-file.rst@247
12:45:02 gibi sean-k-mooney: I will add a separate patch into the series that moves the unplug
12:45:15 efried alex_xu: Yes.

Earlier   Later