Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-08-05
11:30:51 lyarwood artom: how dare you provide useful reviews
11:30:58 lyarwood I mean really
11:31:10 lyarwood you're making the rest of us look bad ^_^
11:31:25 lyarwood or maybe I'm doing that on my own
11:31:59 artom lyarwood, "useful" would be a stretch :P
11:33:09 gibi sean-k-mooney: thanks for the repeated test of the sriov attach. I responded in the review. Unfortunately I cannot reproduce your failures in my env.
11:33:17 artom (Yes, I'm knowingly leaving your last sentence unanswered >;)
11:33:27 gibi brinzhang: ack
11:38:51 openstackgerrit Merged openstack/python-novaclient master: Remove unused code https://review.opendev.org/744136
11:41:11 sean-k-mooney gibi: so in your env removal of direct physical or macvtap devices worked?
11:41:19 gibi sean-k-mooney: yes
11:41:37 gibi the only thing that I can see is the remaining MAC on the VF after macvtap removal
11:41:52 gibi the rest of your failure does not appear to me
11:42:08 sean-k-mooney i used a single vm to do all the testing
11:42:21 sean-k-mooney ill try it again with multiple vms and see it it makes any difference
11:42:53 sean-k-mooney what os and libvirt version are you using?
11:43:15 gibi ubuntu 18.04, libvirt 6.0.0 qemu 4.2
11:43:39 sean-k-mooney the libvirt behavior may have changed? i was using centos 8.1 maybe 8.2 i might be using older libvirt
11:43:49 sean-k-mooney i need to boot up the server to check
11:44:07 gibi the silent failure of macvtap and direct physical removel feels like a problem with findind the device that needs to be removed. I can try to add extra LOGs around that logic to trace the matching in your nev
11:44:11 gibi env
11:45:23 gibi two weeks ago I upgraded the libvirt from 4.0.0 to 6.0.0 and qemu from 2.11 to 4.2 due to a different failure in the "simple" direct case
11:46:38 sean-k-mooney if we need libvirt 6.0.0 we could not that as a min verion i guess.
11:46:42 sean-k-mooney but yes
11:46:53 sean-k-mooney both feel like we just did not find it in the xml and remove it
11:47:53 gibi regarding the leaking MAC addess after macvtap removal, who should do the removal of the MAC from the VF? is it libvirt?
11:49:12 sean-k-mooney gibi: libvirt should altrhough we also have code in nova to clear it for old libvirts
11:49:25 sean-k-mooney its also not reseting the programed vlan on the vf
11:49:33 sean-k-mooney but that is likely the same issue
11:50:14 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L774-L784
11:50:27 sean-k-mooney we seam not to be calling unplug
11:50:51 sean-k-mooney in the direct case it proably is not clearing the trused vf status
11:51:08 sean-k-mooney i didnt actuly test that
11:51:37 sean-k-mooney we should be calling https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L833 as part of detach
11:52:03 gibi sean-k-mooney: thanks I will trace this missing unplug in my env
11:52:06 sean-k-mooney we are apprently https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2201
11:53:14 gibi I even have a log in the dmesg about the MAC removal during macvtap
11:53:15 gibi [1911821.990047] ixgbe 0000:81:00.0: removing MAC on VF 2
11:53:30 gibi but right after it
11:53:31 gibi [1911822.241163] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 2
11:53:42 gibi it seems the vlan is set back to it
11:53:57 sean-k-mooney oh god damit....
11:54:18 sean-k-mooney the neutron sriov nic agent is proably racing with unplug
11:54:51 sean-k-mooney although i guess it could be libvirt?
11:55:14 sean-k-mooney we might want to do unplug after detach?
11:55:28 gibi good points
11:55:45 gibi I will gather logs from libvirt and the neutron agent to see if this is a race
11:57:02 sean-k-mooney [centos@sriov-1 ~]$ libvirtd --version
11:57:04 sean-k-mooney libvirtd (libvirt) 6.0.0
11:57:09 sean-k-mooney so same libvirt version
11:57:46 sean-k-mooney im using differnet nics then you you have nicantic 10G nics and im using 1G e1000 nics but that should not matter
11:58:28 sean-k-mooney @chat:button1[centos@sriov-1 ~]$ /usr/libexec/qemu-kvm --version
11:58:31 sean-k-mooney QEMU emulator version 4.2.0 (qemu-kvm-4.2.0-19.el8)
11:58:41 sean-k-mooney i think that is the same qemu too?
11:58:49 sean-k-mooney yep
11:58:58 sean-k-mooney so ya likely not related to the versions
11:59:16 gibi cool, on set of possible differences is ruled out
11:59:19 gibi one
12:09:28 sean-k-mooney its proably libvirt
12:10:26 stephenfin gibi, lyarwood: Could you folks stick https://review.opendev.org/#/c/744021/ on your respective review queues, please? Feel free to chuck something my way too
12:12:07 lyarwood stephenfin: I was looking at that yesterday
12:12:30 lyarwood stephenfin: I *think* I get it, I just wanted to grep around a little more before voting
12:13:14 stephenfin nw. Lots of context needed for it, unfortunately
12:14:24 lyarwood yup indeed, func tests helped however so thanks for that at least
12:24:03 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Drop support for Xen https://review.opendev.org/743231
12:24:04 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Remove 'hypervisor_version' from 'libvirt_info' https://review.opendev.org/744199
12:35:41 gibi sean-k-mooney: yeah, it is a race between libvirt detaching the device and nova unpluging the vif (and reseting the MAC), If I move the unplug after the detach in the nova code then the VF MAC and VLAN is reset properly after the macvtap port is detached
12:36:05 sean-k-mooney i have been wondering if we should do unplug twice
12:36:30 sean-k-mooney we generally wantto disconnect the device form the network backend before removing it form the vm
12:36:40 sean-k-mooney but just doing it once might be fine too
12:36:43 lyarwood stephenfin: LGTM btw
12:36:51 sean-k-mooney at the end
12:37:25 sean-k-mooney gibi: did you confirm it was libvirt by stoping the sriov nic agent?
12:38:22 sean-k-mooney gibi: or did you manage to find a log message
12:42:47 gibi sean-k-mooney: stopping the neutron nic agent did not solved the race so I assumed it is libvirt
12:42:59 sean-k-mooney yep makes sense to me
12:43:34 sean-k-mooney i saw in the libvirtd log that it does set the mac a number of time sbut i dont have the devstack logs to corralte the timestamps
12:43:42 alex_xu efried: do you know what is the usecase for this https://review.opendev.org/#/c/693414/3/specs/ussuri/approved/provider-config-file.rst@247
12:45:02 gibi sean-k-mooney: I will add a separate patch into the series that moves the unplug
12:45:15 efried The idea there was that you could define a "default" rule to apply to all your compute nodes, but then override it for specific ones.
12:45:15 efried alex_xu: Yes.
12:47:00 efried Used for ironic, but also in cases where you want to have your rules centralized and ansibled out to the hosts.
12:47:13 alex_xu efried: I see, thanks
12:48:37 alex_xu efried: is there any reason we should ignore the addtional inventories and traits when conflict the virt driver managed ones, instead of error out the conflict?
12:49:47 efried We debated this at design time. I can tell you for sure the answer to your question is "yes". But I can't remember exactly why :P
12:50:41 alex_xu efried: ok, so the result is ignore, not the error out, right? I saw the code is error out, not ignore.
12:51:07 efried Oh, whatever the design says is what we decided on.
12:51:28 alex_xu efried: ok, thanks :) I'm not going dump our the history
12:51:51 efried For this issue, I'm reasonably sure whatever is in the design is going to be there because it's what we decided on, not because we accidentally missed it.
12:52:41 alex_xu efried: ok, cool
12:53:44 efried https://specs.openstack.org/openstack/nova-specs/specs/victoria/approved/provider-config-file.html#provider-config-consumption-from-nova says "ignore" under "Provider Tree Merging".
12:55:42 alex_xu efried: yes, that is what I read also
12:56:00 alex_xu I can't thinking of a reason the different between ignore and error out also
12:56:19 efried alex_xu: here's an explanation of that other thing https://review.opendev.org/#/c/693414/3/specs/ussuri/approved/provider-config-file.rst@137
12:58:07 alex_xu nice
13:00:18 efried The conflicts in question should error on startup, but be ignored thereafter.
13:00:18 efried I remember now:
13:00:18 efried alex_xu: https://review.opendev.org/#/c/612497/12/specs/train/approved/provider-config-file.rst@204
13:03:08 alex_xu efried: for the case, there are virt driver managed inventory or trait show up later?
13:26:27 alex_xu efried: virt driver's update_provider_tree always overwrited provider tree's inventory. so the conflict will be found at startup, and there won't be any conflict after startup. so the code feel like right

Earlier   Later