Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-12
07:09:17 openstackgerrit Michael Still proposed openstack/nova master: Create specialist set_macaddr_and_vlan helper. https://review.openstack.org/624227
07:09:17 openstackgerrit Michael Still proposed openstack/nova master: create_veth_pair is unused, remove it. https://review.openstack.org/624226
07:09:18 openstackgerrit Michael Still proposed openstack/nova master: Move set_vf_interface_vlan to be with its only caller. https://review.openstack.org/624229
07:09:18 openstackgerrit Michael Still proposed openstack/nova master: Move create_tap_dev into privsep. https://review.openstack.org/624228
07:09:19 openstackgerrit Michael Still proposed openstack/nova master: Move DHCP releasing to privsep. https://review.openstack.org/624230
09:10:01 openstackgerrit Zhenyu Zheng proposed openstack/nova master: WIP run metadata api per cell https://review.openstack.org/624612
09:19:39 yan0s hi all
09:20:04 yan0s why isn't there a "nova quota-class-list" command?
09:20:25 yan0s or "openstack quota list --class"
09:20:46 yan0s how can I get list of defined quota classes?
09:51:37 ondrejme yan0s: nova quota-defaults ?
10:40:37 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Add workaround to cleanup instance dir during evac with rbd https://review.openstack.org/618478
11:34:19 yan0s ondrejme: it is possible to set different quota per class
11:34:46 yan0s and it possible to see the quota set in this class if you know the class name
11:35:09 yan0s but it is not possible to get a list of available classes
11:36:07 yan0s the command you sent me returns the default quota values
11:42:46 openstackgerrit Lee Yarwood proposed openstack/nova stable/rocky: Only warn about not having computes nodes once in rpcapi https://review.openstack.org/624677
11:53:08 stephenfin lyarwood: I was wondering why nova-api (or was it the conductor?) was spewing all those logs ^
11:56:03 lyarwood stephenfin: that's dansmith's fix, I'm just pulling it back into stable/rocky
12:12:58 openstackgerrit Chris Dent proposed openstack/os-resource-classes master: Sync STANDARDS with placement/nova rc_fields https://review.openstack.org/624370
12:13:00 openstackgerrit Chris Dent proposed openstack/os-resource-classes master: Tune up documentation to be more useful https://review.openstack.org/624384
12:56:23 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Specifying az when restore shelved server https://review.openstack.org/624689
12:58:29 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Specifying az when restore shelved server https://review.openstack.org/624689
13:39:54 adrianc sean-k-mooney: Hi, regarding the issue you experienced yesterday with live-migration, ping me when you are here
13:46:23 sean-k-mooney adrianc: ping
13:46:27 sean-k-mooney :)
13:49:02 adrianc sean-k-mooney: so apparently there is some weird behavior with sriov macvtap when you tear down the VM the VF netdev remains up
13:49:33 adrianc so you have a scenario of duplicate macs in the network
13:50:22 adrianc doing ifconfig <VF-netdev> down on the source VF should fix the issue (at least on my setup it did)
13:52:38 adrianc sean-k-mooney: i had a discussion with moshele about it, apparently this is an old issue which branches off this mail: https://www.redhat.com/archives/libvir-list/2016-September/msg00075.html
13:53:13 sean-k-mooney ah ok
13:53:32 sean-k-mooney so i had considered creating an os-vif plugin for sriov this cycle.
13:53:38 adrianc sriov netdev mac address will be re-allocated only on driver rebind on the VF
13:54:02 sean-k-mooney we can do it in the nova tree but we could also jsut create the plugin and have it set the vf down
13:54:26 adrianc if you do passthrough then it will be properly cleaned, however on macvtap this is not the case
13:54:36 sean-k-mooney ya
13:54:51 adrianc i added a hook in my code to do: interface down in vif.py under unplug_hw_veb()
13:55:14 sean-k-mooney ok cool that should work for now.
13:56:02 sean-k-mooney i would like to move that into os-vif later i also want to move the trusted vf logic into os-vif which was why i was orignally gong to create the sriov plugin
13:56:25 sean-k-mooney adrianc: do you have a new patchset up with that change. ill redeploy and test
13:58:34 adrianc not yet, i need to deploy the change and make sure it actually does the job. it worked manually.
14:00:30 adrianc sean-k-mooney: it should not be tied to the neutron-sriov-live-migration work as it serves as a WA to an existing problem.
14:01:44 adrianc let me know if doing ifconfig <netdev> down on the source vf netdev solves the issue for you. (after migration)
14:01:49 sean-k-mooney that is true but you could make it the base patch of the series so its easy to test all of them
14:02:13 sean-k-mooney sure i need to restack but ill let you know in an hour or so
14:02:27 adrianc sean-k-mooney: you will still need some POC code in neutron for multiple port binding
14:03:42 sean-k-mooney ya i am useing that patch too
14:04:02 sean-k-mooney devstack can deploy with unmerged patches from gerrit
14:04:23 sean-k-mooney but it cant combine two sets fo unrelated patches
14:04:54 sean-k-mooney so to test at the moment i jsut need to point to the top patch on the chain in nova and the neutron patch
14:05:26 adrianc can be applied manually after deploy :)
14:06:12 sean-k-mooney yep ill just create a branch locally and cherrypick them
14:06:18 sean-k-mooney its fine
14:28:47 mriedem melwitt: a change of mine hit the same vif plug timeout thing on that test with 7 ports http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?level=TRACE#_Dec_12_00_01_13_179474
14:29:36 mriedem bauzas: can you look at this? https://review.openstack.org/#/c/615347/ - it drops old confusing compat which is actually contributing to lots of misleading warnings in the n-api logs and gate race failures
14:29:48 bauzas mriedem: sure
14:29:52 sean-k-mooney mriedem ya i have seen that a few times
14:33:43 mriedem it's always the first port that we're missing
14:33:54 mriedem we get 6 of the 7 network-vif-plugged events
14:33:59 sean-k-mooney mriedem: one edgecase that i am awre of is in some configuration we start waithing for the network events too late
14:34:26 sean-k-mooney we start waiting for the event just before we call plug on the interface
14:34:44 sean-k-mooney for some neutron backend they send that event when we call bind
14:34:58 sean-k-mooney i assume this is ml2/ovs however correct?
14:35:15 sean-k-mooney that should not hit the race but i wonde if ther i a missed event in the logs
14:35:24 mriedem yes ovs
14:37:32 sean-k-mooney mriedem: http://logs.openstack.org/46/623246/3/check/nova-multiattach/efa830b/logs/screen-n-cpu.txt.gz?#_Dec_12_00_01_07_481084
14:37:39 mriedem 019494),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap472ab433-bc')
14:37:39 mriedem Dec 11 23:56:11.855811 ubuntu-xenial-inap-mtl01-0001136812 nova-compute[29399]: INFO os_vif [None req-f43f3c35-dc0b-4aa4-bfd0-4046045b315e tempest-TaggedBootDevicesTest-250467933 tempest-TaggedBootDevicesTest-250467933] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:fb:dd:9f,bridge_name='br-int',has_traffic_filtering=True,id=472ab433-bc18-41b5-85ae-009611117b70,network=Network(4988ef3d-6a8f-477f-b0a2-29
14:37:39 mriedem we plug it here
14:38:20 mriedem hmm wtf
14:38:39 mriedem Dec 11 23:55:46.981194 ubuntu-xenial-inap-mtl01-0001136812 neutron-server[20899]: DEBUG neutron.notifiers.nova [-] Sending events: [{'tag': u'472ab433-bc18-41b5-85ae-009611117b70', 'name': 'network-changed', 'server_uuid': u'18e5987e-a535-4044-84aa-dd8f023cedcc'}] {{(pid=21001) send_events /opt/stack/new/neutron/neutron/notifiers/nova.py:245}}
14:38:39 mriedem i also see this right before that port is plugged
14:39:31 mriedem sean-k-mooney: that's a different port
14:39:35 mriedem the unplugged event
14:39:55 sean-k-mooney mriedem: yes i just noticed that
14:42:23 mriedem asking in -neutron
14:53:58 frickler nova stable cores: https://review.openstack.org/619254 is waiting for a second +2, if it gets that, I can start nagging mriedem about stable releases instead ;-)
15:07:52 jangutter mriedem: yonks ago I saw something similar that happened if the neutron-ovsdb linkage got busted.
15:09:31 mriedem Dec 11 23:56:12.725010 ubuntu-xenial-inap-mtl01-0001136812 neutron-openvswitch-agent[21879]: INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-cc26f856-2521-411e-ae10-34ad96b7c665 None None] Port 472ab433-bc18-41b5-85ae-009611117b70 was not found on the integration bridge and will therefore not be processed
15:09:31 mriedem jangutter: i did see this in the neutron agent logs
15:10:35 jangutter mriedem: there's some extra metadata added by libvirt that links the port uuid. years ago I saw neutron triggering off of that.
15:11:13 adrianc sean-k-mooney: OK, so apparently on my setup i need to unbind/bind the VF, taking down the interface is not sufficient. this may be NIC vendor specific, let me know if interface down works for you OR you need to rebind the VF to flush out the MAC from the source node
15:21:36 sean-k-mooney adrianc: rebinding the vf will be a problem if needed
15:22:03 sean-k-mooney nova doen not always have permissions to do that.
15:23:01 mriedem hello people - easy fix to get the postgresql job working again https://review.openstack.org/#/c/619061/
15:23:22 mriedem cdent: ^ since you seem to care about pg
15:23:26 mriedem slaweq: you too ^
15:23:41 adrianc sean-k-mooney: the fix should be in libvirt imo, once the guest is not on the machine, it should properly clean-up. however for the direct SRIOV you will not have this problem
15:24:07 cdent mriedem: it's less care about pg, more care about not-just-mysql
15:24:10 cdent but yeah, looking
15:24:17 sean-k-mooney adrianc: well libvirt or qemu but yes one of the two shoudl notify the pf to reset the vf
15:25:05 sean-k-mooney adrianc: yes direct mode should work as the vf will be reset when it detached form the guest kernel and released to the host kernel
15:25:09 jangutter sean-k-mooney, adrianc: I think qemu does a function-level-reset on the VF when doing iommu passthrough.
15:25:27 sean-k-mooney jangutter: on a live migration
15:25:34 jangutter sean-k-mooney, adrianc: and libvirt does driver binding and unbinding.
15:25:57 sean-k-mooney jangutter: in the macvtap case you do not bind the diriver to vfio-pci
15:26:09 sean-k-mooney jangutter: you leave it bound to the host kernel driver at all times
15:26:09 slaweq mriedem: it is this fix which will finally fix neutron pgsql periodic job, right?
15:26:23 jangutter sean-k-mooney: yep, but permission-wise, libvirt could bind/unbind it as a workaround.
15:26:46 jangutter sean-k-mooney: and generally, you _don't_ want to do a FLR on something bound to a driver.

Earlier   Later