| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-03 | |||
| 15:17:08 | bauzas | sean-k-mooney: I feel so | |
| 15:17:34 | sean-k-mooney | bauzas: that explains a lot .... | |
| 15:18:14 | bauzas | very simple | |
| 15:18:36 | stephenfin | bauzas: I studied Haskell in NZ and adored it (helped me grasp recursion at long last). Alas, it's not something I've had the chance to work with since | |
| 15:18:41 | stephenfin | Prolog, on the other hand. Ick | |
| 15:19:11 | sean-k-mooney | stephenfin: python has really good fuctional programing supprot built in by they way. | |
| 15:19:15 | bauzas | sean-k-mooney: since we're adding a new property, that's not chaning what we pass over RPC | |
| 15:19:25 | bauzas | sean-k-mooney: my only concern is on the left side | |
| 15:19:29 | bauzas | ie.what we remove | |
| 15:20:07 | stephenfin | kashyap: Here's the output of 'virsh dumpxml' http://paste.openstack.org/show/729345/ | |
| 15:20:29 | bauzas | anyway, time for wrapping off | |
| 15:20:58 | kashyap | stephenfin: Nice, /me clicks | |
| 15:21:06 | stephenfin | kashyap: and here are the commands I ran to get there http://paste.openstack.org/show/729348/ | |
| 15:21:18 | bauzas | today was school's back for my daughters, so I want to take time to discuss with them | |
| 15:21:27 | bauzas | so, calling it a day \o | |
| 15:21:40 | sean-k-mooney | bauzas: as you should. o/ | |
| 15:21:43 | stephenfin | kashyap: That's using commit 90cdf807503f94bfbd3ae9a62a9bfe466a18bcad (so about two month old cold) | |
| 15:21:47 | stephenfin | bauzas: Enjoy o/ | |
| 15:21:48 | kashyap | stephenfin: Nice. Can I ask one more, since you have the env? Also related to PCIe | |
| 15:21:56 | stephenfin | kashyap: shoot | |
| 15:22:45 | kashyap | stephenfin: Set the 'num_pcie_ports=2' in nova.conf, restart the Compute service, and again boot an instance afresh, and post the guest XML | |
| 15:22:56 | kashyap | Thanks a for the quick response :-) | |
| 15:23:24 | stephenfin | kashyap: num_pcie_ports ? That's not a nova.conf option, is it? | |
| 15:23:25 | sean-k-mooney | kashyap: wait why is that a nova conf setting and not a flavor extra spec | |
| 15:23:46 | stephenfin | Oh, what do you know, so it is | |
| 15:23:57 | kashyap | sean-k-mooney: Hehe, I haven't checked if there's a flavor extra spec for it or not. | |
| 15:24:00 | sean-k-mooney | stephenfin: really that sucks. | |
| 15:24:14 | sean-k-mooney | stephenfin: that is going to make live migration even more of a pain | |
| 15:24:45 | stephenfin | sean-k-mooney: I think it's like TX/RX queue size. The expectation would be that every node would have the same value (managed by Ansible or whatever other tool you're using) | |
| 15:25:03 | stephenfin | kashyap: With or without the q35 flag? | |
| 15:25:05 | sean-k-mooney | stephenfin: i think setting this suff in the nova conf is a serios antipatern | |
| 15:25:14 | kashyap | stephenfin: With, please. | |
| 15:25:22 | kashyap | stephenfin: Also - PCIe goodies are 'q35' _only_, BTW. | |
| 15:25:40 | sean-k-mooney | stephenfin: that kind of assumtion make upgrading a pain as once set you can never safly change it | |
| 15:26:22 | kashyap | sean-k-mooney: Before we forget, perhaps we should file a bug to revisit the PCIe root ports thingie? | |
| 15:26:51 | sean-k-mooney | which one. the auto allocation of a pcie route per numa node? | |
| 15:28:35 | stephenfin | kashyap: http://paste.openstack.org/show/729349/ | |
| 15:28:52 | stephenfin | sean-k-mooney: Fortunately for you, I have zilch on my plate rn. Send em my way | |
| 15:29:33 | sean-k-mooney | ill create an ether pad with them the issue is i have not got around to filing/writing them all. | |
| 15:31:44 | sean-k-mooney | stephenfin: im not sure that is correct for num_pcie_ports=2 | |
| 15:33:04 | sean-k-mooney | stephenfin: your machine type is pc-i440fx-bionic so you are not enableing q35 chipset so no pcie at all | |
| 15:33:57 | stephenfin | I don't think so either. I was expecting to see a <pci model='pcie-root-port'> element | |
| 15:34:45 | stephenfin | Sorry - <controller type='pci' model='pcie-root-port'> | |
| 15:35:01 | stephenfin | So why didn't that property get applied... | |
| 15:35:35 | sean-k-mooney | well currently everything in that xml is connect to the pci bus not pcie. also as i sad you are not using the q35 chipset so no pcie is available | |
| 15:35:52 | stephenfin | Oh yeah, I meant the 'hw:machine_type' property | |
| 15:36:17 | sean-k-mooney | i think its only read for arm... | |
| 15:36:50 | stephenfin | waaaait | |
| 15:36:56 | sean-k-mooney | it should be read on all archs but just going on the glance metadata docs i think its ignored on x86. i would prefer if it was not | |
| 15:37:09 | stephenfin | I misread kashyap's comments. You can't do this with extra specs | |
| 15:41:10 | stephenfin | kashyap: I misread your instructions. Here's the q35-only output http://paste.openstack.org/show/729350/ | |
| 15:44:38 | stephenfin | kashyap: And with the addition of the num_pcie_ports option http://paste.openstack.org/show/729351/ | |
| 15:44:45 | sean-k-mooney | stephenfin: kashyap that also looks incorrect but for different reasons. you asked for 2 pcie-root-ports and have 4 | |
| 15:44:46 | stephenfin | sean-k-mooney: That looks better ^ | |
| 15:45:05 | stephenfin | sean-k-mooney: Nope, first one is without that option | |
| 15:45:54 | sean-k-mooney | stephenfin: that still has 4 pcie-root-ports | |
| 15:46:06 | stephenfin | Indeed. I'd expect to see 3 | |
| 15:46:27 | sean-k-mooney | why 3 didnt you request 2? | |
| 15:46:58 | stephenfin | because someone can't count :) https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5114-L5119 | |
| 15:48:02 | stephenfin | So that option determines how many extra to add above the original amount | |
| 15:48:10 | stephenfin | *above the minimum required | |
| 15:49:06 | stephenfin | moshele: Quick question: how does one create a netdev for a connectx-3 VF? | |
| 15:49:20 | stephenfin | Assuming that's something you can do? I'm getting warnings without it | |
| 15:50:58 | moshele | stephenfin: https://community.mellanox.com/docs/DOC-1484#jive_content_id_Configuring_8_VFs_on_a_dual_port_NIC_while_all_VFs_are_probed_on_port_1 | |
| 15:51:08 | stephenfin | moshele: ta | |
| 15:51:24 | moshele | stephenfin: connectx-3 VF is not working with openstack | |
| 15:51:33 | stephenfin | ohhh, really? | |
| 15:51:48 | sean-k-mooney | stephenfin: really? that is not what the melonox docs say | |
| 15:52:12 | sean-k-mooney | oh wait connext-x 3 | |
| 15:52:40 | moshele | stephenfin: there is a missing patch https://review.openstack.org/#/c/409526/ | |
| 15:52:57 | moshele | stephenfin: because we one PF for 2 ports | |
| 15:53:21 | moshele | stephenfin: basicly it will not work with duel port nic of cx3 | |
| 15:53:26 | sean-k-mooney | moshele: oh connet-x3 is that card | |
| 15:53:31 | stephenfin | moshele: I saw that. Definitely quirky | |
| 15:53:36 | moshele | stephenfin: brb | |
| 15:53:58 | stephenfin | Guess I can revive that patch and see if we can move it forward. Would be nice to be able to use this NIC | |
| 15:55:19 | sean-k-mooney | stephenfin: that patch is not realated to your issue | |
| 15:57:44 | sean-k-mooney | stephenfin: there is another issue with that card on the neutron side but the failure you are hitting is in nova/libvirt way after teh port is already bound | |
| 15:58:12 | stephenfin | sean-k-mooney: Just realized that code snippet I sent is actually adding two different devices. L5114 is a PCIe root controller while the other is a PCIe root _port_ controller. My bad https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5114-L5119 | |
| 15:58:23 | stephenfin | So I'm guessing the first two ports are automatically added by libvirt | |
| 15:58:37 | sean-k-mooney | stephenfin: looking at https://community.mellanox.com/docs/DOC-2365 did you set probe_vf ? | |
| 15:59:37 | sean-k-mooney | stephenfin: you should set probe_vf=1 | |
| 16:01:45 | sean-k-mooney | actully im not sure.. | |
| 16:09:46 | moshele | stephenfin: yes you should to make it work with a single port vf | |
| 16:10:12 | moshele | stephenfin: what is the issue exactly? | |
| 16:12:54 | sean-k-mooney | moshele: the issue stephenfin is having is that there are no netdevs allcoated for the VFs | |
| 16:13:07 | stephenfin | moshele: Yeah, and I'm seeing this for macvtap as a result http://paste.openstack.org/show/729339/ | |
| 16:13:12 | stephenfin | I'll try the probe option now | |
| 16:13:20 | stephenfin | It wasn't clear what that was needed for | |
| 16:13:29 | sean-k-mooney | moshele: so while ip link shows the vfs allocated form the pf there are no nedevs for the vfs | |
| 16:15:01 | moshele | sean-k-mooney: there should be if you set probe_vf=X | |
| 16:15:40 | sean-k-mooney | moshele: currntly i dont think stephen set that so i assume based on the behavior he is seeing that it defaults to 0 | |
| 16:15:53 | moshele | sean-k-mooney: yes you are right | |
| 16:16:45 | sean-k-mooney | in which case if stephenfin set probe_vf=1 then vnic-type=macvtap may start working | |
| 16:17:23 | sean-k-mooney | if there is no netdev then get_ifname_by_pci_address is going to fail | |
| 16:19:08 | stephenfin | moshele: What's the difference between mlx4.conf and mlx4_core.conf? | |
| 16:20:19 | moshele | stephenfin: no difference you can use ether one I think the mlx4_core.conf comes from Mellanox OFED | |
| 16:20:32 | moshele | stephenfin: brb | |
| 16:30:33 | pvc | No compute node record for host when enabling ironic on packstack | |
| 16:31:49 | stephenfin | moshele: Yup, that was the issue. I know for next time. I am seeing another issue now though, if you have time to look at it http://paste.openstack.org/show/729352/ | |