| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-03 | |||
| 15:24:14 | sean-k-mooney | stephenfin: that is going to make live migration even more of a pain | |
| 15:24:45 | stephenfin | sean-k-mooney: I think it's like TX/RX queue size. The expectation would be that every node would have the same value (managed by Ansible or whatever other tool you're using) | |
| 15:25:03 | stephenfin | kashyap: With or without the q35 flag? | |
| 15:25:05 | sean-k-mooney | stephenfin: i think setting this suff in the nova conf is a serios antipatern | |
| 15:25:14 | kashyap | stephenfin: With, please. | |
| 15:25:22 | kashyap | stephenfin: Also - PCIe goodies are 'q35' _only_, BTW. | |
| 15:25:40 | sean-k-mooney | stephenfin: that kind of assumtion make upgrading a pain as once set you can never safly change it | |
| 15:26:22 | kashyap | sean-k-mooney: Before we forget, perhaps we should file a bug to revisit the PCIe root ports thingie? | |
| 15:26:51 | sean-k-mooney | which one. the auto allocation of a pcie route per numa node? | |
| 15:28:35 | stephenfin | kashyap: http://paste.openstack.org/show/729349/ | |
| 15:28:52 | stephenfin | sean-k-mooney: Fortunately for you, I have zilch on my plate rn. Send em my way | |
| 15:29:33 | sean-k-mooney | ill create an ether pad with them the issue is i have not got around to filing/writing them all. | |
| 15:31:44 | sean-k-mooney | stephenfin: im not sure that is correct for num_pcie_ports=2 | |
| 15:33:04 | sean-k-mooney | stephenfin: your machine type is pc-i440fx-bionic so you are not enableing q35 chipset so no pcie at all | |
| 15:33:57 | stephenfin | I don't think so either. I was expecting to see a <pci model='pcie-root-port'> element | |
| 15:34:45 | stephenfin | Sorry - <controller type='pci' model='pcie-root-port'> | |
| 15:35:01 | stephenfin | So why didn't that property get applied... | |
| 15:35:35 | sean-k-mooney | well currently everything in that xml is connect to the pci bus not pcie. also as i sad you are not using the q35 chipset so no pcie is available | |
| 15:35:52 | stephenfin | Oh yeah, I meant the 'hw:machine_type' property | |
| 15:36:17 | sean-k-mooney | i think its only read for arm... | |
| 15:36:50 | stephenfin | waaaait | |
| 15:36:56 | sean-k-mooney | it should be read on all archs but just going on the glance metadata docs i think its ignored on x86. i would prefer if it was not | |
| 15:37:09 | stephenfin | I misread kashyap's comments. You can't do this with extra specs | |
| 15:41:10 | stephenfin | kashyap: I misread your instructions. Here's the q35-only output http://paste.openstack.org/show/729350/ | |
| 15:44:38 | stephenfin | kashyap: And with the addition of the num_pcie_ports option http://paste.openstack.org/show/729351/ | |
| 15:44:45 | sean-k-mooney | stephenfin: kashyap that also looks incorrect but for different reasons. you asked for 2 pcie-root-ports and have 4 | |
| 15:44:46 | stephenfin | sean-k-mooney: That looks better ^ | |
| 15:45:05 | stephenfin | sean-k-mooney: Nope, first one is without that option | |
| 15:45:54 | sean-k-mooney | stephenfin: that still has 4 pcie-root-ports | |
| 15:46:06 | stephenfin | Indeed. I'd expect to see 3 | |
| 15:46:27 | sean-k-mooney | why 3 didnt you request 2? | |
| 15:46:58 | stephenfin | because someone can't count :) https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5114-L5119 | |
| 15:48:02 | stephenfin | So that option determines how many extra to add above the original amount | |
| 15:48:10 | stephenfin | *above the minimum required | |
| 15:49:06 | stephenfin | moshele: Quick question: how does one create a netdev for a connectx-3 VF? | |
| 15:49:20 | stephenfin | Assuming that's something you can do? I'm getting warnings without it | |
| 15:50:58 | moshele | stephenfin: https://community.mellanox.com/docs/DOC-1484#jive_content_id_Configuring_8_VFs_on_a_dual_port_NIC_while_all_VFs_are_probed_on_port_1 | |
| 15:51:08 | stephenfin | moshele: ta | |
| 15:51:24 | moshele | stephenfin: connectx-3 VF is not working with openstack | |
| 15:51:33 | stephenfin | ohhh, really? | |
| 15:51:48 | sean-k-mooney | stephenfin: really? that is not what the melonox docs say | |
| 15:52:12 | sean-k-mooney | oh wait connext-x 3 | |
| 15:52:40 | moshele | stephenfin: there is a missing patch https://review.openstack.org/#/c/409526/ | |
| 15:52:57 | moshele | stephenfin: because we one PF for 2 ports | |
| 15:53:21 | moshele | stephenfin: basicly it will not work with duel port nic of cx3 | |
| 15:53:26 | sean-k-mooney | moshele: oh connet-x3 is that card | |
| 15:53:31 | stephenfin | moshele: I saw that. Definitely quirky | |
| 15:53:36 | moshele | stephenfin: brb | |
| 15:53:58 | stephenfin | Guess I can revive that patch and see if we can move it forward. Would be nice to be able to use this NIC | |
| 15:55:19 | sean-k-mooney | stephenfin: that patch is not realated to your issue | |
| 15:57:44 | sean-k-mooney | stephenfin: there is another issue with that card on the neutron side but the failure you are hitting is in nova/libvirt way after teh port is already bound | |
| 15:58:12 | stephenfin | sean-k-mooney: Just realized that code snippet I sent is actually adding two different devices. L5114 is a PCIe root controller while the other is a PCIe root _port_ controller. My bad https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5114-L5119 | |
| 15:58:23 | stephenfin | So I'm guessing the first two ports are automatically added by libvirt | |
| 15:58:37 | sean-k-mooney | stephenfin: looking at https://community.mellanox.com/docs/DOC-2365 did you set probe_vf ? | |
| 15:59:37 | sean-k-mooney | stephenfin: you should set probe_vf=1 | |
| 16:01:45 | sean-k-mooney | actully im not sure.. | |
| 16:09:46 | moshele | stephenfin: yes you should to make it work with a single port vf | |
| 16:10:12 | moshele | stephenfin: what is the issue exactly? | |
| 16:12:54 | sean-k-mooney | moshele: the issue stephenfin is having is that there are no netdevs allcoated for the VFs | |
| 16:13:07 | stephenfin | moshele: Yeah, and I'm seeing this for macvtap as a result http://paste.openstack.org/show/729339/ | |
| 16:13:12 | stephenfin | I'll try the probe option now | |
| 16:13:20 | stephenfin | It wasn't clear what that was needed for | |
| 16:13:29 | sean-k-mooney | moshele: so while ip link shows the vfs allocated form the pf there are no nedevs for the vfs | |
| 16:15:01 | moshele | sean-k-mooney: there should be if you set probe_vf=X | |
| 16:15:40 | sean-k-mooney | moshele: currntly i dont think stephen set that so i assume based on the behavior he is seeing that it defaults to 0 | |
| 16:15:53 | moshele | sean-k-mooney: yes you are right | |
| 16:16:45 | sean-k-mooney | in which case if stephenfin set probe_vf=1 then vnic-type=macvtap may start working | |
| 16:17:23 | sean-k-mooney | if there is no netdev then get_ifname_by_pci_address is going to fail | |
| 16:19:08 | stephenfin | moshele: What's the difference between mlx4.conf and mlx4_core.conf? | |
| 16:20:19 | moshele | stephenfin: no difference you can use ether one I think the mlx4_core.conf comes from Mellanox OFED | |
| 16:20:32 | moshele | stephenfin: brb | |
| 16:30:33 | pvc | No compute node record for host when enabling ironic on packstack | |
| 16:31:49 | stephenfin | moshele: Yup, that was the issue. I know for next time. I am seeing another issue now though, if you have time to look at it http://paste.openstack.org/show/729352/ | |
| 16:32:04 | stephenfin | (the issue is at the end. The rest is the configuration I have) | |
| 16:33:56 | sean-k-mooney | stephenfin: if you do a virsh nodedev-list can you see net_enp2s1d1_fa_16_3e_38_ea_a6 | |
| 16:34:41 | stephenfin | I can't. I'm not sure where it got that from | |
| 16:34:52 | stephenfin | Actually, it only started happening once I created an instance with macvtap | |
| 16:35:02 | stephenfin | I wonder if this is another reflection of the dual-port issues | |
| 16:35:08 | stephenfin | *another example | |
| 16:36:48 | pvc | No compute node record for host when enabling ironic on packstack | |
| 16:36:50 | pvc | anyone | |
| 16:36:51 | pvc | can help me | |
| 16:37:50 | sean-k-mooney | pvc: sorry you proably wont find that many people with packstack knowladge here | |
| 16:43:32 | pvc | but the main problem is | |
| 16:43:36 | pvc | it cannot find any hypervisor | |
| 16:43:41 | pvc | because i enable the ironic driver instead of libvir | |
| 16:44:42 | sean-k-mooney | pvc: when you enable ironic it should still run at least one instance of the nova-compute agent that loads the ironic driver. this agent should create at least one compute service record | |
| 16:45:33 | sean-k-mooney | each node you enrole in ionic should then be exposed as a seperate compute node/ hypervior record mapped to that compute servcice | |
| 16:46:44 | sean-k-mooney | if the ironic nova-compute agent does not start or fails for some reason you will have no compute node entries | |
| 16:49:27 | pvc | http://paste.openstack.org/show/729355/ | |
| 16:50:06 | sean-k-mooney | pvc: yes.. did you manually add nodes to ironic. also this does not show you the compute noes | |
| 16:50:38 | pvc | ahmm no. for now we're not manually adding nodes to ironic | |
| 16:50:41 | pvc | we just install openstack using packstack | |
| 16:50:45 | pvc | i found something simirlar | |
| 16:50:49 | pvc | http://getvasanth.blogspot.com/2015/09/getting-start-with-ironic.html | |
| 16:51:35 | sean-k-mooney | pvc: what happens if you run openstack compute service list and openstack compute agent list | |
| 16:52:20 | pvc | http://paste.openstack.org/show/729356/ | |
| 16:53:27 | sean-k-mooney | pvc: well it looks like you have 3 registers compute services that said i would have expected to also see them in the agent list | |
| 16:54:08 | pvc | but the server i'm using have an ipmi | |
| 16:54:14 | sean-k-mooney | in either case teh service state is up so i would suspec the nova pices is running correctly | |