| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-03 | |||
| 15:11:31 | stephenfin | bauzas: Yup, that's fair :) I figured as much | |
| 15:12:06 | bauzas | sean-k-mooney: well, it would be an only-stable change, then ? | |
| 15:12:24 | bauzas | if so, that's difficult | |
| 15:12:39 | kashyap | Hey folks, can anyone do a small test for me: Boot a Nova instance with Q35 machine type, and post the complete guest XML somewhere, please | |
| 15:12:39 | stephenfin | Hmm, I wonder how I can resolve the issue, in that case. We could stick on more sellotape but I haven't even thought how I'd do that | |
| 15:12:54 | kashyap | You can set Q35 in two ways: either vai the Nova config: `hw_machine_type=x86_64=q35`, or on the Glance image image: `openstack image set --property hw_machine_type=x86_64=q35 Fedora28Template` | |
| 15:12:57 | stephenfin | kashyap: What properties would I need to set to do that? | |
| 15:12:58 | stephenfin | Ah :) | |
| 15:13:04 | sean-k-mooney | bauzas: rewriting the numa code to be fucntional instead of imperitve. i was talking generally not for this specific patch | |
| 15:13:17 | stephenfin | sean-k-mooney: oh, gotcha | |
| 15:13:51 | sean-k-mooney | stephenfin: for this patch the only object change i see is a new property | |
| 15:13:55 | stephenfin | Yeah, all that code is crufty af. I might look to take that "track CPUs in placement" spec from jaypipes this cycle, assuming he's oversubscribed as usual | |
| 15:14:10 | stephenfin | ...though I don't really want to wade into the great placement war of 2018 | |
| 15:14:51 | kashyap | stephenfin: Thank you! | |
| 15:15:14 | sean-k-mooney | stephenfin: well we could clean up that code without ever touching placement but thats a different topic. e.g. keep existing functionality jsut do it differently | |
| 15:16:04 | bauzas | sean-k-mooney: oh well, sure we could do it | |
| 15:16:18 | bauzas | or use Haskell :p | |
| 15:16:30 | kashyap | stephenfin: I need to head to my Dutch class (starts afresh) in a few; when you get to it, post it here, I'll take a look. Gracias! | |
| 15:16:48 | bauzas | (FWIW, I *love* Haskell) | |
| 15:16:53 | sean-k-mooney | stephenfin: bauzas anyway back to https://review.openstack.org/#/c/588570/2 the only object change there is a syntasied property that will now be included in teh serialisation of the object so i think this is backportable as there is no rpc change right? | |
| 15:17:00 | kashyap | (s/starts afresh/new level starts today) | |
| 15:17:08 | bauzas | sean-k-mooney: I feel so | |
| 15:17:34 | sean-k-mooney | bauzas: that explains a lot .... | |
| 15:18:14 | bauzas | very simple | |
| 15:18:36 | stephenfin | bauzas: I studied Haskell in NZ and adored it (helped me grasp recursion at long last). Alas, it's not something I've had the chance to work with since | |
| 15:18:41 | stephenfin | Prolog, on the other hand. Ick | |
| 15:19:11 | sean-k-mooney | stephenfin: python has really good fuctional programing supprot built in by they way. | |
| 15:19:15 | bauzas | sean-k-mooney: since we're adding a new property, that's not chaning what we pass over RPC | |
| 15:19:25 | bauzas | sean-k-mooney: my only concern is on the left side | |
| 15:19:29 | bauzas | ie.what we remove | |
| 15:20:07 | stephenfin | kashyap: Here's the output of 'virsh dumpxml' http://paste.openstack.org/show/729345/ | |
| 15:20:29 | bauzas | anyway, time for wrapping off | |
| 15:20:58 | kashyap | stephenfin: Nice, /me clicks | |
| 15:21:06 | stephenfin | kashyap: and here are the commands I ran to get there http://paste.openstack.org/show/729348/ | |
| 15:21:18 | bauzas | today was school's back for my daughters, so I want to take time to discuss with them | |
| 15:21:27 | bauzas | so, calling it a day \o | |
| 15:21:40 | sean-k-mooney | bauzas: as you should. o/ | |
| 15:21:43 | stephenfin | kashyap: That's using commit 90cdf807503f94bfbd3ae9a62a9bfe466a18bcad (so about two month old cold) | |
| 15:21:47 | stephenfin | bauzas: Enjoy o/ | |
| 15:21:48 | kashyap | stephenfin: Nice. Can I ask one more, since you have the env? Also related to PCIe | |
| 15:21:56 | stephenfin | kashyap: shoot | |
| 15:22:45 | kashyap | stephenfin: Set the 'num_pcie_ports=2' in nova.conf, restart the Compute service, and again boot an instance afresh, and post the guest XML | |
| 15:22:56 | kashyap | Thanks a for the quick response :-) | |
| 15:23:24 | stephenfin | kashyap: num_pcie_ports ? That's not a nova.conf option, is it? | |
| 15:23:25 | sean-k-mooney | kashyap: wait why is that a nova conf setting and not a flavor extra spec | |
| 15:23:46 | stephenfin | Oh, what do you know, so it is | |
| 15:23:57 | kashyap | sean-k-mooney: Hehe, I haven't checked if there's a flavor extra spec for it or not. | |
| 15:24:00 | sean-k-mooney | stephenfin: really that sucks. | |
| 15:24:14 | sean-k-mooney | stephenfin: that is going to make live migration even more of a pain | |
| 15:24:45 | stephenfin | sean-k-mooney: I think it's like TX/RX queue size. The expectation would be that every node would have the same value (managed by Ansible or whatever other tool you're using) | |
| 15:25:03 | stephenfin | kashyap: With or without the q35 flag? | |
| 15:25:05 | sean-k-mooney | stephenfin: i think setting this suff in the nova conf is a serios antipatern | |
| 15:25:14 | kashyap | stephenfin: With, please. | |
| 15:25:22 | kashyap | stephenfin: Also - PCIe goodies are 'q35' _only_, BTW. | |
| 15:25:40 | sean-k-mooney | stephenfin: that kind of assumtion make upgrading a pain as once set you can never safly change it | |
| 15:26:22 | kashyap | sean-k-mooney: Before we forget, perhaps we should file a bug to revisit the PCIe root ports thingie? | |
| 15:26:51 | sean-k-mooney | which one. the auto allocation of a pcie route per numa node? | |
| 15:28:35 | stephenfin | kashyap: http://paste.openstack.org/show/729349/ | |
| 15:28:52 | stephenfin | sean-k-mooney: Fortunately for you, I have zilch on my plate rn. Send em my way | |
| 15:29:33 | sean-k-mooney | ill create an ether pad with them the issue is i have not got around to filing/writing them all. | |
| 15:31:44 | sean-k-mooney | stephenfin: im not sure that is correct for num_pcie_ports=2 | |
| 15:33:04 | sean-k-mooney | stephenfin: your machine type is pc-i440fx-bionic so you are not enableing q35 chipset so no pcie at all | |
| 15:33:57 | stephenfin | I don't think so either. I was expecting to see a <pci model='pcie-root-port'> element | |
| 15:34:45 | stephenfin | Sorry - <controller type='pci' model='pcie-root-port'> | |
| 15:35:01 | stephenfin | So why didn't that property get applied... | |
| 15:35:35 | sean-k-mooney | well currently everything in that xml is connect to the pci bus not pcie. also as i sad you are not using the q35 chipset so no pcie is available | |
| 15:35:52 | stephenfin | Oh yeah, I meant the 'hw:machine_type' property | |
| 15:36:17 | sean-k-mooney | i think its only read for arm... | |
| 15:36:50 | stephenfin | waaaait | |
| 15:36:56 | sean-k-mooney | it should be read on all archs but just going on the glance metadata docs i think its ignored on x86. i would prefer if it was not | |
| 15:37:09 | stephenfin | I misread kashyap's comments. You can't do this with extra specs | |
| 15:41:10 | stephenfin | kashyap: I misread your instructions. Here's the q35-only output http://paste.openstack.org/show/729350/ | |
| 15:44:38 | stephenfin | kashyap: And with the addition of the num_pcie_ports option http://paste.openstack.org/show/729351/ | |
| 15:44:45 | sean-k-mooney | stephenfin: kashyap that also looks incorrect but for different reasons. you asked for 2 pcie-root-ports and have 4 | |
| 15:44:46 | stephenfin | sean-k-mooney: That looks better ^ | |
| 15:45:05 | stephenfin | sean-k-mooney: Nope, first one is without that option | |
| 15:45:54 | sean-k-mooney | stephenfin: that still has 4 pcie-root-ports | |
| 15:46:06 | stephenfin | Indeed. I'd expect to see 3 | |
| 15:46:27 | sean-k-mooney | why 3 didnt you request 2? | |
| 15:46:58 | stephenfin | because someone can't count :) https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5114-L5119 | |
| 15:48:02 | stephenfin | So that option determines how many extra to add above the original amount | |
| 15:48:10 | stephenfin | *above the minimum required | |
| 15:49:06 | stephenfin | moshele: Quick question: how does one create a netdev for a connectx-3 VF? | |
| 15:49:20 | stephenfin | Assuming that's something you can do? I'm getting warnings without it | |
| 15:50:58 | moshele | stephenfin: https://community.mellanox.com/docs/DOC-1484#jive_content_id_Configuring_8_VFs_on_a_dual_port_NIC_while_all_VFs_are_probed_on_port_1 | |
| 15:51:08 | stephenfin | moshele: ta | |
| 15:51:24 | moshele | stephenfin: connectx-3 VF is not working with openstack | |
| 15:51:33 | stephenfin | ohhh, really? | |
| 15:51:48 | sean-k-mooney | stephenfin: really? that is not what the melonox docs say | |
| 15:52:12 | sean-k-mooney | oh wait connext-x 3 | |
| 15:52:40 | moshele | stephenfin: there is a missing patch https://review.openstack.org/#/c/409526/ | |
| 15:52:57 | moshele | stephenfin: because we one PF for 2 ports | |
| 15:53:21 | moshele | stephenfin: basicly it will not work with duel port nic of cx3 | |
| 15:53:26 | sean-k-mooney | moshele: oh connet-x3 is that card | |
| 15:53:31 | stephenfin | moshele: I saw that. Definitely quirky | |
| 15:53:36 | moshele | stephenfin: brb | |
| 15:53:58 | stephenfin | Guess I can revive that patch and see if we can move it forward. Would be nice to be able to use this NIC | |
| 15:55:19 | sean-k-mooney | stephenfin: that patch is not realated to your issue | |
| 15:57:44 | sean-k-mooney | stephenfin: there is another issue with that card on the neutron side but the failure you are hitting is in nova/libvirt way after teh port is already bound | |
| 15:58:12 | stephenfin | sean-k-mooney: Just realized that code snippet I sent is actually adding two different devices. L5114 is a PCIe root controller while the other is a PCIe root _port_ controller. My bad https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5114-L5119 | |