| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-31 | |||
| 14:44:18 | jangutter | tbachman: that is... hairy. https://docs.openstack.org/neutron/latest/admin/config-sriov.html <--- I presume you've been over this a couple of times. | |
| 14:44:28 | tbachman | jangutter: ack | |
| 14:44:40 | tbachman | someone else wondered if I could do something with the vif drivers | |
| 14:44:52 | tbachman | but I think at that point, it may be too late | |
| 14:45:23 | tbachman | (and by ack, I meant to say akc on both the doc and the “hairy” bit ;-) ) | |
| 14:45:24 | jangutter | tbachman: yeah, let me find the code that does the actual allocation. | |
| 14:45:42 | efried | In the glorious future, SRIOV VFs will be resources with traits and stuff. So you'd be able to pick 'em however you want. | |
| 14:45:53 | tbachman | efried: :) | |
| 14:47:40 | cdent | mriedem, dansmith: rgerganov found the root of the problem in the import ordering (the patch that changed things). I reckon he'll have a patch soonish | |
| 14:47:49 | jangutter | tbachman: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L1888 <--- I _think_ this is the piece of code that actually generates the resource request. | |
| 14:48:25 | tbachman | jangutter: ack. mriedem had pointed me to that earlier | |
| 14:48:57 | jangutter | tbachman: so from that code, there is the physnet tag (physical_network) | |
| 14:49:01 | sahid | tbachman: hum like that i don't know, did you try to match by devname? | |
| 14:49:28 | tbachman | sahid: oh — wasn’t aware devname was available in the whitelist | |
| 14:49:35 | tbachman | I guess I should explain my current approah | |
| 14:49:54 | tbachman | in nova.conf, I use the whitelist to provide the physnet => NIC mappings | |
| 14:50:15 | tbachman | but the problem is that the neutron network has only one static segment in this case | |
| 14:50:23 | tbachman | so, although I could assign each NIC it’s own physnet | |
| 14:50:28 | tbachman | the network still only has one | |
| 14:52:19 | sahid | tbachman: when you whitlist NIC you can use devname, all the NIC can be associated with the same physnet | |
| 14:52:25 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999 | |
| 14:52:49 | sahid | i'm wondering whether you can create a port in neutron and use port binding to match the devanme | |
| 14:52:51 | tbachman | sahid: sounds good — but when I create the instance, how do I associate that with the instance | |
| 14:53:00 | tbachman | sahid: that’s what I was looking into | |
| 14:53:24 | jangutter | sahid, tbachman: is the dev_name actually _part_ of that pci request generated? | |
| 14:53:48 | tbachman | jangutter: sahid: this is my understanding so far | |
| 14:53:53 | mriedem | sean-k-mooney: interesting bug https://bugs.launchpad.net/nova/+bug/1784579 | |
| 14:53:53 | openstack | Launchpad bug 1784579 in OpenStack Compute (nova) "unable to live migrate instance after update to queens" [Undecided,New] | |
| 14:54:04 | tbachman | I create the neutron network, which has a physnet | |
| 14:54:16 | tbachman | I create a port in neutron, with the “direct” attribute | |
| 14:54:23 | sean-k-mooney | mriedem: taking a look now | |
| 14:54:40 | openstackgerrit | Merged openstack/nova master: Remove superfluous network stubbing in func test https://review.openstack.org/587134 | |
| 14:54:41 | tbachman | nova-api calls into neutron to discover the physnet for the port | |
| 14:54:54 | tbachman | for the network, I should say | |
| 14:55:11 | tbachman | it provides pci request objects to nova-compute | |
| 14:55:23 | sahid | jangutter: devname is considered as a tag | |
| 14:55:35 | tbachman | nova-compute uses its map and the request objects to determine which SRIOV NIC to select | |
| 14:55:49 | sahid | so basically we should try to match it with the request spec generated using port binding profile | |
| 14:55:57 | tbachman | sahid: exactly | |
| 14:56:18 | sean-k-mooney | mriedem: looks like neutron was not able to bind the vif on the destination so set the vif type to binding failed and we eventual get to the code to build the os-vif object and rais that exception | |
| 14:56:46 | mriedem | yes | |
| 14:56:59 | mriedem | and nova persists the "binding_failed" vif type in the info cache so we blow up on compute restart | |
| 14:57:11 | tbachman | sahid: I was able to create a “hack” in nova, where I put the “physical_network” property in the binding:profile member when I create the port in neutron | |
| 14:57:17 | sean-k-mooney | mriedem: still reading the bug content however. your multiple binding changes should help with this | |
| 14:57:17 | jangutter | tbachmanm sahid: With vnic_type=DIRECT, you the physical network from Neutron too. There are other places in the VIF Neutron stores things (like binding:profile), but... I'm not quite sure if Nova hooks 'em up in the PCI request. | |
| 14:57:45 | tbachman | and have nova-api use this if present, instead of the physnet it gets for the network | |
| 14:57:47 | sean-k-mooney | mriedem: oh i had not got that far. | |
| 14:58:12 | tbachman | that approach does have some issues though — like you have to be admin to be able to set that property | |
| 14:58:27 | tbachman | (and of course requires a change into neutron) | |
| 14:58:39 | dansmith | mriedem: yeah that binding_failed thing has been around for a while I think | |
| 14:58:43 | dansmith | I was just looking at that | |
| 14:58:50 | tbachman | perhaps the other approach is to address the HPB workflow | |
| 14:59:03 | tbachman | I guess I was just wondering if there were any other possible options | |
| 14:59:13 | dansmith | sahid: you might want to look at this bug. it's talking about GPUs, but just doing regular pci passthrough with them: https://bugs.launchpad.net/nova/+bug/1780441 | |
| 14:59:13 | openstack | Launchpad bug 1780441 in OpenStack Compute (nova) "Rebuild does not respect number of PCIe devices" [Undecided,New] | |
| 15:00:15 | sahid | dansmith: ack | |
| 15:00:35 | _ix | Good morning folks. I've got a hypervisor question related to nova, but it's in some older neutron confs for the nova user. Would you happen to have any insight as to which neutron service makes use of the nova group for vif binding and the like? | |
| 15:01:12 | _ix | I'm re-rolling some passwords, and I'm really just curious to know if I can safely restart some of these neutron processes without affecting the running machines and their network connectivity. | |
| 15:03:03 | tbachman | jangutter: sahid: thanks for your input. I’ll probably look into the HPB approach as well. | |
| 15:03:15 | tbachman | (i.e. see what it would take to make that work) | |
| 15:04:31 | jangutter | tbachman, sahid: I'd be very interested in finding out if it's actually possible to get that triggered using unmodified Nova, but I have this sneaky fear it isn't. | |
| 15:04:54 | sahid | jangutter: i think using devname and binding profile should work | |
| 15:05:08 | jangutter | tbachman, sahid: I think general PCI passthrough as dansmith mentioned might very well work, but then it's not controlled via Neutron. | |
| 15:05:15 | tbachman | jangutter: yeah, ufnortunately I haven’t found a way just yet. As you mentioned, I don’t believe nova consumes the binding:profile pieces | |
| 15:05:16 | sahid | it should be possible to also create an alias | |
| 15:05:37 | tbachman | I did look at alias, but I didn’t see a way of tying that to the physnets | |
| 15:05:38 | sahid | so select the device based on the flavor | |
| 15:05:40 | dansmith | jangutter: I didn't suggest that, I was mentioning a bug to sahid, unrelated to your discussion :) | |
| 15:05:41 | tbachman | to a unique NIC | |
| 15:05:53 | tbachman | (e.g.. it has vendor/device IDs, but not bus ID) | |
| 15:06:10 | jangutter | sahid: yeah, I last looked at this code during the Pike cycle, and back then, you didn't have the fine-grained ability to pick a specific PCI device aside from the physical_network tag... I think. | |
| 15:07:04 | jangutter | dansmith: Ah, serendipity! But similar code! | |
| 15:13:15 | openstackgerrit | Merged openstack/nova master: Fix formatting for vcpu_pin_set and reserved_huge_pages https://review.openstack.org/587206 | |
| 15:13:23 | openstackgerrit | Merged openstack/nova master: [placement] api-ref: add traits parameter https://review.openstack.org/578048 | |
| 15:13:24 | sean-k-mooney | jangutter: the ablitiy to pass a vendor id/product id via the binding profile was implemented but never merged | |
| 15:14:08 | mriedem | i don't know where we store off that binding_failed vif type, unless it's coming from neutron? | |
| 15:14:20 | dansmith | it's coming from neutron, | |
| 15:14:21 | sean-k-mooney | mriedem: it is | |
| 15:14:24 | dansmith | I didn't think we persist it | |
| 15:14:36 | dansmith | that's why I suggested on the bug to check those logs, | |
| 15:14:49 | dansmith | because I imagine the compute restart thing is just failing to re-bind or plug or whatever | |
| 15:14:53 | mriedem | i'm adding an except handler to _init_instance so we don't block compute restart | |
| 15:14:54 | sean-k-mooney | mriedem: this is coming form the port status update notification | |
| 15:15:07 | mriedem | sean-k-mooney: oh the external event? | |
| 15:15:12 | sean-k-mooney | yep | |
| 15:15:18 | dansmith | mriedem: well, what do we do? start it up without networking? | |
| 15:15:24 | mriedem | no, | |
| 15:15:26 | sean-k-mooney | the same one we were workign around for the multiple port bindings | |
| 15:15:35 | mriedem | we do what we used to do when we'd get VirtualInterfacePlugException for the same thing before the os-vif conversion | |
| 15:15:48 | mriedem | which is not block compute restart and handle the error | |
| 15:15:56 | mriedem | i'll have it up shortly | |
| 15:16:36 | sean-k-mooney | mriedem: we should be binding the vm back to the source host when the livemigration fails. | |
| 15:17:31 | sean-k-mooney | on nova compute restart we could try to fix the vif bindings for the instance | |
| 15:17:43 | mriedem | we don't do anything like that in _rollback_live_migration | |
| 15:18:22 | mriedem | if neutron is telling us that the port's vif type is 'binding_failed', how do we recover to get the proper vif type? | |
| 15:18:54 | dansmith | do we need it? | |
| 15:18:56 | sean-k-mooney | mriedem: it told us the vif type was binding failed for the dest node | |
| 15:18:59 | dansmith | if binding_failed, we stop doing whatever we're doing | |
| 15:19:52 | sean-k-mooney | mriedem: we can rebind for the current node and see if neutron returns a valid vif. | |
| 15:20:27 | sean-k-mooney | dansmith: well we cant generate the xml for an instance with vif_type binding-faild. | |