| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-03 | |||
| 13:37:45 | sean-k-mooney | amarao: perhapes invert that to hw:consoles=<one of vnc|spice|serial|None> | |
| 13:38:13 | amarao | .. and if this field is absent, everything is permitted. ack. | |
| 13:38:14 | sean-k-mooney | amarao: yes blueprints are baseicelay jsut what is the problem you would like to fix | |
| 13:38:36 | sean-k-mooney | amarao: yep if the field is not there just do waht we do today | |
| 13:39:25 | sean-k-mooney | amarao: i dont think we actully allow more then one console currently hence the one of but that is something we could figure out as we start looking at the implementaion | |
| 13:39:47 | sean-k-mooney | e.g. i dont think you can have an instance with both spice and vnc enabled | |
| 13:41:17 | amarao | ack, updated. | |
| 13:41:29 | amarao | It's more about serial/vnc thing, actually. | |
| 13:42:31 | sean-k-mooney | ya serial likely can be mix with vnc as they attach do different virualised hardware | |
| 13:42:44 | sean-k-mooney | rdp/vnc/spice all share teh virutal gpu | |
| 13:43:39 | amarao | Anyway, we are still on mitaka (sad, but true), so I'll continue to search some kind of dirty hack for that. Thank you for help. | |
| 13:43:46 | sean-k-mooney | the other think about inverting it is we could model what consoles are available on each host as traits in the placement api and use this extraspec if present to land on a host that is able to support that console | |
| 13:44:30 | stephenfin | Afternoon, gibi. Think this is something you'd be happy reviewing? https://review.openstack.org/#/c/595592/ | |
| 13:45:13 | sean-k-mooney | amarao: no worries, you may be able to use policies but that is not a area im familar with unfrotuetly. | |
| 13:46:23 | amarao | ... policy.json is like a path in a swamp. Everyone knew it exist, but rarely someone passes through. :) | |
| 13:49:30 | gibi | stephenfin: sure, looking | |
| 13:58:50 | gibi | stephenfin: https://review.openstack.org/#/c/595592/ looks good overall. I'm wondering how we can tests this other than in the unit test that is in the patch | |
| 13:59:53 | stephenfin | gibi: I'm not sure how we could do it in the gate due to the hardware dependencies. I do have an SR-IOV machine locally that I could validate the fix with, if you'd like | |
| 14:00:40 | stephenfin | gibi: Downstream we will likely look at automating this as part of the whitebox-tempest-plugin Tempest plugin, but how we actually run those tests is still very much in the air | |
| 14:02:52 | bauzas | given we're on a US holiday today, I guess we won't have any scheduler meeting? | |
| 14:02:53 | gibi | stephenfin: if it does not take more than 2 hours of your time to locally test it then could you please run that test? It would give sizeable confidence boost to me towards this patch | |
| 14:03:17 | stephenfin | gibi: No problem, should only take me 20 minutes | |
| 14:03:25 | gibi | bauzas: cdent asked it in openstack-placement where I said it is OK to me to skip | |
| 14:03:33 | gibi | stephenfin: thanks a lot | |
| 14:03:33 | bauzas | k | |
| 14:07:57 | moshele | stephenfin: Mellanox CI was update to test this | |
| 14:16:16 | sean-k-mooney | stephenfin: gibi moshele if you are refering to https://review.openstack.org/#/c/595592/3 i think that is a pretty safe change | |
| 14:18:41 | gibi | sean-k-mooney: yes, we are talking about that. If stephenfin's timeline is correct he will have test results in 5 minutes anyhow :) | |
| 14:20:14 | sean-k-mooney | stephenfin: if not i can prep an sriov env if needed but ill have to do an os reinstall on the node so it will be more like 2-3 hours before ill be able to check this directly. | |
| 14:20:36 | sean-k-mooney | * or rather if you have issues | |
| 14:23:09 | stephenfin | sean-k-mooney: Think I'm all good. Will known in 60 seconds :P | |
| 14:23:46 | sean-k-mooney | stephenfin: i take it you worked out your sriov issues then? | |
| 14:24:32 | stephenfin | gibi: Without the fix http://paste.openstack.org/show/729332/ | |
| 14:24:52 | stephenfin | sean-k-mooney: Sure did. It was a really stupid typo | |
| 14:25:08 | stephenfin | sean-k-mooney: and those warnings didn't seem to affect anything. | |
| 14:25:09 | sean-k-mooney | ha they are always the best typos | |
| 14:25:17 | sean-k-mooney | stephenfin: ya they dont | |
| 14:25:42 | sean-k-mooney | libvirt is just unhappy that a netdev does not exist for the device | |
| 14:25:43 | stephenfin | sean-k-mooney: I'm not sure why they're warnings actually and not debug (or removed entirely). Must ask moshele that | |
| 14:25:52 | stephenfin | sean-k-mooney: No, we emit those logs | |
| 14:26:46 | sean-k-mooney | stephenfin: in that case we must be trying to read stuff via ip or sysfs that we should not assume is available | |
| 14:27:24 | stephenfin | sean-k-mooney: Yeah, this is what I see http://paste.openstack.org/show/729333/ | |
| 14:27:26 | sean-k-mooney | stephenfin: in gerearal we assumt that pci device of type pf or vf are network devices which is not always correct | |
| 14:28:16 | stephenfin | gibi: and with the fix http://paste.openstack.org/show/729334/ | |
| 14:28:19 | sean-k-mooney | ya this is likely coming form the code that tryis to get the netdev feature flags | |
| 14:29:17 | sean-k-mooney | we shold only do that if a netdev exists for the vf. i dont see why this should be a warning instead of debug as you said | |
| 14:29:50 | sean-k-mooney | stephenfin: do you have the queue lenght set in your conf? | |
| 14:30:44 | sean-k-mooney | stephenfin: if not can you add it an restart the n-cpu agent and try one more time with both vnic-type=macvtap and vnic-type=direct | |
| 14:31:27 | stephenfin | sean-k-mooney: nope http://paste.openstack.org/show/729336/ | |
| 14:32:50 | sean-k-mooney | ? that is the domain xml for a direct passhtoug device | |
| 14:33:18 | sean-k-mooney | do you have the queue lenght set in /etc/nova/nova-cpu.conf | |
| 14:33:49 | sean-k-mooney | if so then that is correct for direct but we should see it populated in the driver section for macvtap | |
| 14:34:36 | stephenfin | sean-k-mooney: Oh, good point. 02:00.0 is the PF. That's a 'direct' attach | |
| 14:35:00 | stephenfin | Used 'openstack port create --network 84b18250-c0d3-4594-92d1-19328fb37da5 --vnic-type direct sriov-port' | |
| 14:35:10 | sean-k-mooney | well direct could also be a vf | |
| 14:35:40 | stephenfin | Based on the address, that's the PF. I probably have my whitelist set up wrong | |
| 14:36:00 | sean-k-mooney | stephenfin: yes with that commandline we do not expect the domain xml to contain the queulenght | |
| 14:36:19 | sean-k-mooney | the fact its a pf or vf is irelevent in this case | |
| 14:36:33 | stephenfin | Cool. So that's working as expected | |
| 14:36:53 | sean-k-mooney | yes but you have not confirms if you set the queue lenght in your config | |
| 14:36:54 | stephenfin | The macvtap route is interesting though. I'm seeing this | |
| 14:37:05 | stephenfin | sean-k-mooney: Oh, sorry. Yes, I have set it | |
| 14:37:25 | sean-k-mooney | ok cool. so for macvtap what is the result? | |
| 14:37:43 | stephenfin | and as a result, got this error (without the fix) http://paste.openstack.org/show/729332/ | |
| 14:38:11 | stephenfin | sean-k-mooney: For macvtap, I'm seeing http://paste.openstack.org/show/729339/ | |
| 14:38:27 | stephenfin | so it turns out those warnings are a big deal :) | |
| 14:38:40 | stephenfin | and I didn't think they were simply because it was the PF being attached instead of the VF | |
| 14:39:05 | sean-k-mooney | stephenfin: is the pf still bound? | |
| 14:39:11 | sean-k-mooney | e.g to a vm | |
| 14:39:17 | stephenfin | No, I killed that VM | |
| 14:39:31 | stephenfin | There are no VMs running | |
| 14:39:34 | sean-k-mooney | you should be able to use the pf for the macvtap if you delete the neturon port that is currectly using it | |
| 14:40:12 | sean-k-mooney | actullly if you deleted the vm that might be enough | |
| 14:40:53 | sean-k-mooney | stephenfin: it makes sense however if the netdev does not exist then you cant create a macvtap from it | |
| 14:42:04 | stephenfin | sean-k-mooney: Nope, same error after deleting the two ports :/ | |
| 14:42:17 | stephenfin | Yeah, it does. I've no idea how to create said netdevs though | |
| 14:42:42 | sean-k-mooney | stephenfin: can you try deaclloating the vf by echoing 0 into sriov_num_vf then restart the n-cpu agent so that only the pf is available | |
| 14:43:22 | sean-k-mooney | stephenfin: the pf shows up with ifconfig/ip link right? | |
| 14:43:41 | stephenfin | sean-k-mooney: Yup, along with its VMs. It's in up state | |
| 14:44:05 | sean-k-mooney | stephenfin: VMs? | |
| 14:44:13 | stephenfin | VFs, sorry | |
| 14:44:28 | stephenfin | Heh, "-su: echo: write error: Function not implemented" | |
| 14:45:12 | sean-k-mooney | stephenfin: you could just change your whitelisnt to onely have 02:00:00 address | |
| 14:45:27 | openstackgerrit | sahid proposed openstack/nova stable/rocky: hardware: fix memory check usage for small/large pages https://review.openstack.org/599413 | |
| 14:47:01 | sean-k-mooney | stephenfin: gibi anyway it sounds like the patch is working as expect in terms of fixing the direct case. the macvtap case likely will work if the nic was correctly configured | |
| 14:51:27 | openstackgerrit | sahid proposed openstack/nova stable/rocky: libvirt: Always escape IPv6 addresses when used in migration URI https://review.openstack.org/599419 | |
| 14:55:12 | gibi | stephenfin, sean-k-mooney: thanks for the extra check. I'm a lot more confident to +A that patch | |
| 15:06:44 | bauzas | stephenfin: I was looking at https://review.openstack.org/#/c/588570/2 | |
| 15:07:10 | bauzas | stephenfin: maybe because it's changing a lot of tech debt, that's why people wonder if that's a feature backport | |
| 15:07:17 | bauzas | while not, of course | |
| 15:07:35 | bauzas | after all, the change is hairy, you know | |
| 15:07:53 | bauzas | and https://review.openstack.org/#/c/588570/2/nova/objects/numa.py is a bit concerning | |
| 15:08:01 | stephenfin | bauzas: Yeah, I get that. Unfortunately it's pretty much unavoidable though :( | |
| 15:09:02 | stephenfin | bauzas: Is it? None of those functions are stored in a DB so there's no issues around versioning | |
| 15:10:28 | bauzas | stephenfin: yup, we're not changing the RPC API | |
| 15:10:33 | sean-k-mooney | stephenfin: well we could greatly simply that code by rewriting it in a declarive/fuctional style with filters and transforms instead of the imperitive stily that is currently used for numa affinity | |
| 15:11:00 | bauzas | stephenfin: I meant, when someone provides a backport change asking to modify an o.vo object, we need to look more | |
| 15:11:17 | sean-k-mooney | so the complexity is not entirely due to the problem statement however that is not backportable in anyway | |
| 15:11:27 | bauzas | and why it's taking me a long time to look at it | |
| 15:11:31 | stephenfin | bauzas: Yup, that's fair :) I figured as much | |