| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-09 | |||
| 12:40:22 | sean-k-mooney | if we can live with the complexity then we proably shoudl keep it | |
| 12:40:28 | sean-k-mooney | ralonsoh: i found it | |
| 12:40:36 | gibi | OK, I will plan with the complexity | |
| 12:40:37 | ralonsoh | ah perfect | |
| 12:40:39 | sean-k-mooney | ralonsoh: https://review.opendev.org/q/topic:bp%252Fenable-sriov-nic-features | |
| 12:40:47 | sean-k-mooney | ralonsoh: your code for tracking pci device in placment | |
| 12:41:12 | sean-k-mooney | ralonsoh: we are just discussing the spec to enable it. gibi will be taking on that feature | |
| 12:41:20 | ralonsoh | perfect | |
| 12:41:39 | gibi | ralonsoh: https://review.opendev.org/c/openstack/nova-specs/+/791047/4/specs/zed/approved/pci-device-tracking-in-placement.rst this is the spec if you are interested :) | |
| 12:41:47 | ralonsoh | sure | |
| 12:42:01 | gibi | sean-k-mooney: so I have one more open question | |
| 12:42:08 | sean-k-mooney | gibi: go for it | |
| 12:42:09 | gibi | upgrade | |
| 12:42:19 | gibi | obviously rolling upgrade is a pain | |
| 12:42:35 | sean-k-mooney | ah well you burn down the datacenter and build a new one in the ashes | |
| 12:42:45 | sean-k-mooney | obvisouly the least painful approch | |
| 12:42:46 | gibi | in the PCPU case we did a fallback query | |
| 12:43:01 | gibi | to allow schedling to not-yet upgraded computes | |
| 12:43:05 | sean-k-mooney | yep | |
| 12:43:13 | sean-k-mooney | we could make the prefilter configurable | |
| 12:43:14 | gibi | I'm not sure how we did the allocation in that case | |
| 12:43:19 | gibi | but | |
| 12:43:51 | gibi | in the PCI case if we select a host based on the fallback query then on that host the scheduler will not allocate PCI devices in placement | |
| 12:44:12 | sean-k-mooney | right so i would not use a fallback | |
| 12:44:21 | sean-k-mooney | by default we shoudl reprot the inventores to placemnt | |
| 12:44:28 | sean-k-mooney | and ahve a prefileter | |
| 12:44:43 | sean-k-mooney | the prefilter would add the pci device request to the query | |
| 12:44:52 | sean-k-mooney | and we disable it by default in zed | |
| 12:44:59 | sean-k-mooney | then enable it by default in AA | |
| 12:45:18 | sean-k-mooney | so you would rolling upgrade to Zed | |
| 12:45:29 | sean-k-mooney | then enable the prefilter once all host are upgraded | |
| 12:45:59 | sean-k-mooney | you likely would have to then do a heal-allocation like command to update the allcotiosn of existign instances | |
| 12:46:07 | gibi | ahh i see | |
| 12:46:34 | sean-k-mooney | we could also have a nova-status check | |
| 12:46:36 | gibi | so we report devices but we don't allocate yet | |
| 12:46:44 | sean-k-mooney | yep | |
| 12:46:52 | gibi | then when every compute is ready we do a heal and then start allocating | |
| 12:47:06 | sean-k-mooney | yes using the claim in the pci_devices table | |
| 12:47:26 | gibi | yepp we keep using the claim and the pci_device table | |
| 12:47:29 | gibi | anyhow | |
| 12:47:36 | gibi | as that tracks exact VF PCI addresses | |
| 12:47:41 | gibi | Placement wont | |
| 12:47:48 | sean-k-mooney | yep so form the contolers | |
| 12:48:00 | sean-k-mooney | we will have all the info in the pci_devices table to heal the allocations | |
| 12:48:17 | sean-k-mooney | since we also have the parent adresses | |
| 12:48:24 | sean-k-mooney | we can constuct the RP names | |
| 12:49:07 | gibi | yepp | |
| 12:49:39 | sean-k-mooney | we could consider | |
| 12:49:51 | sean-k-mooney | if we can activate teh filter based on min compute service version | |
| 12:50:21 | sean-k-mooney | if we were to do that we would likely need the compute agent to heal the allcoations automaticaly | |
| 12:50:38 | sean-k-mooney | perhaps on starup or in the upsadate_avaiable_resouces periodic task | |
| 12:51:29 | sean-k-mooney | im not sure if we want that level of compleixty but we already do reshapes in init_host | |
| 12:52:11 | sean-k-mooney | do you think that is too much "magic"/complexity | |
| 12:52:38 | sean-k-mooney | it would make the operator experince much nicer as it woudl just start working once everythin was upgraded | |
| 12:52:42 | gibi | this reshap will be just RP creation, we won't move things | |
| 12:52:59 | gibi | so calling that code from periodic feels OK | |
| 12:53:14 | gibi | if we have that then it is safe to enable the prefilter automatically | |
| 12:53:27 | gibi | by the compute min version | |
| 12:53:31 | sean-k-mooney | right so when the compaute-agent start with the new code. the first time it create the inventroeis it would also update the allcoations | |
| 12:53:55 | sean-k-mooney | and then the prefilter woudl activate once all computes are upgraded | |
| 12:54:01 | sean-k-mooney | based on min verion check | |
| 12:54:25 | sean-k-mooney | the proably i see with this would be move operations before the prefilter is enabled | |
| 12:54:38 | sean-k-mooney | unless we have it continue to heal | |
| 12:54:55 | sean-k-mooney | untill the min version reaces the required version | |
| 12:55:31 | sean-k-mooney | there would be some inconsitency for a time but the pci_tracker would enforce the corret behaivor with regards to not over subscibing | |
| 12:55:56 | gibi | yeah we have the pci tracker and pci claim as a fallback | |
| 12:56:03 | gibi | so we can move even if the prefilter is disabled | |
| 12:56:28 | gibi | just have to have a way to heal the placement allocation | |
| 12:56:33 | gibi | eventually | |
| 12:56:36 | sean-k-mooney | yep | |
| 12:56:48 | gibi | OK, I think I got my answers | |
| 12:56:55 | gibi | thank you for your time | |
| 12:56:59 | gibi | I really appreciate it | |
| 12:57:00 | sean-k-mooney | which again can use current and min service version to disable the healing when its not needed | |
| 12:57:04 | sean-k-mooney | no worries | |
| 12:57:12 | sean-k-mooney | im excited to see this moving forward | |
| 12:57:20 | sean-k-mooney | will you summerise this in the spec | |
| 12:57:26 | sean-k-mooney | perhaps like to the irc logs | |
| 12:57:33 | sean-k-mooney | *link | |
| 12:57:36 | gibi | I will do the summary | |
| 12:57:41 | gibi | and linking to the log | |
| 12:57:47 | gibi | then I will respin the spec | |
| 12:58:02 | gibi | and trim the questions | |
| 12:58:34 | gibi | I'm excited to stat coding up some of these in nova and watch them fail in the func env :) | |
| 12:58:39 | gibi | it will be fun | |
| 12:59:02 | sean-k-mooney | gibi: on a related not you reviewed Uggla spec. there was kind of an open question regarding updating hte AZ when you specify a host did you weigh in on that. | |
| 12:59:33 | gibi | I saw it and I think it was settled, I had no objection. But then I will doulecheck | |
| 12:59:38 | gibi | doublecheck | |
| 12:59:57 | sean-k-mooney | gibi: ack ill try and review it again shortly so | |
| 13:01:11 | sean-k-mooney | gibi: on a more selfish note i could also use your input on something else but its not super urgent https://review.opendev.org/c/openstack/nova/+/841017/1/nova/virt/libvirt/driver.py | |
| 13:02:02 | sean-k-mooney | i dont think that is 100% correct but i works for vdpa i need to test it with VFs and other vnic-types | |
| 13:02:48 | sean-k-mooney | basically we are curently unpluging neutron interface using _detach_pci_dev for suspend | |
| 13:03:07 | sean-k-mooney | that does not work for vdpa and im pretty sure it does not work in general | |
| 13:03:41 | sean-k-mooney | so i need to verify that and file a bug | |
| 13:07:15 | gibi | I never tried suspend with PCI / neutron SRIOV. So I neither confirm now deny that it works | |
| 13:07:38 | sean-k-mooney | it used to but its been a very long time since i checked it. | |
| 13:08:01 | sean-k-mooney | so ya i need to test it with differnt backends | |
| 13:08:15 | gibi | but your comments seems valid that if something is an interface then that cannot be detached as a hostdev | |
| 13:09:29 | sean-k-mooney | i have the ablity to test hardware offloaded ovs and sriov at home and i still have the servers i was usign for vdpa although ill be giving those abck today | |
| 13:10:08 | sean-k-mooney | so i can see if i can test the differnt combinations | |