| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-07 | |||
| 14:19:24 | sean-k-mooney | but i was having trouble geting cirrus to boot with uefi in genreal and that is required for aarch64 | |
| 14:19:44 | sean-k-mooney | so uefi on x86 and aarch64 did not seam to work with cirros | |
| 14:20:14 | sean-k-mooney | i got an error with the alingment or either the filesystme or uefi firmware in the console and then no output | |
| 14:20:29 | sean-k-mooney | chateaulav: what have you been using to test with qemu directly? | |
| 14:20:48 | chateaulav | sean-k-mooney: yeah havent tried the cirros on aarch hardware, but on x86 works fine. | |
| 14:20:59 | chateaulav | have code running in a deployment ostack env | |
| 14:21:00 | sean-k-mooney | did you do an install form iso or something else | |
| 14:21:37 | sean-k-mooney | chateaulav: ok so you did arch64 via qemu on x86 with the aarch64 cirros image | |
| 14:21:44 | chateaulav | http://download.cirros-cloud.net/0.5.2/cirros-0.5.2-aarch64-disk.img | |
| 14:21:44 | chateaulav | sean-k-mooney: qcow | |
| 14:21:57 | chateaulav | yes | |
| 14:22:19 | sean-k-mooney | ya i tried that and it woudl not boot on may ubuntu aarch64 vm on my macbook air | |
| 14:22:38 | sean-k-mooney | i was trying to see if i could use that as a arm dev env | |
| 14:23:03 | sean-k-mooney | chateaulav: did you put anything special in the glacne metadtaa | |
| 14:23:19 | chateaulav | hw_emulation_architecture='aarch64', hw_firmware_type='uefi', hw_machine_type='virt' | |
| 14:23:35 | sean-k-mooney | ya ok that is what i was expecting | |
| 14:23:43 | sean-k-mooney | thanks | |
| 14:24:03 | sean-k-mooney | i can give it a try again i pulled the image form github so maybe there is a delta | |
| 14:24:31 | sean-k-mooney | from here https://github.com/cirros-dev/cirros/releases/tag/0.5.2 | |
| 14:24:33 | chateaulav | the ubuntu one is kinda weird because of how they setup the video aspect but i havent had issues with other vendors. it still builds but i only ever have ssh | |
| 14:25:01 | dmitriis | sean-k-mooney: RE https://review.opendev.org/c/openstack/nova/+/824833/7/nova/network/neutron.py#1536 shouldn't we always be able to get a VF num for a valid VF PCI address? This would only raise if a device somehow got unbound just before this code ran and a symlink to physfn of a VF is not present to retrieve the VF num OR if the PCI address is | |
| 14:25:01 | dmitriis | not valid in the first place. Happy to remove this check and let operators to figure it out in case it happens but just curious. | |
| 14:25:55 | sean-k-mooney | dmitriis: im not sure if all vendors always export the symilink to work that out | |
| 14:26:34 | sean-k-mooney | i would expect that the vf number should generally be avaialble but rather then raise in this funciton i think we shoudl just not include the info if not avaiable | |
| 14:26:41 | sean-k-mooney | as a general pattern | |
| 14:28:28 | opendevreview | Elod Illes proposed openstack/nova stable/wallaby: workarounds: Add libvirt_disable_apic https://review.opendev.org/c/openstack/nova/+/805628 | |
| 14:28:45 | sean-k-mooney | dmitriis: i say perfer as im open to being conviced otherwise but in general i woudl prefer not to reate retiving info as an error. its true it might mean that device cannot be remote managed but in that case the operator has incorrectly tagged it. on the neutron side the ml2/driver can check the profile and fail the binding if the info is not present tha it needs | |
| 14:32:36 | dmitriis | sean-k-mooney: I think it's the generic SR-IOV handling code in the kernel that creates the sysfs entry | |
| 14:32:36 | dmitriis | https://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L144-L147 | |
| 14:32:36 | dmitriis | https://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L295 | |
| 14:32:36 | dmitriis | In this case, the case where the info won't be retrievable will likely never happen so I have no issue in making it optional. Just trying to reason about what to put into a comment there. | |
| 14:34:04 | sean-k-mooney | just say somthign like. this should be created by default on all modeern kernels but we make it optional to cater for expotic hardware or older kernel where this may not be ture | |
| 14:34:40 | dmitriis | sean-k-mooney: ack, will do | |
| 14:35:55 | sean-k-mooney | dmitriis looking at 4.14 for example im not sure it does the same https://github.com/torvalds/linux/blob/v4.14/drivers/pci/iov.c#L165-L176 | |
| 14:35:59 | sean-k-mooney | well | |
| 14:36:14 | sean-k-mooney | it has the physfn symlink | |
| 14:36:25 | sean-k-mooney | i guess it has the virtfn symlink too | |
| 14:36:46 | sean-k-mooney | so i guess it would not break on centos 8 | |
| 14:37:17 | sean-k-mooney | dmitriis: i know mellonox/nvida changed where the representor netdevs were create for example in a relitivly recent kernel | |
| 14:37:46 | sean-k-mooney | https://github.com/openstack/os-vif/commit/b37de19c58c877f5174d76d0a4ba5ab519f464e8#diff-087288ddacb7516a4762ef38c4361754c2d6bb07a52952b06aa6f607dc3811fe | |
| 14:38:02 | sean-k-mooney | 5.8 https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1 | |
| 14:39:05 | sean-k-mooney | dmitriis: im trying to avoid failure in nova when thing like that ar changed | |
| 14:39:54 | dmitriis | sean-k-mooney: yeah, probably worth one VM failing to boot than the whole daemon dying because of the exception | |
| 14:39:56 | dmitriis | totally agree | |
| 14:40:50 | sean-k-mooney | yep plus as i said in the neutron ml2 driver you can fail the port binding an log an explit error message if the required info is not present | |
| 14:41:18 | dmitriis | sean-k-mooney: also very true | |
| 14:41:25 | sean-k-mooney | you could proably also add a check in nova at a differnt point if you wanted before that but i woudl just log the error form neutron personlaly | |
| 14:42:18 | dmitriis | sean-k-mooney: I'll log a warning in Nova in this case and simply return no data in the profile and let Neutron fail to bind | |
| 14:42:38 | sean-k-mooney | well you shoudl not return an empty profile | |
| 14:42:52 | sean-k-mooney | this code will be used for other VFs that may not be remote managed | |
| 14:43:04 | sean-k-mooney | and tehy do not need the vf number | |
| 14:43:11 | sean-k-mooney | we pass the vf pci adress | |
| 14:43:20 | sean-k-mooney | that is all they need in the general case | |
| 14:43:38 | dmitriis | rephrasing: 1. if PF MAC cannot be retrieved, do not include vf_num or card_serial_number either | |
| 14:43:56 | dmitriis | 2. if vf_num cannot be retrieved, do not include pf_mac and card_serial_number | |
| 14:44:24 | sean-k-mooney | ah well ya perhaps | |
| 14:44:39 | sean-k-mooney | i was thinking just include all the info you can always but i guess that works too | |
| 14:45:35 | dmitriis | ok, just need to fix the tests a little and I'll re-upload | |
| 14:45:45 | sean-k-mooney | ack | |
| 14:46:57 | sean-k-mooney | dmitriis: by the way have you tested this feature with a second ovs on the compute host | |
| 14:47:27 | sean-k-mooney | we had some internal question regarding can we have both offpath port and on path port on the same host | |
| 14:48:09 | sean-k-mooney | the reason for that was to use vm management port with vnic_type=normal and datapalne prots with vnic_type=smartnic | |
| 14:48:20 | dmitriis | sean-k-mooney: i.e. having a DPU and a regular SR-IOV card | |
| 14:48:37 | sean-k-mooney | well dpu and another nic that can be used with ovs or linux bridge | |
| 14:48:41 | dmitriis | right | |
| 14:49:03 | dmitriis | I haven't tested that explicitly but just trying to think what would that entail | |
| 14:49:07 | sean-k-mooney | basically it was raised that by using the dpu you are limiting the numer or port that can be created to the number of vfs | |
| 14:49:22 | sean-k-mooney | dmitriis: i think it should just work honestly | |
| 14:49:32 | sean-k-mooney | i was just wondering if you had considerd it | |
| 14:49:50 | dmitriis | sean-k-mooney: yes, the limitation on the number of ports have definitely come up in our conversations | |
| 14:50:11 | sean-k-mooney | dmitriis: all it really requires is the ml2 dirver changes to not require you to designate a host as only dpu enabled | |
| 14:50:18 | dmitriis | and nothing immediately jumps into mind regarding OVS on the host being a problem | |
| 14:50:23 | sean-k-mooney | e.g. make the binding desiion per port based on vnic_type | |
| 14:50:54 | sean-k-mooney | ack that is what my assement was too | |
| 14:51:04 | dmitriis | sean-k-mooney: yeah and IIRC it's just based on the vnic_type, we just added handling for vnic_type smartnic to the OVN mechanism driver | |
| 14:51:14 | dmitriis | so it should just bind a normal port regularly | |
| 14:51:20 | sean-k-mooney | yep | |
| 14:51:23 | sean-k-mooney | cool | |
| 14:51:44 | sean-k-mooney | we are still trying to assess what woudl be requried to eventually support this in our product | |
| 14:52:04 | dmitriis | sean-k-mooney: yeah, plus hardware is hard to come by | |
| 14:52:12 | sean-k-mooney | that too | |
| 14:52:45 | dmitriis | I think this feature will be useful for FPGA-based SmartNICs as well. I've seen the ones with a separate CPU but with an FPGA instead of an ASIC | |
| 14:53:24 | dmitriis | considering the DPDK dataplane usage at the SmartNIC's CPU side for that with rte_flow for offload | |
| 14:53:26 | sean-k-mooney | i do potentially have access to a BF2 that i coudl use but only 1 so i obviouly cant test multi node and in general an all in one deployment while useful is not fully reflective of how it would be deployed in a datacenter | |
| 14:53:34 | dmitriis | the Nova and Neutron bits would work the same though | |
| 14:53:46 | sean-k-mooney | dmitriis: yes in princial it could be | |
| 14:53:54 | dmitriis | sean-k-mooney: I have an idea on how to test multi-node with just one BF2 | |
| 14:54:15 | sean-k-mooney | the only fpga based smartnic i have use used armstong os with only 2 arm cores and i think 4GB of ram | |
| 14:54:44 | sean-k-mooney | dmitriis: run two copies of nova-compute with difernt host values in teh conf on the same host | |
| 14:54:53 | sean-k-mooney | and use differnt ports on the BF2? | |
| 14:54:53 | dmitriis | sean-k-mooney: 2 VMs with nested virt a the hypervisor, plus 2 VMs at the DPU side. Each VM uses 1 PF. VPD can be faked by bind mounting a file in the right sysfs location | |
| 14:55:08 | dmitriis | BF2's ARM CPU is virt-capable btw xD | |
| 14:55:43 | sean-k-mooney | dmitriis: that proably wotn work becaue wehn the PF is passed to the qemu instacle the pcie capablityies are not in the pci config space in teh guest | |
| 14:56:10 | sean-k-mooney | lspci in the guest will only print the pci capablities not the extened pcie cabalities like sriov | |
| 14:56:15 | sean-k-mooney | in my previous experince | |
| 14:56:20 | sean-k-mooney | even when using q35 | |
| 14:56:28 | dmitriis | sean-k-mooney: I was about to ask about q35 :^) | |
| 14:56:30 | sean-k-mooney | if it does work let me know :) | |
| 14:56:37 | dmitriis | sean-k-mooney: there's another option | |
| 14:56:49 | dmitriis | BF2 on one host + ConnectX on another | |