| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-07 | |||
| 13:51:28 | dmitriis | sean-k-mooney: Thinking about it now, I assumed that since we are in the neutron-related code path and got a PCI device allocated which is a netdev VF, I am safe to assume the PF is also a netdev. However, this may not be true all the time. | |
| 13:51:32 | dmitriis | sean-k-mooney: ack | |
| 13:52:07 | sean-k-mooney | dmitriis: ya that was the assumtion that bit us for the calvim thunderx | |
| 13:52:13 | sean-k-mooney | i even hat that in a comment | |
| 13:52:26 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/777679/3/nova/virt/libvirt/host.py | |
| 13:52:47 | sean-k-mooney | so no we cant assume that | |
| 13:53:08 | dmitriis | sean-k-mooney: yes, I recall fixing something for that comment but I definitely missed this part | |
| 13:54:39 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/805649 | |
| 14:05:28 | gibi | melwitt: fyi the placement perfload job (non voting) probably broken since the consumer types feature https://zuul.opendev.org/t/openstack/build/0d0e850ea9de47e8bf30ad1dbd1d3b91/log/job-output.txt#1163 | |
| 14:05:47 | gibi | melwitt: I think we don't really look at that job so I'm not sure we want to fix it | |
| 14:08:01 | chateaulav | gibi: for the hanging comment about the tempest testing, you mean in addition to an additional ci job, correct? | |
| 14:08:21 | chateaulav | for https://review.opendev.org/c/openstack/nova/+/822053, and thanks for the continued reviews! | |
| 14:08:57 | gibi | chateaulav: yeah so I think a separate job that test some emulation (maybe aarch64) with tempest. But I guess some of the existing tempest test will not work with emulation so we need a trimmed down test list | |
| 14:11:21 | chateaulav | ok, makes sense for the most part. first time messing with tempest, but ill get something together | |
| 14:13:27 | gibi | chateaulav: thanks | |
| 14:17:43 | sean-k-mooney | gibi: chateaulav by the way have either of you got a working vm image for this testing. i think the centos9s cloud image will booth properly on arm (it did on my mac nativly) but i could not get cirros to work | |
| 14:17:56 | sean-k-mooney | the centos image is huge in comparison at almost 700mb | |
| 14:18:03 | gibi | sean-k-mooney: I did not tired | |
| 14:18:48 | sean-k-mooney | ok well that might be one of the chanllages with doing the arm testing via emulation but it is likely fine | |
| 14:19:00 | sean-k-mooney | we can have devstack download addtional images | |
| 14:19:24 | sean-k-mooney | but i was having trouble geting cirrus to boot with uefi in genreal and that is required for aarch64 | |
| 14:19:44 | sean-k-mooney | so uefi on x86 and aarch64 did not seam to work with cirros | |
| 14:20:14 | sean-k-mooney | i got an error with the alingment or either the filesystme or uefi firmware in the console and then no output | |
| 14:20:29 | sean-k-mooney | chateaulav: what have you been using to test with qemu directly? | |
| 14:20:48 | chateaulav | sean-k-mooney: yeah havent tried the cirros on aarch hardware, but on x86 works fine. | |
| 14:20:59 | chateaulav | have code running in a deployment ostack env | |
| 14:21:00 | sean-k-mooney | did you do an install form iso or something else | |
| 14:21:37 | sean-k-mooney | chateaulav: ok so you did arch64 via qemu on x86 with the aarch64 cirros image | |
| 14:21:44 | chateaulav | sean-k-mooney: qcow | |
| 14:21:44 | chateaulav | http://download.cirros-cloud.net/0.5.2/cirros-0.5.2-aarch64-disk.img | |
| 14:21:57 | chateaulav | yes | |
| 14:22:19 | sean-k-mooney | ya i tried that and it woudl not boot on may ubuntu aarch64 vm on my macbook air | |
| 14:22:38 | sean-k-mooney | i was trying to see if i could use that as a arm dev env | |
| 14:23:03 | sean-k-mooney | chateaulav: did you put anything special in the glacne metadtaa | |
| 14:23:19 | chateaulav | hw_emulation_architecture='aarch64', hw_firmware_type='uefi', hw_machine_type='virt' | |
| 14:23:35 | sean-k-mooney | ya ok that is what i was expecting | |
| 14:23:43 | sean-k-mooney | thanks | |
| 14:24:03 | sean-k-mooney | i can give it a try again i pulled the image form github so maybe there is a delta | |
| 14:24:31 | sean-k-mooney | from here https://github.com/cirros-dev/cirros/releases/tag/0.5.2 | |
| 14:24:33 | chateaulav | the ubuntu one is kinda weird because of how they setup the video aspect but i havent had issues with other vendors. it still builds but i only ever have ssh | |
| 14:25:01 | dmitriis | not valid in the first place. Happy to remove this check and let operators to figure it out in case it happens but just curious. | |
| 14:25:01 | dmitriis | sean-k-mooney: RE https://review.opendev.org/c/openstack/nova/+/824833/7/nova/network/neutron.py#1536 shouldn't we always be able to get a VF num for a valid VF PCI address? This would only raise if a device somehow got unbound just before this code ran and a symlink to physfn of a VF is not present to retrieve the VF num OR if the PCI address is | |
| 14:25:55 | sean-k-mooney | dmitriis: im not sure if all vendors always export the symilink to work that out | |
| 14:26:34 | sean-k-mooney | i would expect that the vf number should generally be avaialble but rather then raise in this funciton i think we shoudl just not include the info if not avaiable | |
| 14:26:41 | sean-k-mooney | as a general pattern | |
| 14:28:28 | opendevreview | Elod Illes proposed openstack/nova stable/wallaby: workarounds: Add libvirt_disable_apic https://review.opendev.org/c/openstack/nova/+/805628 | |
| 14:28:45 | sean-k-mooney | dmitriis: i say perfer as im open to being conviced otherwise but in general i woudl prefer not to reate retiving info as an error. its true it might mean that device cannot be remote managed but in that case the operator has incorrectly tagged it. on the neutron side the ml2/driver can check the profile and fail the binding if the info is not present tha it needs | |
| 14:32:36 | dmitriis | In this case, the case where the info won't be retrievable will likely never happen so I have no issue in making it optional. Just trying to reason about what to put into a comment there. | |
| 14:32:36 | dmitriis | https://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L295 | |
| 14:32:36 | dmitriis | https://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L144-L147 | |
| 14:32:36 | dmitriis | sean-k-mooney: I think it's the generic SR-IOV handling code in the kernel that creates the sysfs entry | |
| 14:34:04 | sean-k-mooney | just say somthign like. this should be created by default on all modeern kernels but we make it optional to cater for expotic hardware or older kernel where this may not be ture | |
| 14:34:40 | dmitriis | sean-k-mooney: ack, will do | |
| 14:35:55 | sean-k-mooney | dmitriis looking at 4.14 for example im not sure it does the same https://github.com/torvalds/linux/blob/v4.14/drivers/pci/iov.c#L165-L176 | |
| 14:35:59 | sean-k-mooney | well | |
| 14:36:14 | sean-k-mooney | it has the physfn symlink | |
| 14:36:25 | sean-k-mooney | i guess it has the virtfn symlink too | |
| 14:36:46 | sean-k-mooney | so i guess it would not break on centos 8 | |
| 14:37:17 | sean-k-mooney | dmitriis: i know mellonox/nvida changed where the representor netdevs were create for example in a relitivly recent kernel | |
| 14:37:46 | sean-k-mooney | https://github.com/openstack/os-vif/commit/b37de19c58c877f5174d76d0a4ba5ab519f464e8#diff-087288ddacb7516a4762ef38c4361754c2d6bb07a52952b06aa6f607dc3811fe | |
| 14:38:02 | sean-k-mooney | 5.8 https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1 | |
| 14:39:05 | sean-k-mooney | dmitriis: im trying to avoid failure in nova when thing like that ar changed | |
| 14:39:54 | dmitriis | sean-k-mooney: yeah, probably worth one VM failing to boot than the whole daemon dying because of the exception | |
| 14:39:56 | dmitriis | totally agree | |
| 14:40:50 | sean-k-mooney | yep plus as i said in the neutron ml2 driver you can fail the port binding an log an explit error message if the required info is not present | |
| 14:41:18 | dmitriis | sean-k-mooney: also very true | |
| 14:41:25 | sean-k-mooney | you could proably also add a check in nova at a differnt point if you wanted before that but i woudl just log the error form neutron personlaly | |
| 14:42:18 | dmitriis | sean-k-mooney: I'll log a warning in Nova in this case and simply return no data in the profile and let Neutron fail to bind | |
| 14:42:38 | sean-k-mooney | well you shoudl not return an empty profile | |
| 14:42:52 | sean-k-mooney | this code will be used for other VFs that may not be remote managed | |
| 14:43:04 | sean-k-mooney | and tehy do not need the vf number | |
| 14:43:11 | sean-k-mooney | we pass the vf pci adress | |
| 14:43:20 | sean-k-mooney | that is all they need in the general case | |
| 14:43:38 | dmitriis | rephrasing: 1. if PF MAC cannot be retrieved, do not include vf_num or card_serial_number either | |
| 14:43:56 | dmitriis | 2. if vf_num cannot be retrieved, do not include pf_mac and card_serial_number | |
| 14:44:24 | sean-k-mooney | ah well ya perhaps | |
| 14:44:39 | sean-k-mooney | i was thinking just include all the info you can always but i guess that works too | |
| 14:45:35 | dmitriis | ok, just need to fix the tests a little and I'll re-upload | |
| 14:45:45 | sean-k-mooney | ack | |
| 14:46:57 | sean-k-mooney | dmitriis: by the way have you tested this feature with a second ovs on the compute host | |
| 14:47:27 | sean-k-mooney | we had some internal question regarding can we have both offpath port and on path port on the same host | |
| 14:48:09 | sean-k-mooney | the reason for that was to use vm management port with vnic_type=normal and datapalne prots with vnic_type=smartnic | |
| 14:48:20 | dmitriis | sean-k-mooney: i.e. having a DPU and a regular SR-IOV card | |
| 14:48:37 | sean-k-mooney | well dpu and another nic that can be used with ovs or linux bridge | |
| 14:48:41 | dmitriis | right | |
| 14:49:03 | dmitriis | I haven't tested that explicitly but just trying to think what would that entail | |
| 14:49:07 | sean-k-mooney | basically it was raised that by using the dpu you are limiting the numer or port that can be created to the number of vfs | |
| 14:49:22 | sean-k-mooney | dmitriis: i think it should just work honestly | |
| 14:49:32 | sean-k-mooney | i was just wondering if you had considerd it | |
| 14:49:50 | dmitriis | sean-k-mooney: yes, the limitation on the number of ports have definitely come up in our conversations | |
| 14:50:11 | sean-k-mooney | dmitriis: all it really requires is the ml2 dirver changes to not require you to designate a host as only dpu enabled | |
| 14:50:18 | dmitriis | and nothing immediately jumps into mind regarding OVS on the host being a problem | |
| 14:50:23 | sean-k-mooney | e.g. make the binding desiion per port based on vnic_type | |
| 14:50:54 | sean-k-mooney | ack that is what my assement was too | |
| 14:51:04 | dmitriis | sean-k-mooney: yeah and IIRC it's just based on the vnic_type, we just added handling for vnic_type smartnic to the OVN mechanism driver | |
| 14:51:14 | dmitriis | so it should just bind a normal port regularly | |
| 14:51:20 | sean-k-mooney | yep | |
| 14:51:23 | sean-k-mooney | cool | |
| 14:51:44 | sean-k-mooney | we are still trying to assess what woudl be requried to eventually support this in our product | |
| 14:52:04 | dmitriis | sean-k-mooney: yeah, plus hardware is hard to come by | |