| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-07 | |||
| 13:21:11 | bauzas | D+4 I mean | |
| 13:22:31 | gibi | yeah | |
| 13:22:39 | gibi | I heard similar stories | |
| 13:25:06 | bauzas | from what I read, this is specific to Omicron | |
| 13:25:20 | bauzas | for Delta, this worked | |
| 13:25:57 | bauzas | but now, symptoms arrive before the viral charge | |
| 13:26:07 | bauzas | or at the same time | |
| 13:26:41 | bauzas | so, previously, testing yourself at D+3 was verifying your viral charge | |
| 13:28:18 | bauzas | while now, your viral charge would only arrive around D+5 | |
| 13:28:31 | bauzas | the more people know, the better it will be | |
| 13:40:06 | sean-k-mooney | dmitriis: -1 for https://review.opendev.org/c/openstack/nova/+/824833/7/nova/network/neutron.py#1527 | |
| 13:40:39 | dmitriis | sean-k-mooney: looking | |
| 13:41:18 | sean-k-mooney | tl;dr we cant assume a PF has a netdev | |
| 13:41:37 | sean-k-mooney | so you cant assume it has a mac | |
| 13:42:06 | sean-k-mooney | your new fucntion will be called for all VFs not just remote managed ones | |
| 13:42:12 | sean-k-mooney | so you need to not raise | |
| 13:42:28 | sean-k-mooney | just dont include the info if it cant be recived | |
| 13:42:36 | sean-k-mooney | *retrived | |
| 13:43:45 | dmitriis | sean-k-mooney: yeah, a PF can be something else, much like a PF with a netdev can have non-netdev VFs | |
| 13:43:47 | dmitriis | good point | |
| 13:44:28 | sean-k-mooney | we broke this in the past | |
| 13:44:59 | sean-k-mooney | by always trying to get the netdev name for bandwith qos and i broke it differently in a different bugfix | |
| 13:45:07 | sean-k-mooney | so we have been bitten by it twice | |
| 13:46:40 | sean-k-mooney | dmitriis: actully just to reinforce that you cant assume all VFs are nics | |
| 13:46:58 | gibi | ohh, I knew that we cannot assume that netdev exists but I did not realized that to get the MAC we need the netdev | |
| 13:47:01 | sean-k-mooney | that is what we broke with the init qos support we broke vf for qat | |
| 13:47:45 | dmitriis | sean-k-mooney: yeah, Bluefield2 devs have a management VF, for example, which is not a netdev | |
| 13:47:48 | sean-k-mooney | i can triple check how we do the lookup but in general the mac wont be in /sys if the netdev is not created | |
| 13:50:07 | sean-k-mooney | gibi: we do the lookup like this https://github.com/openstack/nova/blob/master/nova/pci/utils.py#L173-L191 | |
| 13:50:20 | sean-k-mooney | which i woudl expect to fail sicne we are geting the netdev path | |
| 13:50:32 | gibi | yeah you are correct | |
| 13:51:01 | sean-k-mooney | dmitriis: anyway hopefully that will be a minor change jsut dont populate the filed if the data is not available | |
| 13:51:21 | sean-k-mooney | im going to continue reviewing the rest of the series in the mean time | |
| 13:51:28 | dmitriis | sean-k-mooney: Thinking about it now, I assumed that since we are in the neutron-related code path and got a PCI device allocated which is a netdev VF, I am safe to assume the PF is also a netdev. However, this may not be true all the time. | |
| 13:51:32 | dmitriis | sean-k-mooney: ack | |
| 13:52:07 | sean-k-mooney | dmitriis: ya that was the assumtion that bit us for the calvim thunderx | |
| 13:52:13 | sean-k-mooney | i even hat that in a comment | |
| 13:52:26 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/777679/3/nova/virt/libvirt/host.py | |
| 13:52:47 | sean-k-mooney | so no we cant assume that | |
| 13:53:08 | dmitriis | sean-k-mooney: yes, I recall fixing something for that comment but I definitely missed this part | |
| 13:54:39 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/805649 | |
| 14:05:28 | gibi | melwitt: fyi the placement perfload job (non voting) probably broken since the consumer types feature https://zuul.opendev.org/t/openstack/build/0d0e850ea9de47e8bf30ad1dbd1d3b91/log/job-output.txt#1163 | |
| 14:05:47 | gibi | melwitt: I think we don't really look at that job so I'm not sure we want to fix it | |
| 14:08:01 | chateaulav | gibi: for the hanging comment about the tempest testing, you mean in addition to an additional ci job, correct? | |
| 14:08:21 | chateaulav | for https://review.opendev.org/c/openstack/nova/+/822053, and thanks for the continued reviews! | |
| 14:08:57 | gibi | chateaulav: yeah so I think a separate job that test some emulation (maybe aarch64) with tempest. But I guess some of the existing tempest test will not work with emulation so we need a trimmed down test list | |
| 14:11:21 | chateaulav | ok, makes sense for the most part. first time messing with tempest, but ill get something together | |
| 14:13:27 | gibi | chateaulav: thanks | |
| 14:17:43 | sean-k-mooney | gibi: chateaulav by the way have either of you got a working vm image for this testing. i think the centos9s cloud image will booth properly on arm (it did on my mac nativly) but i could not get cirros to work | |
| 14:17:56 | sean-k-mooney | the centos image is huge in comparison at almost 700mb | |
| 14:18:03 | gibi | sean-k-mooney: I did not tired | |
| 14:18:48 | sean-k-mooney | ok well that might be one of the chanllages with doing the arm testing via emulation but it is likely fine | |
| 14:19:00 | sean-k-mooney | we can have devstack download addtional images | |
| 14:19:24 | sean-k-mooney | but i was having trouble geting cirrus to boot with uefi in genreal and that is required for aarch64 | |
| 14:19:44 | sean-k-mooney | so uefi on x86 and aarch64 did not seam to work with cirros | |
| 14:20:14 | sean-k-mooney | i got an error with the alingment or either the filesystme or uefi firmware in the console and then no output | |
| 14:20:29 | sean-k-mooney | chateaulav: what have you been using to test with qemu directly? | |
| 14:20:48 | chateaulav | sean-k-mooney: yeah havent tried the cirros on aarch hardware, but on x86 works fine. | |
| 14:20:59 | chateaulav | have code running in a deployment ostack env | |
| 14:21:00 | sean-k-mooney | did you do an install form iso or something else | |
| 14:21:37 | sean-k-mooney | chateaulav: ok so you did arch64 via qemu on x86 with the aarch64 cirros image | |
| 14:21:44 | chateaulav | sean-k-mooney: qcow | |
| 14:21:44 | chateaulav | http://download.cirros-cloud.net/0.5.2/cirros-0.5.2-aarch64-disk.img | |
| 14:21:57 | chateaulav | yes | |
| 14:22:19 | sean-k-mooney | ya i tried that and it woudl not boot on may ubuntu aarch64 vm on my macbook air | |
| 14:22:38 | sean-k-mooney | i was trying to see if i could use that as a arm dev env | |
| 14:23:03 | sean-k-mooney | chateaulav: did you put anything special in the glacne metadtaa | |
| 14:23:19 | chateaulav | hw_emulation_architecture='aarch64', hw_firmware_type='uefi', hw_machine_type='virt' | |
| 14:23:35 | sean-k-mooney | ya ok that is what i was expecting | |
| 14:23:43 | sean-k-mooney | thanks | |
| 14:24:03 | sean-k-mooney | i can give it a try again i pulled the image form github so maybe there is a delta | |
| 14:24:31 | sean-k-mooney | from here https://github.com/cirros-dev/cirros/releases/tag/0.5.2 | |
| 14:24:33 | chateaulav | the ubuntu one is kinda weird because of how they setup the video aspect but i havent had issues with other vendors. it still builds but i only ever have ssh | |
| 14:25:01 | dmitriis | not valid in the first place. Happy to remove this check and let operators to figure it out in case it happens but just curious. | |
| 14:25:01 | dmitriis | sean-k-mooney: RE https://review.opendev.org/c/openstack/nova/+/824833/7/nova/network/neutron.py#1536 shouldn't we always be able to get a VF num for a valid VF PCI address? This would only raise if a device somehow got unbound just before this code ran and a symlink to physfn of a VF is not present to retrieve the VF num OR if the PCI address is | |
| 14:25:55 | sean-k-mooney | dmitriis: im not sure if all vendors always export the symilink to work that out | |
| 14:26:34 | sean-k-mooney | i would expect that the vf number should generally be avaialble but rather then raise in this funciton i think we shoudl just not include the info if not avaiable | |
| 14:26:41 | sean-k-mooney | as a general pattern | |
| 14:28:28 | opendevreview | Elod Illes proposed openstack/nova stable/wallaby: workarounds: Add libvirt_disable_apic https://review.opendev.org/c/openstack/nova/+/805628 | |
| 14:28:45 | sean-k-mooney | dmitriis: i say perfer as im open to being conviced otherwise but in general i woudl prefer not to reate retiving info as an error. its true it might mean that device cannot be remote managed but in that case the operator has incorrectly tagged it. on the neutron side the ml2/driver can check the profile and fail the binding if the info is not present tha it needs | |
| 14:32:36 | dmitriis | In this case, the case where the info won't be retrievable will likely never happen so I have no issue in making it optional. Just trying to reason about what to put into a comment there. | |
| 14:32:36 | dmitriis | https://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L295 | |
| 14:32:36 | dmitriis | https://github.com/torvalds/linux/blob/v5.16/drivers/pci/iov.c#L144-L147 | |
| 14:32:36 | dmitriis | sean-k-mooney: I think it's the generic SR-IOV handling code in the kernel that creates the sysfs entry | |
| 14:34:04 | sean-k-mooney | just say somthign like. this should be created by default on all modeern kernels but we make it optional to cater for expotic hardware or older kernel where this may not be ture | |
| 14:34:40 | dmitriis | sean-k-mooney: ack, will do | |
| 14:35:55 | sean-k-mooney | dmitriis looking at 4.14 for example im not sure it does the same https://github.com/torvalds/linux/blob/v4.14/drivers/pci/iov.c#L165-L176 | |
| 14:35:59 | sean-k-mooney | well | |
| 14:36:14 | sean-k-mooney | it has the physfn symlink | |
| 14:36:25 | sean-k-mooney | i guess it has the virtfn symlink too | |
| 14:36:46 | sean-k-mooney | so i guess it would not break on centos 8 | |
| 14:37:17 | sean-k-mooney | dmitriis: i know mellonox/nvida changed where the representor netdevs were create for example in a relitivly recent kernel | |
| 14:37:46 | sean-k-mooney | https://github.com/openstack/os-vif/commit/b37de19c58c877f5174d76d0a4ba5ab519f464e8#diff-087288ddacb7516a4762ef38c4361754c2d6bb07a52952b06aa6f607dc3811fe | |
| 14:38:02 | sean-k-mooney | 5.8 https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1 | |
| 14:39:05 | sean-k-mooney | dmitriis: im trying to avoid failure in nova when thing like that ar changed | |
| 14:39:54 | dmitriis | sean-k-mooney: yeah, probably worth one VM failing to boot than the whole daemon dying because of the exception | |
| 14:39:56 | dmitriis | totally agree | |
| 14:40:50 | sean-k-mooney | yep plus as i said in the neutron ml2 driver you can fail the port binding an log an explit error message if the required info is not present | |
| 14:41:18 | dmitriis | sean-k-mooney: also very true | |
| 14:41:25 | sean-k-mooney | you could proably also add a check in nova at a differnt point if you wanted before that but i woudl just log the error form neutron personlaly | |