| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-03 | |||
| 14:39:02 | sean-k-mooney | then we will not filter on any mdev type or trait and will just select host with free VGPU inventory | |
| 14:39:15 | sean-k-mooney | which is fine if all your gpus are the same | |
| 14:39:37 | sean-k-mooney | if you have differnt ones configured you shoudl use traits or the new generic mdev feature instead ot differenciate | |
| 14:40:01 | sean-k-mooney | tobias-urdin: is there anything in particalr that you wanted to know beyond that overview | |
| 14:41:21 | tobias-urdin | CUSTOM_NVIDIA_2 traits, flavors with resources:VGPU=1 and trait:CUSTOM_NVIDIA_1=required etc | |
| 14:41:21 | tobias-urdin | so let's say i have two cards 0000:3b:00.0 and 0000:af:00.0 that is NVIDIA A10 cards, I enable VFs on those and get like 30 each let's say on 0000:3b:x.x 0000:af:x.x and assign enabled_mdev_types=nvidia-1, nvidia-2 and [mdev_nvidia-1]/device_addresses=<all VF for 0000:3b:x.x> (and same for 0000:af card) - assign those CUSTOM_NVIDIA_1 and | |
| 14:42:11 | tobias-urdin | how is placement populated for all those <computenode>_pci_0000_3b_x_x RPs to know how many it is left when it a 1:1 mapping in that RP per PCI dev addr | |
| 14:51:15 | ade_lee_ | sean-k-mooney, hey -- can you take a look at the failing experimental fips job on https://review.opendev.org/c/openstack/tempest/+/831607 | |
| 14:51:35 | ade_lee_ | sean-k-mooney, this time its on centos-9 | |
| 14:52:11 | ade_lee_ | sean-k-mooney, dansmith : https://zuul.opendev.org/t/openstack/build/0a4f8346b89f4bbfa92135dbdbf811f9 | |
| 14:52:45 | sean-k-mooney | so rescue and temp url | |
| 14:54:00 | ade_lee_ | sean-k-mooney, ack - what does that point to? | |
| 14:54:52 | dansmith | check the cinder logs? | |
| 14:55:01 | sean-k-mooney | rescure failde because fo this | |
| 14:55:18 | sean-k-mooney | Waiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance 7323f68a-b4dc-4630-b4fd-bd7a7f69d4f4 is timed out. | |
| 14:55:36 | sean-k-mooney | so that looks like the intermiting libvirt volume detach issue | |
| 14:56:07 | sean-k-mooney | ya it is internal error: unable to execute QEMU command 'device_del': Device virtio-disk1 is already in the process of unplug. | |
| 14:56:48 | sean-k-mooney | that will be fixed by a new qemu soon we hope | |
| 14:56:57 | dansmith | seems like a lot of these centos job fails are qemu/libvirt related | |
| 14:57:01 | dansmith | which is pretty disturbing :/ | |
| 14:57:44 | sean-k-mooney | if it makes you feel better those also fail on rhel downstream | |
| 14:57:54 | dansmith | not really :) | |
| 14:58:04 | gibi | the centos jobs are running with newer libvirt and qemu than the ubunut jobs so we see the new failure modes there first | |
| 14:58:29 | sean-k-mooney | currntly yes although it prevoulsy was the other way around | |
| 14:59:18 | sean-k-mooney | gibi: dansmith what is more disturbing to me is this is using the pc machine type | |
| 14:59:22 | sean-k-mooney | not q35 | |
| 14:59:24 | dansmith | hmm | |
| 15:00:26 | sean-k-mooney | gibi: so this is partly realted to the fact that even with the event based case we still retry | |
| 15:00:51 | sean-k-mooney | but fundimetnally qemu is taking a long time to detach | |
| 15:01:08 | sean-k-mooney | which it shoudl not, the wait for sshable/pingable tempest change might help | |
| 15:01:16 | sean-k-mooney | if this is because the os is not ready | |
| 15:01:47 | sean-k-mooney | but this happeing a lot lately | |
| 15:02:16 | sean-k-mooney | what do the ObjectTempUrlTest test do | |
| 15:02:25 | sean-k-mooney | are they swift related im not familar with them | |
| 15:03:38 | sean-k-mooney | GET https://149.202.163.165:8080/v1/AUTH_ab7063290b7341eeb77f5198d9e09903/tempest-TestContainer-735377846/tempest-TestObject-1768297629 | |
| 15:03:47 | sean-k-mooney | that looks like possible swift to me | |
| 15:05:34 | sean-k-mooney | ade_lee_: in anycase the rescue failure does not look fips related | |
| 15:07:03 | tobias-urdin | sean-k-mooney: any input on above? | |
| 15:09:03 | ade_lee_ | sean-k-mooney, ack - I didn't think it was, but unfortunately , it is blocking the fips patches. Do we have any workarounds/possible fixes ? I'll ask the swift folks about the swift issues. | |
| 15:10:10 | ade_lee_ | sean-k-mooney, is there a BZ /launchpad to track these libvirt/qemu issues? | |
| 15:10:37 | gibi | sean-k-mooney: yeah, the base case retry is something we can remove when we switch to qemu 6.2 as a minimum | |
| 15:10:49 | gibi | (or something around 6.2 Im not sure) | |
| 15:15:26 | sean-k-mooney | we proably could make it conditonal on the version before we raise our minium | |
| 15:15:36 | gibi | yeah, good point | |
| 15:15:57 | sean-k-mooney | tobias-urdin: oh i missed your follow ups | |
| 15:16:02 | sean-k-mooney | ill read back one sec | |
| 15:16:18 | sean-k-mooney | ade_lee_: why is it blocking? | |
| 15:16:32 | sean-k-mooney | the job is non voting right | |
| 15:17:09 | sean-k-mooney | i think we could proceed with this failure unless it reliably fails every time in the fips job? | |
| 15:18:52 | ade_lee_ | sean-k-mooney, fair enough -- maybe what we do then is change the job to be for centos-9 - and then merge it | |
| 15:19:22 | sean-k-mooney | basically what i woudl hope is when we fix this normally it would be fixed for fips | |
| 15:26:28 | ade_lee_ | sean-k-mooney, ack - I'll update to centos 9 - and then ping for reviews. do we have any sense of when it will be fixed ? its showing up in glance reviews, cinder etc .. | |
| 15:26:56 | ade_lee_ | sean-k-mooney, a BZ will be super helpful so I can track things | |
| 15:30:09 | sean-k-mooney | so we kind fo do have one for qemu and there are a few cix issues | |
| 15:30:29 | sean-k-mooney | we dont have a singel one for nova for example because its not really a nova issue | |
| 15:30:45 | sean-k-mooney | we think that some of the recent bugfixes in qemu and libvirt will help | |
| 15:33:26 | opendevreview | ribaudr proposed openstack/python-novaclient master: Microversion 2.91: Support specifying destination host to unshelve https://review.opendev.org/c/openstack/python-novaclient/+/831651 | |
| 16:19:07 | tobias-urdin | sean-k-mooney: no hurry, let me know when you have a second :) | |
| 16:19:44 | sean-k-mooney | tobias-urdin: sorry im in a meeting downstream which is why i did not respond | |
| 16:34:21 | opendevreview | Merged openstack/python-novaclient stable/yoga: Update .gitreview for stable/yoga https://review.opendev.org/c/openstack/python-novaclient/+/831704 | |
| 16:34:24 | opendevreview | Merged openstack/python-novaclient stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga https://review.opendev.org/c/openstack/python-novaclient/+/831705 | |
| 17:28:04 | Uggla | sean-k-mooney, fyi now the unshelve to host is complete with client part as well. | |
| 17:56:30 | opendevreview | Merged openstack/osc-placement stable/yoga: Update .gitreview for stable/yoga https://review.opendev.org/c/openstack/osc-placement/+/831695 | |
| 17:57:28 | opendevreview | Merged openstack/osc-placement stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga https://review.opendev.org/c/openstack/osc-placement/+/831698 | |
| 18:12:17 | sean-k-mooney | Uggla: ack ill try and review what you have proably monday | |
| 18:33:57 | Uggla | sean-k-mooney, no hurries I think. | |
| 18:51:08 | sean-k-mooney | are all patches in https://review.opendev.org/q/topic:bp%252Funshelve-to-host | |
| 18:51:37 | sean-k-mooney | if so ill add that to my review-list bookmark folder for monday | |
| 20:13:55 | opendevreview | Ade Lee proposed openstack/nova master: Test setting the nova job to centos-9-stream https://review.opendev.org/c/openstack/nova/+/831844 | |
| #openstack-nova - 2022-03-04 | |||
| 03:46:01 | opendevreview | melanie witt proposed openstack/nova master: Follow up for unified limits https://review.opendev.org/c/openstack/nova/+/830918 | |
| 03:46:01 | opendevreview | melanie witt proposed openstack/nova master: Enable unified limits in the nova-next job https://review.opendev.org/c/openstack/nova/+/789963 | |
| 08:40:04 | Uggla | sean-k-mooney, yep all patches (specs, api, novaclient, openstackclient) are available under this topic : https://review.opendev.org/q/topic:bp%252Funshelve-to-host | |
| 09:50:49 | opendevreview | Andre Aranha proposed openstack/nova master: Change test fips job from centos8 to centos-9 https://review.opendev.org/c/openstack/nova/+/831917 | |
| 11:01:40 | gibi | sean-k-mooney: hi! ralonsoh left +2 on the neutron PF MAC patch https://review.opendev.org/c/openstack/neutron/+/829247/2 So it seem neutron is OK not to persist the MAC in the port.mac_address just in the binding profile. You had comments about the test case in that patch. Do you still feel that is needed? | |
| 11:02:20 | ralonsoh | I've added this patch to the Neutron drivers meeting today | |
| 11:02:32 | ralonsoh | (to the meeting agenda) | |
| 11:02:56 | gibi | ralonsoh: ack, I will join to the meeting | |
| 11:17:30 | sean-k-mooney | gibi: my comments were just about showing that the profile mac was what was actully being used | |
| 11:17:43 | sean-k-mooney | gibi so those are not needed if we are ok with that | |
| 11:17:51 | gibi | OK, thanks | |
| 11:18:17 | sean-k-mooney | i.e. the change would have demonstrated that the mac was not persisted to the port.mac | |
| 11:18:35 | gibi | hm, then it still make sense to have that test case | |
| 11:18:47 | gibi | to show that we are still not persisting | |
| 11:18:54 | gibi | it in the port.mac_address | |
| 11:19:05 | gibi | I will create a follow up then | |
| 11:19:21 | gibi | and depending on the driver's meeting outcome I can squash it | |
| 12:18:33 | sean-k-mooney | gibi: so that deprecation warning is actully already fixed in secretstorage | |
| 12:18:46 | sean-k-mooney | but i had to manually install the latest version | |
| 12:18:51 | sean-k-mooney | its fixed in 3.3.1 | |
| 12:19:02 | sean-k-mooney | but for some reason 2.3.1 was insalled by devstack | |
| 12:20:05 | sean-k-mooney | upperconstaits allwos 3.3.1 so not sure why this was not installed | |
| 12:26:34 | gibi | interesting | |
| 12:28:32 | gibi | I also see 2.3.1 in my devstack too | |
| 12:32:02 | gibi | devstack stack.sh log does not explain where this is coming from | |
| 12:36:39 | sean-k-mooney | it must be a downgrade form say keyring or something else | |
| 12:37:35 | sean-k-mooney | actully it requires >=3.2 | |
| 12:37:37 | sean-k-mooney | https://github.com/jaraco/keyring/blob/main/setup.cfg#L24 | |
| 12:38:25 | sean-k-mooney | i have a really old verison fo keyring 18.0.1 | |
| 12:38:45 | sean-k-mooney | which pulls in <3 | |
| 12:38:47 | sean-k-mooney | https://github.com/jaraco/keyring/blob/18.0.1/setup.cfg#L35 | |