| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-23 | |||
| 16:45:46 | sean-k-mooney | so we could fix this by pinning setuptools on stable until they move to unmaintainted | |
| 16:45:54 | sean-k-mooney | welll or after i guess | |
| 16:46:04 | sean-k-mooney | specificlay 2023.1 and older | |
| 16:46:25 | clarkb | no you cannot pin setuptools | |
| 16:46:30 | sean-k-mooney | no ? | |
| 16:46:42 | sean-k-mooney | i know we cant do it in requrieemnts | |
| 16:46:54 | clarkb | setuptools must be installed before your package installations occur. It is a chicken and egg problem unless you use pyproject.toml universally (which we do not) | |
| 16:46:58 | sean-k-mooney | but cant we do it explcitly in the jobs to <70.0.0 | |
| 16:47:05 | clarkb | no | |
| 16:47:17 | sean-k-mooney | ok cause im pretty sure we pinned in in the past | |
| 16:47:20 | clarkb | I mean maybe you could make the ci jobs do that but it is a bad approach because you can't do that in the outside world | |
| 16:47:25 | sean-k-mooney | im aware of the chicken and egg situration | |
| 16:47:38 | clarkb | its the wrong approach until you switch to pyproject.toml | |
| 16:47:48 | clarkb | once you do that you can specify what packages you need to build and install your package | |
| 16:47:50 | sean-k-mooney | well the other option is to raise the min of packaging on stable | |
| 16:48:02 | clarkb | until then you need to assume someone is pulling latest setuptools and it will explode for them | |
| 16:48:06 | sean-k-mooney | which we coudl do but its agaisnt stable policy to raise miniums | |
| 16:48:22 | clarkb | yes this is a perfect example of where that policy should probably be bent | |
| 16:48:42 | clarkb | the software is broken as is. The fix is to change the constraints. Policy shouldn't stand in the way of that | |
| 16:48:50 | sean-k-mooney | is there a patch to do that? it would be good to get ci results and see what the fallout is | |
| 16:49:32 | clarkb | I am not aware of any such patch | |
| 17:07:51 | clarkb | I ended up pushing https://review.opendev.org/c/openstack/requirements/+/924764 | |
| 17:08:19 | clarkb | We can see if testing works and take it from there | |
| 17:40:28 | sean-k-mooney | clarkb: sorry was on an internal call but yes that is what i was thinking of doing too | |
| 17:43:28 | opendevreview | sean mooney proposed openstack/nova master: update nova-live-migration to use corrrect disk-formats https://review.opendev.org/c/openstack/nova/+/924746 | |
| 17:43:57 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing nova-live-migation fix https://review.opendev.org/c/openstack/nova/+/924762 | |
| 18:05:50 | opendevreview | Merged openstack/os-vif master: Remove old excludes https://review.opendev.org/c/openstack/os-vif/+/917611 | |
| 18:45:37 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test additional AMI exclusion https://review.opendev.org/c/openstack/nova/+/924775 | |
| 18:45:47 | dansmith | sean-k-mooney: we can squash this ^ if it works | |
| 18:46:02 | dansmith | might as well start the check timer on it while we look further | |
| 18:57:49 | sean-k-mooney | ack | |
| 19:19:18 | _colby | Hey All. We have been runing vGPU on openstack Yoga on centos 8 just fine. We recently added a new hypervisor and went with centos9 since 8 has been moved to EOL. The node with CentOS9 seems to be acting very differently. Its creating all the resource providers (even though we specify the addresses to use in nove). Are there issues with the software versionf of libvirt in centos 9 that are causing issues? | |
| 19:38:13 | opendevreview | sean mooney proposed openstack/nova master: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924731 | |
| 19:57:34 | _colby | also when we attempt to spin up a vgpu instance it does not seem to be creating any of the mediated devices. Is there a way to get this working on centos9? We plan to upgrade openstack versions soo but would like to get this hypervisor running | |
| 20:00:04 | opendevreview | sean mooney proposed openstack/nova stable/2024.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924732 | |
| 20:03:04 | opendevreview | sean mooney proposed openstack/nova stable/2024.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924732 | |
| 20:03:54 | opendevreview | sean mooney proposed openstack/nova stable/2023.2: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924733 | |
| 20:05:47 | opendevreview | sean mooney proposed openstack/nova stable/2023.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924734 | |
| 20:07:30 | sean-k-mooney | _colby: it is possibel to use vgpus on centos 9 however nvidia chagned the feature set for some of there existign gpus in later driver version and remvoed vgpu support for some older cards | |
| 20:07:45 | sean-k-mooney | _colby: that may be whaat you are encoutering but im not sure | |
| 20:08:43 | _colby | we are using A40 cardsd | |
| 20:09:33 | _colby | I did upgrade the nvidia drivers from the other hypervisors | |
| 20:17:14 | _colby | Ill try reverting nvidia software version to match other hypervisors and see what happens | |
| 20:22:19 | sean-k-mooney | _colby: bauzas know more about this then i do and might be able to tell you what version things changed in | |
| 20:22:43 | sean-k-mooney | _colby: but they are based in france and done for today so they may be able to respond in the morning | |
| 20:28:37 | _colby | ok thanks @sean-k-mooney | |
| 21:23:11 | opendevreview | Merged openstack/nova master: conf: Clarify '[api] response_validation help' text https://review.opendev.org/c/openstack/nova/+/924501 | |
| 23:39:59 | opendevreview | Merged openstack/nova master: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924731 | |
| #openstack-nova - 2024-07-24 | |||
| 05:41:53 | ykarel | Hi after https://review.opendev.org/c/openstack/nova/+/924731 - Change force_format strategy to catch mismatches | |
| 05:42:07 | ykarel | many tests are failing involving snapshot/rescue, is this something known already? | |
| 05:43:56 | ykarel | may be now as don't see any recent bug for it, will report | |
| 05:57:28 | ykarel | reported https://bugs.launchpad.net/nova/+bug/2073944 | |
| 06:11:06 | ralonsoh | sean-k-mooney, ^ | |
| 06:11:18 | ralonsoh | bauzas, ^ | |
| 06:11:33 | ralonsoh | upsss maybe you didn't see the message | |
| 06:11:39 | ralonsoh | https://bugs.launchpad.net/nova/+bug/2073944 | |
| 06:11:46 | ralonsoh | since https://review.opendev.org/c/openstack/nova/+/924731 | |
| 07:18:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: test downstream cross repo testing https://review.opendev.org/c/openstack/nova/+/924823 | |
| 07:59:23 | sean-k-mooney | ralonsoh: is that on neutron ci | |
| 08:00:15 | sean-k-mooney | ykarel: ralonsoh: if i recall correctly neutron is using the uec images (ami format) and has not neessialy configure tempest correctly for that | |
| 08:00:36 | ralonsoh | let me check that | |
| 08:01:40 | sean-k-mooney | https://github.com/openstack/neutron/blob/master/zuul.d/tempest-multinode.yaml#L89-L90 | |
| 08:02:49 | ralonsoh | sean-k-mooney, that change was to address https://bugs.launchpad.net/nova/+bug/1939108 | |
| 08:02:51 | sean-k-mooney | tempest is by default configured ot expect qcow | |
| 08:02:53 | ralonsoh | https://review.opendev.org/c/openstack/neutron/+/916629 | |
| 08:03:29 | sean-k-mooney | ralonsoh: right we have since determined that, using the uec image does not actully fix the probelm | |
| 08:03:44 | sean-k-mooney | it makes it less likely but we did it last cycle and the issue didnt go away for us | |
| 08:03:58 | ralonsoh | sean-k-mooney, ok, I'll revert the neutron patch then | |
| 08:05:10 | sean-k-mooney | reverting is one approch the other is to set the approrate disk/container formats in the tempest config | |
| 08:05:51 | sean-k-mooney | for 2024.1 we are also using thte uec images and im currently looking at regerting that to bring it inline with master | |
| 08:05:52 | ralonsoh | but if using uec images makes the issue lest likely to happen | |
| 08:06:05 | ralonsoh | what should I configure in tempest for that? | |
| 08:06:23 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/924746/2 | |
| 08:06:26 | sean-k-mooney | i think that | |
| 08:06:35 | sean-k-mooney | but in general the uec image just change the timeign | |
| 08:06:57 | sean-k-mooney | which means if you dont land on a slow node you might get lucky | |
| 08:07:16 | ralonsoh | yeah, that's another issue | |
| 08:07:38 | sean-k-mooney | ralonsoh: so Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. | |
| 08:07:50 | sean-k-mooney | is actully fixed by using cirros 0.6.2 | |
| 08:08:44 | sean-k-mooney | that specific kernel panic in https://bugs.launchpad.net/nova/+bug/1939108 was a know kernel bug that was fixed in the cirros 0.6.x series | |
| 08:09:25 | ralonsoh | ok, so if this issue is solved in 0.6.2, I could also use qcow2 images too | |
| 08:09:27 | ralonsoh | right? | |
| 08:09:35 | sean-k-mooney | yep | |
| 08:09:40 | ralonsoh | that will fix the current problem | |
| 08:09:46 | ralonsoh | perfect then! | |
| 08:09:58 | sean-k-mooney | the other kernel panics im hoping have been fixed by https://review.opendev.org/c/openstack/devstack/+/924094 | |
| 08:10:04 | bauzas | just saw _colby's question, I probably need more details | |
| 08:10:23 | bauzas | like the nvidia driver version they use | |
| 08:10:46 | sean-k-mooney | bauzas: ya i was hoping you recalled when nvidai split the driver in 2 | |
| 08:10:56 | bauzas | was a long time ago | |
| 08:10:58 | sean-k-mooney | they proably didnt make that chagne for centos8 | |
| 08:11:01 | bauzas | GRID16 | |
| 08:11:18 | sean-k-mooney | but not that they are on centos 9 the preobaly pulled the latest dirvers | |
| 08:11:26 | bauzas | I can lookup the A40 support for GRID16 | |
| 08:11:52 | bauzas | yeah | |
| 08:12:09 | bauzas | but GRID 16 and 17 transition around RHEL9 | |
| 08:12:25 | bauzas | both support some RHEL9 versions, just not the same | |
| 08:12:36 | bauzas | GRID16 is also a LTE version | |
| 08:12:53 | bauzas | while GRID17 is only a production version, ie. with a 1-year term support | |
| 08:14:05 | bauzas | https://docs.nvidia.com/vgpu/16.0/product-support-matrix/index.html#abstract__red-hat-el-kvm | |