| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-23 | |||
| 18:45:37 | opendevreview | Dan Smith proposed openstack/nova master: DNM: Test additional AMI exclusion https://review.opendev.org/c/openstack/nova/+/924775 | |
| 18:45:47 | dansmith | sean-k-mooney: we can squash this ^ if it works | |
| 18:46:02 | dansmith | might as well start the check timer on it while we look further | |
| 18:57:49 | sean-k-mooney | ack | |
| 19:19:18 | _colby | Hey All. We have been runing vGPU on openstack Yoga on centos 8 just fine. We recently added a new hypervisor and went with centos9 since 8 has been moved to EOL. The node with CentOS9 seems to be acting very differently. Its creating all the resource providers (even though we specify the addresses to use in nove). Are there issues with the software versionf of libvirt in centos 9 that are causing issues? | |
| 19:38:13 | opendevreview | sean mooney proposed openstack/nova master: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924731 | |
| 19:57:34 | _colby | also when we attempt to spin up a vgpu instance it does not seem to be creating any of the mediated devices. Is there a way to get this working on centos9? We plan to upgrade openstack versions soo but would like to get this hypervisor running | |
| 20:00:04 | opendevreview | sean mooney proposed openstack/nova stable/2024.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924732 | |
| 20:03:04 | opendevreview | sean mooney proposed openstack/nova stable/2024.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924732 | |
| 20:03:54 | opendevreview | sean mooney proposed openstack/nova stable/2023.2: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924733 | |
| 20:05:47 | opendevreview | sean mooney proposed openstack/nova stable/2023.1: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924734 | |
| 20:07:30 | sean-k-mooney | _colby: it is possibel to use vgpus on centos 9 however nvidia chagned the feature set for some of there existign gpus in later driver version and remvoed vgpu support for some older cards | |
| 20:07:45 | sean-k-mooney | _colby: that may be whaat you are encoutering but im not sure | |
| 20:08:43 | _colby | we are using A40 cardsd | |
| 20:09:33 | _colby | I did upgrade the nvidia drivers from the other hypervisors | |
| 20:17:14 | _colby | Ill try reverting nvidia software version to match other hypervisors and see what happens | |
| 20:22:19 | sean-k-mooney | _colby: bauzas know more about this then i do and might be able to tell you what version things changed in | |
| 20:22:43 | sean-k-mooney | _colby: but they are based in france and done for today so they may be able to respond in the morning | |
| 20:28:37 | _colby | ok thanks @sean-k-mooney | |
| 21:23:11 | opendevreview | Merged openstack/nova master: conf: Clarify '[api] response_validation help' text https://review.opendev.org/c/openstack/nova/+/924501 | |
| 23:39:59 | opendevreview | Merged openstack/nova master: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924731 | |
| #openstack-nova - 2024-07-24 | |||
| 05:41:53 | ykarel | Hi after https://review.opendev.org/c/openstack/nova/+/924731 - Change force_format strategy to catch mismatches | |
| 05:42:07 | ykarel | many tests are failing involving snapshot/rescue, is this something known already? | |
| 05:43:56 | ykarel | may be now as don't see any recent bug for it, will report | |
| 05:57:28 | ykarel | reported https://bugs.launchpad.net/nova/+bug/2073944 | |
| 06:11:06 | ralonsoh | sean-k-mooney, ^ | |
| 06:11:18 | ralonsoh | bauzas, ^ | |
| 06:11:33 | ralonsoh | upsss maybe you didn't see the message | |
| 06:11:39 | ralonsoh | https://bugs.launchpad.net/nova/+bug/2073944 | |
| 06:11:46 | ralonsoh | since https://review.opendev.org/c/openstack/nova/+/924731 | |
| 07:18:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: test downstream cross repo testing https://review.opendev.org/c/openstack/nova/+/924823 | |
| 07:59:23 | sean-k-mooney | ralonsoh: is that on neutron ci | |
| 08:00:15 | sean-k-mooney | ykarel: ralonsoh: if i recall correctly neutron is using the uec images (ami format) and has not neessialy configure tempest correctly for that | |
| 08:00:36 | ralonsoh | let me check that | |
| 08:01:40 | sean-k-mooney | https://github.com/openstack/neutron/blob/master/zuul.d/tempest-multinode.yaml#L89-L90 | |
| 08:02:49 | ralonsoh | sean-k-mooney, that change was to address https://bugs.launchpad.net/nova/+bug/1939108 | |
| 08:02:51 | sean-k-mooney | tempest is by default configured ot expect qcow | |
| 08:02:53 | ralonsoh | https://review.opendev.org/c/openstack/neutron/+/916629 | |
| 08:03:29 | sean-k-mooney | ralonsoh: right we have since determined that, using the uec image does not actully fix the probelm | |
| 08:03:44 | sean-k-mooney | it makes it less likely but we did it last cycle and the issue didnt go away for us | |
| 08:03:58 | ralonsoh | sean-k-mooney, ok, I'll revert the neutron patch then | |
| 08:05:10 | sean-k-mooney | reverting is one approch the other is to set the approrate disk/container formats in the tempest config | |
| 08:05:51 | sean-k-mooney | for 2024.1 we are also using thte uec images and im currently looking at regerting that to bring it inline with master | |
| 08:05:52 | ralonsoh | but if using uec images makes the issue lest likely to happen | |
| 08:06:05 | ralonsoh | what should I configure in tempest for that? | |
| 08:06:23 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/924746/2 | |
| 08:06:26 | sean-k-mooney | i think that | |
| 08:06:35 | sean-k-mooney | but in general the uec image just change the timeign | |
| 08:06:57 | sean-k-mooney | which means if you dont land on a slow node you might get lucky | |
| 08:07:16 | ralonsoh | yeah, that's another issue | |
| 08:07:38 | sean-k-mooney | ralonsoh: so Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. | |
| 08:07:50 | sean-k-mooney | is actully fixed by using cirros 0.6.2 | |
| 08:08:44 | sean-k-mooney | that specific kernel panic in https://bugs.launchpad.net/nova/+bug/1939108 was a know kernel bug that was fixed in the cirros 0.6.x series | |
| 08:09:25 | ralonsoh | ok, so if this issue is solved in 0.6.2, I could also use qcow2 images too | |
| 08:09:27 | ralonsoh | right? | |
| 08:09:35 | sean-k-mooney | yep | |
| 08:09:40 | ralonsoh | that will fix the current problem | |
| 08:09:46 | ralonsoh | perfect then! | |
| 08:09:58 | sean-k-mooney | the other kernel panics im hoping have been fixed by https://review.opendev.org/c/openstack/devstack/+/924094 | |
| 08:10:04 | bauzas | just saw _colby's question, I probably need more details | |
| 08:10:23 | bauzas | like the nvidia driver version they use | |
| 08:10:46 | sean-k-mooney | bauzas: ya i was hoping you recalled when nvidai split the driver in 2 | |
| 08:10:56 | bauzas | was a long time ago | |
| 08:10:58 | sean-k-mooney | they proably didnt make that chagne for centos8 | |
| 08:11:01 | bauzas | GRID16 | |
| 08:11:18 | sean-k-mooney | but not that they are on centos 9 the preobaly pulled the latest dirvers | |
| 08:11:26 | bauzas | I can lookup the A40 support for GRID16 | |
| 08:11:52 | bauzas | yeah | |
| 08:12:09 | bauzas | but GRID 16 and 17 transition around RHEL9 | |
| 08:12:25 | bauzas | both support some RHEL9 versions, just not the same | |
| 08:12:36 | bauzas | GRID16 is also a LTE version | |
| 08:12:53 | bauzas | while GRID17 is only a production version, ie. with a 1-year term support | |
| 08:14:05 | bauzas | https://docs.nvidia.com/vgpu/16.0/product-support-matrix/index.html#abstract__red-hat-el-kvm | |
| 08:37:54 | ykarel | ralonsoh, sean-k-mooney but we still hit kernel panic with those 0.6.2 qcow images, i think i have never seen that issue with uec images | |
| 08:37:55 | ykarel | https://32b7a8914f85d2150a67-dfd2b77d4a65df68dc83e2cbebde05a8.ssl.cf1.rackcdn.com/924004/4/check/tempest-full-py3/861ecb0/testr_results.html | |
| 08:39:50 | ykarel | i mean never seen that issue with uec images in neutron jobs that we track | |
| 08:41:08 | ralonsoh | ykarel, we can also add https://review.opendev.org/c/openstack/nova/+/924746/2 in our jobs | |
| 08:41:55 | ykarel | ralonsoh, yeap +1 to that as we should also avoid those random kernel panics in our jobs | |
| 08:42:28 | ralonsoh | ykarel, perfect, I | |
| 08:42:34 | ralonsoh | I'll change the neutron patch | |
| 09:02:31 | opendevreview | Mark Goddard proposed openstack/nova master: ironic: Let Ironic handle deployment cleanup actions during destroy https://review.opendev.org/c/openstack/nova/+/883411 | |
| 09:04:13 | sean-k-mooney | ykarel: we have see that issue with the uec images, when we swapped to them the kernel panices reduced but they never went away | |
| 09:05:16 | sean-k-mooney | ykarel: i will also poitn out that creating vms with uec images is not how 99% of people use openstack | |
| 09:06:06 | sean-k-mooney | its why nova in particaly cant just use uec images even if that was a 100% reliable solution which it is not. | |
| 09:38:53 | frickler | sean-k-mooney: iiuc the 2024.1 fix is blocked by https://review.opendev.org/c/openstack/nova/+/923831 now? so it could be rebase on top of it to get the tests to pass and reviewers could approve things in parallel? | |
| 09:41:05 | sean-k-mooney | its not blocked | |
| 09:41:11 | sean-k-mooney | frickler: well it is | |
| 09:41:16 | sean-k-mooney | but rebasing is not the answer | |
| 09:41:23 | sean-k-mooney | we can merge https://review.opendev.org/c/openstack/nova/+/923831 | |
| 09:41:26 | sean-k-mooney | and then just rehceck | |
| 09:42:02 | sean-k-mooney | i woudl prefer to not have to respin the cherrypics for the sha but i can if needed | |
| 09:42:11 | frickler | which takes more time than to rebase now. so anyway, bauzas gibi could you look at ^^ please | |
| 09:43:09 | sean-k-mooney | frickler: i was about to ping bauzas, gibi is on pto today but i thinke we can proceed with this without there review | |
| 09:44:22 | frickler | yeah, sorry for being impatient, I was just hoping that with just a single commit per branch this whole thing would be done within hours and not take days again | |
| 09:45:07 | sean-k-mooney | frickler: well that why i started backporting https://review.opendev.org/c/openstack/nova/+/923831 2 weeks ago | |
| 09:45:34 | sean-k-mooney | i had hoped ot have that in place before this hit | |
| 09:45:57 | sean-k-mooney | the prvious cve was also impacted by theuse of the uec images | |
| 09:46:30 | sean-k-mooney | but that ended up bing partly mitigated | |
| 10:01:07 | ykarel | sean-k-mooney, ack atleast in our jobs we have not reproduced that issue with uec image | |
| 10:06:44 | frickler | sean-k-mooney: I noticed that I do have stable core powers, so added +2, but leaving the approval for you | |