| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-23 | |||
| 20:07:45 | sean-k-mooney | _colby: that may be whaat you are encoutering but im not sure | |
| 20:08:43 | _colby | we are using A40 cardsd | |
| 20:09:33 | _colby | I did upgrade the nvidia drivers from the other hypervisors | |
| 20:17:14 | _colby | Ill try reverting nvidia software version to match other hypervisors and see what happens | |
| 20:22:19 | sean-k-mooney | _colby: bauzas know more about this then i do and might be able to tell you what version things changed in | |
| 20:22:43 | sean-k-mooney | _colby: but they are based in france and done for today so they may be able to respond in the morning | |
| 20:28:37 | _colby | ok thanks @sean-k-mooney | |
| 21:23:11 | opendevreview | Merged openstack/nova master: conf: Clarify '[api] response_validation help' text https://review.opendev.org/c/openstack/nova/+/924501 | |
| 23:39:59 | opendevreview | Merged openstack/nova master: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924731 | |
| #openstack-nova - 2024-07-24 | |||
| 05:41:53 | ykarel | Hi after https://review.opendev.org/c/openstack/nova/+/924731 - Change force_format strategy to catch mismatches | |
| 05:42:07 | ykarel | many tests are failing involving snapshot/rescue, is this something known already? | |
| 05:43:56 | ykarel | may be now as don't see any recent bug for it, will report | |
| 05:57:28 | ykarel | reported https://bugs.launchpad.net/nova/+bug/2073944 | |
| 06:11:06 | ralonsoh | sean-k-mooney, ^ | |
| 06:11:18 | ralonsoh | bauzas, ^ | |
| 06:11:33 | ralonsoh | upsss maybe you didn't see the message | |
| 06:11:39 | ralonsoh | https://bugs.launchpad.net/nova/+bug/2073944 | |
| 06:11:46 | ralonsoh | since https://review.opendev.org/c/openstack/nova/+/924731 | |
| 07:18:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: test downstream cross repo testing https://review.opendev.org/c/openstack/nova/+/924823 | |
| 07:59:23 | sean-k-mooney | ralonsoh: is that on neutron ci | |
| 08:00:15 | sean-k-mooney | ykarel: ralonsoh: if i recall correctly neutron is using the uec images (ami format) and has not neessialy configure tempest correctly for that | |
| 08:00:36 | ralonsoh | let me check that | |
| 08:01:40 | sean-k-mooney | https://github.com/openstack/neutron/blob/master/zuul.d/tempest-multinode.yaml#L89-L90 | |
| 08:02:49 | ralonsoh | sean-k-mooney, that change was to address https://bugs.launchpad.net/nova/+bug/1939108 | |
| 08:02:51 | sean-k-mooney | tempest is by default configured ot expect qcow | |
| 08:02:53 | ralonsoh | https://review.opendev.org/c/openstack/neutron/+/916629 | |
| 08:03:29 | sean-k-mooney | ralonsoh: right we have since determined that, using the uec image does not actully fix the probelm | |
| 08:03:44 | sean-k-mooney | it makes it less likely but we did it last cycle and the issue didnt go away for us | |
| 08:03:58 | ralonsoh | sean-k-mooney, ok, I'll revert the neutron patch then | |
| 08:05:10 | sean-k-mooney | reverting is one approch the other is to set the approrate disk/container formats in the tempest config | |
| 08:05:51 | sean-k-mooney | for 2024.1 we are also using thte uec images and im currently looking at regerting that to bring it inline with master | |
| 08:05:52 | ralonsoh | but if using uec images makes the issue lest likely to happen | |
| 08:06:05 | ralonsoh | what should I configure in tempest for that? | |
| 08:06:23 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/924746/2 | |
| 08:06:26 | sean-k-mooney | i think that | |
| 08:06:35 | sean-k-mooney | but in general the uec image just change the timeign | |
| 08:06:57 | sean-k-mooney | which means if you dont land on a slow node you might get lucky | |
| 08:07:16 | ralonsoh | yeah, that's another issue | |
| 08:07:38 | sean-k-mooney | ralonsoh: so Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. | |
| 08:07:50 | sean-k-mooney | is actully fixed by using cirros 0.6.2 | |
| 08:08:44 | sean-k-mooney | that specific kernel panic in https://bugs.launchpad.net/nova/+bug/1939108 was a know kernel bug that was fixed in the cirros 0.6.x series | |
| 08:09:25 | ralonsoh | ok, so if this issue is solved in 0.6.2, I could also use qcow2 images too | |
| 08:09:27 | ralonsoh | right? | |
| 08:09:35 | sean-k-mooney | yep | |
| 08:09:40 | ralonsoh | that will fix the current problem | |
| 08:09:46 | ralonsoh | perfect then! | |
| 08:09:58 | sean-k-mooney | the other kernel panics im hoping have been fixed by https://review.opendev.org/c/openstack/devstack/+/924094 | |
| 08:10:04 | bauzas | just saw _colby's question, I probably need more details | |
| 08:10:23 | bauzas | like the nvidia driver version they use | |
| 08:10:46 | sean-k-mooney | bauzas: ya i was hoping you recalled when nvidai split the driver in 2 | |
| 08:10:56 | bauzas | was a long time ago | |
| 08:10:58 | sean-k-mooney | they proably didnt make that chagne for centos8 | |
| 08:11:01 | bauzas | GRID16 | |
| 08:11:18 | sean-k-mooney | but not that they are on centos 9 the preobaly pulled the latest dirvers | |
| 08:11:26 | bauzas | I can lookup the A40 support for GRID16 | |
| 08:11:52 | bauzas | yeah | |
| 08:12:09 | bauzas | but GRID 16 and 17 transition around RHEL9 | |
| 08:12:25 | bauzas | both support some RHEL9 versions, just not the same | |
| 08:12:36 | bauzas | GRID16 is also a LTE version | |
| 08:12:53 | bauzas | while GRID17 is only a production version, ie. with a 1-year term support | |
| 08:14:05 | bauzas | https://docs.nvidia.com/vgpu/16.0/product-support-matrix/index.html#abstract__red-hat-el-kvm | |
| 08:37:54 | ykarel | ralonsoh, sean-k-mooney but we still hit kernel panic with those 0.6.2 qcow images, i think i have never seen that issue with uec images | |
| 08:37:55 | ykarel | https://32b7a8914f85d2150a67-dfd2b77d4a65df68dc83e2cbebde05a8.ssl.cf1.rackcdn.com/924004/4/check/tempest-full-py3/861ecb0/testr_results.html | |
| 08:39:50 | ykarel | i mean never seen that issue with uec images in neutron jobs that we track | |
| 08:41:08 | ralonsoh | ykarel, we can also add https://review.opendev.org/c/openstack/nova/+/924746/2 in our jobs | |
| 08:41:55 | ykarel | ralonsoh, yeap +1 to that as we should also avoid those random kernel panics in our jobs | |
| 08:42:28 | ralonsoh | ykarel, perfect, I | |
| 08:42:34 | ralonsoh | I'll change the neutron patch | |
| 09:02:31 | opendevreview | Mark Goddard proposed openstack/nova master: ironic: Let Ironic handle deployment cleanup actions during destroy https://review.opendev.org/c/openstack/nova/+/883411 | |
| 09:04:13 | sean-k-mooney | ykarel: we have see that issue with the uec images, when we swapped to them the kernel panices reduced but they never went away | |
| 09:05:16 | sean-k-mooney | ykarel: i will also poitn out that creating vms with uec images is not how 99% of people use openstack | |
| 09:06:06 | sean-k-mooney | its why nova in particaly cant just use uec images even if that was a 100% reliable solution which it is not. | |
| 09:38:53 | frickler | sean-k-mooney: iiuc the 2024.1 fix is blocked by https://review.opendev.org/c/openstack/nova/+/923831 now? so it could be rebase on top of it to get the tests to pass and reviewers could approve things in parallel? | |
| 09:41:05 | sean-k-mooney | its not blocked | |
| 09:41:11 | sean-k-mooney | frickler: well it is | |
| 09:41:16 | sean-k-mooney | but rebasing is not the answer | |
| 09:41:23 | sean-k-mooney | we can merge https://review.opendev.org/c/openstack/nova/+/923831 | |
| 09:41:26 | sean-k-mooney | and then just rehceck | |
| 09:42:02 | sean-k-mooney | i woudl prefer to not have to respin the cherrypics for the sha but i can if needed | |
| 09:42:11 | frickler | which takes more time than to rebase now. so anyway, bauzas gibi could you look at ^^ please | |
| 09:43:09 | sean-k-mooney | frickler: i was about to ping bauzas, gibi is on pto today but i thinke we can proceed with this without there review | |
| 09:44:22 | frickler | yeah, sorry for being impatient, I was just hoping that with just a single commit per branch this whole thing would be done within hours and not take days again | |
| 09:45:07 | sean-k-mooney | frickler: well that why i started backporting https://review.opendev.org/c/openstack/nova/+/923831 2 weeks ago | |
| 09:45:34 | sean-k-mooney | i had hoped ot have that in place before this hit | |
| 09:45:57 | sean-k-mooney | the prvious cve was also impacted by theuse of the uec images | |
| 09:46:30 | sean-k-mooney | but that ended up bing partly mitigated | |
| 10:01:07 | ykarel | sean-k-mooney, ack atleast in our jobs we have not reproduced that issue with uec image | |
| 10:06:44 | frickler | sean-k-mooney: I noticed that I do have stable core powers, so added +2, but leaving the approval for you | |
| 10:13:25 | sean-k-mooney | ill give bauzas an hour or so and move forward if he or stephenfin dont have time to review https://review.opendev.org/c/openstack/nova/+/923831 by noon(utc+1) | |
| 10:16:08 | bauzas | here now | |
| 10:17:10 | bauzas | https://review.opendev.org/c/openstack/nova/+/923831 +Wd | |
| 10:17:22 | sean-k-mooney | bauzas: thanks | |
| 10:49:08 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add config option to require secure SPICE. https://review.opendev.org/c/openstack/nova/+/922544 | |
| 10:49:09 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Add guest devices to support SPICE USB. https://review.opendev.org/c/openstack/nova/+/922547 | |
| 10:49:09 | opendevreview | Michael Still proposed openstack/nova master: libvirt: allow concurrent access to SPICE consoles. https://review.opendev.org/c/openstack/nova/+/922546 | |
| 10:49:10 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Optionally support sound when using SPICE. https://review.opendev.org/c/openstack/nova/+/922549 | |
| 10:49:10 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Optionally enable SPICE debug logging. https://review.opendev.org/c/openstack/nova/+/922548 | |
| 10:49:11 | opendevreview | Michael Still proposed openstack/nova master: libvirt: allow direct SPICE connections to qemu https://review.opendev.org/c/openstack/nova/+/924844 | |
| 10:49:44 | mikal | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/924844/1 is my first swing at the reworking of the Nova APIs for the SPICE consoles. | |
| 10:51:12 | sean-k-mooney | mikal: stephenfin is working on a paralle series to add openapi schema validation that might collide with that by the way but we can cross that bridge wehn we get there | |