| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-07-23 | |||
| 20:28:37 | _colby | ok thanks @sean-k-mooney | |
| 21:23:11 | opendevreview | Merged openstack/nova master: conf: Clarify '[api] response_validation help' text https://review.opendev.org/c/openstack/nova/+/924501 | |
| 23:39:59 | opendevreview | Merged openstack/nova master: Change force_format strategy to catch mismatches https://review.opendev.org/c/openstack/nova/+/924731 | |
| #openstack-nova - 2024-07-24 | |||
| 05:41:53 | ykarel | Hi after https://review.opendev.org/c/openstack/nova/+/924731 - Change force_format strategy to catch mismatches | |
| 05:42:07 | ykarel | many tests are failing involving snapshot/rescue, is this something known already? | |
| 05:43:56 | ykarel | may be now as don't see any recent bug for it, will report | |
| 05:57:28 | ykarel | reported https://bugs.launchpad.net/nova/+bug/2073944 | |
| 06:11:06 | ralonsoh | sean-k-mooney, ^ | |
| 06:11:18 | ralonsoh | bauzas, ^ | |
| 06:11:33 | ralonsoh | upsss maybe you didn't see the message | |
| 06:11:39 | ralonsoh | https://bugs.launchpad.net/nova/+bug/2073944 | |
| 06:11:46 | ralonsoh | since https://review.opendev.org/c/openstack/nova/+/924731 | |
| 07:18:35 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: test downstream cross repo testing https://review.opendev.org/c/openstack/nova/+/924823 | |
| 07:59:23 | sean-k-mooney | ralonsoh: is that on neutron ci | |
| 08:00:15 | sean-k-mooney | ykarel: ralonsoh: if i recall correctly neutron is using the uec images (ami format) and has not neessialy configure tempest correctly for that | |
| 08:00:36 | ralonsoh | let me check that | |
| 08:01:40 | sean-k-mooney | https://github.com/openstack/neutron/blob/master/zuul.d/tempest-multinode.yaml#L89-L90 | |
| 08:02:49 | ralonsoh | sean-k-mooney, that change was to address https://bugs.launchpad.net/nova/+bug/1939108 | |
| 08:02:51 | sean-k-mooney | tempest is by default configured ot expect qcow | |
| 08:02:53 | ralonsoh | https://review.opendev.org/c/openstack/neutron/+/916629 | |
| 08:03:29 | sean-k-mooney | ralonsoh: right we have since determined that, using the uec image does not actully fix the probelm | |
| 08:03:44 | sean-k-mooney | it makes it less likely but we did it last cycle and the issue didnt go away for us | |
| 08:03:58 | ralonsoh | sean-k-mooney, ok, I'll revert the neutron patch then | |
| 08:05:10 | sean-k-mooney | reverting is one approch the other is to set the approrate disk/container formats in the tempest config | |
| 08:05:51 | sean-k-mooney | for 2024.1 we are also using thte uec images and im currently looking at regerting that to bring it inline with master | |
| 08:05:52 | ralonsoh | but if using uec images makes the issue lest likely to happen | |
| 08:06:05 | ralonsoh | what should I configure in tempest for that? | |
| 08:06:23 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/924746/2 | |
| 08:06:26 | sean-k-mooney | i think that | |
| 08:06:35 | sean-k-mooney | but in general the uec image just change the timeign | |
| 08:06:57 | sean-k-mooney | which means if you dont land on a slow node you might get lucky | |
| 08:07:16 | ralonsoh | yeah, that's another issue | |
| 08:07:38 | sean-k-mooney | ralonsoh: so Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. | |
| 08:07:50 | sean-k-mooney | is actully fixed by using cirros 0.6.2 | |
| 08:08:44 | sean-k-mooney | that specific kernel panic in https://bugs.launchpad.net/nova/+bug/1939108 was a know kernel bug that was fixed in the cirros 0.6.x series | |
| 08:09:25 | ralonsoh | ok, so if this issue is solved in 0.6.2, I could also use qcow2 images too | |
| 08:09:27 | ralonsoh | right? | |
| 08:09:35 | sean-k-mooney | yep | |
| 08:09:40 | ralonsoh | that will fix the current problem | |
| 08:09:46 | ralonsoh | perfect then! | |
| 08:09:58 | sean-k-mooney | the other kernel panics im hoping have been fixed by https://review.opendev.org/c/openstack/devstack/+/924094 | |
| 08:10:04 | bauzas | just saw _colby's question, I probably need more details | |
| 08:10:23 | bauzas | like the nvidia driver version they use | |
| 08:10:46 | sean-k-mooney | bauzas: ya i was hoping you recalled when nvidai split the driver in 2 | |
| 08:10:56 | bauzas | was a long time ago | |
| 08:10:58 | sean-k-mooney | they proably didnt make that chagne for centos8 | |
| 08:11:01 | bauzas | GRID16 | |
| 08:11:18 | sean-k-mooney | but not that they are on centos 9 the preobaly pulled the latest dirvers | |
| 08:11:26 | bauzas | I can lookup the A40 support for GRID16 | |
| 08:11:52 | bauzas | yeah | |
| 08:12:09 | bauzas | but GRID 16 and 17 transition around RHEL9 | |
| 08:12:25 | bauzas | both support some RHEL9 versions, just not the same | |
| 08:12:36 | bauzas | GRID16 is also a LTE version | |
| 08:12:53 | bauzas | while GRID17 is only a production version, ie. with a 1-year term support | |
| 08:14:05 | bauzas | https://docs.nvidia.com/vgpu/16.0/product-support-matrix/index.html#abstract__red-hat-el-kvm | |
| 08:37:54 | ykarel | ralonsoh, sean-k-mooney but we still hit kernel panic with those 0.6.2 qcow images, i think i have never seen that issue with uec images | |
| 08:37:55 | ykarel | https://32b7a8914f85d2150a67-dfd2b77d4a65df68dc83e2cbebde05a8.ssl.cf1.rackcdn.com/924004/4/check/tempest-full-py3/861ecb0/testr_results.html | |
| 08:39:50 | ykarel | i mean never seen that issue with uec images in neutron jobs that we track | |
| 08:41:08 | ralonsoh | ykarel, we can also add https://review.opendev.org/c/openstack/nova/+/924746/2 in our jobs | |
| 08:41:55 | ykarel | ralonsoh, yeap +1 to that as we should also avoid those random kernel panics in our jobs | |
| 08:42:28 | ralonsoh | ykarel, perfect, I | |
| 08:42:34 | ralonsoh | I'll change the neutron patch | |
| 09:02:31 | opendevreview | Mark Goddard proposed openstack/nova master: ironic: Let Ironic handle deployment cleanup actions during destroy https://review.opendev.org/c/openstack/nova/+/883411 | |
| 09:04:13 | sean-k-mooney | ykarel: we have see that issue with the uec images, when we swapped to them the kernel panices reduced but they never went away | |
| 09:05:16 | sean-k-mooney | ykarel: i will also poitn out that creating vms with uec images is not how 99% of people use openstack | |
| 09:06:06 | sean-k-mooney | its why nova in particaly cant just use uec images even if that was a 100% reliable solution which it is not. | |
| 09:38:53 | frickler | sean-k-mooney: iiuc the 2024.1 fix is blocked by https://review.opendev.org/c/openstack/nova/+/923831 now? so it could be rebase on top of it to get the tests to pass and reviewers could approve things in parallel? | |
| 09:41:05 | sean-k-mooney | its not blocked | |
| 09:41:11 | sean-k-mooney | frickler: well it is | |
| 09:41:16 | sean-k-mooney | but rebasing is not the answer | |
| 09:41:23 | sean-k-mooney | we can merge https://review.opendev.org/c/openstack/nova/+/923831 | |
| 09:41:26 | sean-k-mooney | and then just rehceck | |
| 09:42:02 | sean-k-mooney | i woudl prefer to not have to respin the cherrypics for the sha but i can if needed | |
| 09:42:11 | frickler | which takes more time than to rebase now. so anyway, bauzas gibi could you look at ^^ please | |
| 09:43:09 | sean-k-mooney | frickler: i was about to ping bauzas, gibi is on pto today but i thinke we can proceed with this without there review | |
| 09:44:22 | frickler | yeah, sorry for being impatient, I was just hoping that with just a single commit per branch this whole thing would be done within hours and not take days again | |
| 09:45:07 | sean-k-mooney | frickler: well that why i started backporting https://review.opendev.org/c/openstack/nova/+/923831 2 weeks ago | |
| 09:45:34 | sean-k-mooney | i had hoped ot have that in place before this hit | |
| 09:45:57 | sean-k-mooney | the prvious cve was also impacted by theuse of the uec images | |
| 09:46:30 | sean-k-mooney | but that ended up bing partly mitigated | |
| 10:01:07 | ykarel | sean-k-mooney, ack atleast in our jobs we have not reproduced that issue with uec image | |
| 10:06:44 | frickler | sean-k-mooney: I noticed that I do have stable core powers, so added +2, but leaving the approval for you | |
| 10:13:25 | sean-k-mooney | ill give bauzas an hour or so and move forward if he or stephenfin dont have time to review https://review.opendev.org/c/openstack/nova/+/923831 by noon(utc+1) | |
| 10:16:08 | bauzas | here now | |
| 10:17:10 | bauzas | https://review.opendev.org/c/openstack/nova/+/923831 +Wd | |
| 10:17:22 | sean-k-mooney | bauzas: thanks | |
| 10:49:08 | opendevreview | Michael Still proposed openstack/nova master: libvirt: Add config option to require secure SPICE. https://review.opendev.org/c/openstack/nova/+/922544 | |
| 10:49:09 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Add guest devices to support SPICE USB. https://review.opendev.org/c/openstack/nova/+/922547 | |
| 10:49:09 | opendevreview | Michael Still proposed openstack/nova master: libvirt: allow concurrent access to SPICE consoles. https://review.opendev.org/c/openstack/nova/+/922546 | |
| 10:49:10 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Optionally support sound when using SPICE. https://review.opendev.org/c/openstack/nova/+/922549 | |
| 10:49:10 | opendevreview | Michael Still proposed openstack/nova master: WIP: libvirt: Optionally enable SPICE debug logging. https://review.opendev.org/c/openstack/nova/+/922548 | |
| 10:49:11 | opendevreview | Michael Still proposed openstack/nova master: libvirt: allow direct SPICE connections to qemu https://review.opendev.org/c/openstack/nova/+/924844 | |
| 10:49:44 | mikal | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/924844/1 is my first swing at the reworking of the Nova APIs for the SPICE consoles. | |
| 10:51:12 | sean-k-mooney | mikal: stephenfin is working on a paralle series to add openapi schema validation that might collide with that by the way but we can cross that bridge wehn we get there | |
| 10:51:55 | mikal | sean-k-mooney: yeah, fair enough. Imma just gonna keep rebasing and uploading for now. | |
| 10:53:38 | sean-k-mooney | mikal: currently my focuse is unfortuentlly on backporting a cve fix upstream and downstream so while i skimmed that i proably wont be able to start reviewing properly unitl next week | |
| 10:53:41 | mikal | sean-k-mooney: assuming that the bottom three of those pass CI (and the first two already have before), I'll move onto reimplementing the sound model as an extra spec next. There is also a patch in that stack which adds USB redirect devices, I am undecided if I should make the number of devices configurable in extra specs as well. | |
| 10:53:51 | sean-k-mooney | hopefullly we will get most of the cve patches landed today | |
| 10:54:39 | mikal | sean-k-mooney: yeah fair enough, I'm just a bit concerned about the code complete deadline, so I want to get them out there as quick as I can. | |
| 10:54:40 | sean-k-mooney | we make the number of serial ports configurable in the flavor extra specs and image properlteis i belive | |