| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-16 | |||
| 16:00:31 | grandchild | o/ | |
| 16:00:37 | bauzas | heya | |
| 16:00:41 | dansmith | o/ (kinda) | |
| 16:00:44 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:00:52 | bauzas | let's try to have a short meeting | |
| 16:01:02 | bauzas | some folks have another meeting in 15 mins | |
| 16:01:07 | bauzas | #topic Bugs (stuck/critical) | |
| 16:01:11 | bauzas | #info No Critical bug | |
| 16:01:15 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 46 new untriaged bugs (+3 since the last meeting) | |
| 16:01:19 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:01:35 | bauzas | Uggla worked a lot on the bugs | |
| 16:01:57 | bauzas | Uggla: have you modified the status for all of the ones you looked at ? | |
| 16:02:50 | fwiesel | o/ | |
| 16:03:02 | bauzas | anyway, moving on | |
| 16:03:16 | bauzas | elodilles: are you okay if you could look at the bugs next week ? | |
| 16:03:18 | elodilles | sure o/ | |
| 16:04:04 | gibi | o/ | |
| 16:04:35 | bauzas | cool | |
| 16:04:39 | bauzas | #info bug baton is elodilles | |
| 16:04:44 | bauzas | #info bug baton is elodilles | |
| 16:04:46 | bauzas | shit | |
| 16:04:50 | bauzas | #topic Gate status | |
| 16:04:54 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:04:58 | bauzas | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | |
| 16:05:02 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:05:07 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:05:31 | bauzas | do guys want to discuss about the guest kernel issues ? | |
| 16:05:46 | bauzas | (periodic runs are all ok) | |
| 16:06:30 | bauzas | looks not, moving on then | |
| 16:06:35 | bauzas | #topic Release Planning | |
| 16:06:39 | bauzas | #link https://releases.openstack.org/caracal/schedule.html#nova | |
| 16:06:42 | bauzas | #info Caracal-3 (and feature freeze) milestone in 6 weeks | |
| 16:06:46 | bauzas | #topic Review priorities | |
| 16:07:06 | bauzas | as a reminder, please look at this etherpad if you want to review our implementations :)= | |
| 16:07:10 | bauzas | #link https://etherpad.opendev.org/p/nova-caracal-status | |
| 16:07:17 | bauzas | #topic Stable Branches | |
| 16:07:21 | bauzas | elodilles: your time | |
| 16:07:25 | elodilles | o/ | |
| 16:07:32 | elodilles | nothing to report actually | |
| 16:07:37 | elodilles | state is the same as last week | |
| 16:07:57 | elodilles | (i've pinged release cores to approve Nova's Zed release) | |
| 16:08:02 | bauzas | ++ | |
| 16:08:02 | elodilles | that's all | |
| 16:08:13 | bauzas | I think I did +1 for the zed release | |
| 16:08:21 | elodilles | yepp, you did | |
| 16:08:25 | bauzas | cool | |
| 16:08:36 | bauzas | elodilles: thanks | |
| 16:08:39 | elodilles | thanks too :) | |
| 16:08:40 | bauzas | #topic vmwareapi 3rd-party CI efforts Highlights | |
| 16:08:50 | bauzas | grandchild: fwiesel: your time :) | |
| 16:09:01 | fwiesel | #Info ETA on exemption for public access by end of week. Hopefully under DNS name: openstack-ci-logs.global.cloud.sap | |
| 16:09:12 | bauzas | <3 | |
| 16:09:14 | fwiesel | So, not much happened. Still wrapped in red tape. But at least I have a dns name. | |
| 16:09:35 | bauzas | so we'll wait :) | |
| 16:09:36 | fwiesel | That's it from my side. Any questions? | |
| 16:09:41 | bauzas | thanks a lot | |
| 16:09:44 | bauzas | nope from me | |
| 16:09:50 | fwiesel | You're welcome. | |
| 16:10:27 | bauzas | I think we said we should wait until milestone-2 for seeing whether we would remove vmwareapi, but given what you did and what you continue, of course we won't | |
| 16:10:45 | bauzas | so let's continue to see what things happen | |
| 16:11:01 | fwiesel | Thanks. Much appreciated. | |
| 16:11:08 | bauzas | thanks folks for the work, definitely nice | |
| 16:11:21 | bauzas | #topic Open discussion | |
| 16:11:32 | bauzas | nothing in the agenda, anything anyone ? | |
| 16:12:18 | bauzas | looks not, | |
| 16:12:23 | bauzas | thanks all, was a quick one | |
| 16:12:27 | bauzas | #endmeeting | |
| 16:12:27 | opendevmeet | Meeting ended Tue Jan 16 16:12:27 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:12:27 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-01-16-16.00.html | |
| 16:12:27 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-01-16-16.00.txt | |
| 16:12:27 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-01-16-16.00.log.html | |
| 16:13:02 | fwiesel | Thanks chat with you next week, hopefully with more news | |
| 16:14:39 | bauzas | ++ | |
| 16:26:16 | Uggla | @bauzas, sorry I was discussing with Artom, and I did not managed to answer. | |
| 16:27:47 | Uggla | Yep I updated the status of the bugs. I read a lot of them, but did not manage to triage a lot of them. | |
| 16:29:06 | opendevreview | Takashi Kajinami proposed openstack/placement master: Bump hacking https://review.opendev.org/c/openstack/placement/+/905706 | |
| 16:29:08 | Uggla | I think https://bugs.launchpad.net/nova/+bug/2048154 is important to be fixed. I hope @Amit could have a look. | |
| 16:30:26 | opendevreview | Takashi Kajinami proposed openstack/python-novaclient master: Bump hacking https://review.opendev.org/c/openstack/python-novaclient/+/905707 | |
| 16:41:58 | opendevreview | Takashi Kajinami proposed openstack/osc-placement master: Bump hacking https://review.opendev.org/c/openstack/osc-placement/+/905715 | |
| 16:44:24 | frickler | kashyap: dansmith: added some comments on the etherpad. given that the issue (afaict) only happens for volume resize tests, I'm wondering how we could verify that no actual data corruption is happening on the volume in that scenario | |
| 16:44:42 | frickler | because if it does happen, chasing possible kernel bugs is kind of moot | |
| 16:45:14 | kashyap | frickler: Hi, yeah, I've just seen your comment on the Etherpad. Good question. (I'm in a meeting, and will be slow here) | |
| 16:45:31 | dansmith | frickler: these test instances haven't even made it out of initial kernel boot, so no real opportunity to corrupt anything yet | |
| 16:46:05 | frickler | is this happening on the initial boot, not after resize? | |
| 16:46:06 | dansmith | frickler: and we're not booting from the volumes either, since we're using the preloaded kernel boot method, which means corruption of the on-disk data causing the crash wouldn't be a thing | |
| 16:47:34 | dansmith | before the recent switch to kernel preload that could have been a possible option, but with kernel preload, disk corruption due to the resize should be ruled out AFAIK | |
| 16:49:09 | dansmith | I'm also not sure these are only happening on resize tests | |
| 16:57:37 | frickler | well the errors seem to be happening around the time when the switch to the actual root-fs would be happening. and in the traceback I was looking at, it was happening after the resize. do you happen to have a pointer to the preloaded kernel change? | |
| 16:59:00 | dansmith | frickler: I think it's still loading modules from the initramfs when it crashes | |
| 17:00:03 | dansmith | frickler: https://review.opendev.org/c/openstack/nova/+/902217 | |
| 17:00:07 | dansmith | (happened while I was out) | |
| 17:01:41 | dansmith | frickler: I think what you're seeing there is that it tried to load all the modules from the initramfs that it needed to have access to storage, which all crashed, and then it kept going trying to do the switch_root and failed to do so | |
| 17:02:26 | dansmith | it did get vda from the virtio driver so I would think it would have worked even, but with that level of broken having happened who knows | |
| 17:03:23 | frickler | from a quick scan, that patch only changes some specific nova jobs, not tempest-integrated-compute? | |
| 17:04:10 | dansmith | frickler: tbf, I hadn't even looked at the patch, I was going based on sean-k-mooney, melwitt saying nova-next was the only one still doing image boot | |
| 17:05:13 | dansmith | the original bug we opened for this GPF was before we even had the resize-volume-backed test, FWIW :) | |
| 17:20:51 | tkajinam | o/ I'd appreciate it if https://review.opendev.org/c/openstack/nova/+/905314 can get some attention because it's now blocking requirement bump. | |
| 17:22:51 | dansmith | frickler: yeah, that instance in question is doing preloaded kernel boot: | |
| 17:22:52 | dansmith | Jan 16 11:45:59.955345 np0036421623 nova-compute[74855]: <kernel>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/kernel</kernel> | |
| 17:22:56 | dansmith | Jan 16 11:45:59.955345 np0036421623 nova-compute[74855]: <initrd>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/ramdisk</initrd> | |