| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-28 | |||
| 16:25:43 | bauzas | I've seen sean-k-mooney already reviewing it so I'm fine with moving it to the 'accepted bugfixes' section | |
| 16:27:16 | bauzas | anyway | |
| 16:27:19 | bauzas | moving on | |
| 16:27:40 | bauzas | #topic Stable Branches | |
| 16:27:45 | bauzas | elodilles: go for it | |
| 16:27:46 | sean-k-mooney | yes its a non trivial fix but im reviewing it | |
| 16:27:57 | elodilles | #info stable gates don't seem blocked | |
| 16:28:07 | elodilles | #info stable release patches still open for review: https://review.opendev.org/q/project:openstack/releases+is:open+intopic:nova | |
| 16:28:18 | elodilles | +1: yoga is going to be unmaintained, so final stable/yoga release should happen ASAP - https://etherpad.opendev.org/p/nova-stable-yoga-eom | |
| 16:28:27 | elodilles | actually, i've updated the release patches ^^^ | |
| 16:28:46 | elodilles | to contain the recently merged stable backports | |
| 16:29:00 | elodilles | and i suggest to not to wait anymore | |
| 16:29:13 | elodilles | the patches were proposed 1 month ago :( | |
| 16:29:23 | elodilles | and we can release *anytime* | |
| 16:29:42 | elodilles | (except yoga of course, as that release is the final one) | |
| 16:30:10 | elodilles | so release liaisons or PTL pls review o:) | |
| 16:30:49 | elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 16:30:49 | elodilles | and the usual, last item: | |
| 16:30:58 | elodilles | that's all from me | |
| 16:31:27 | bauzas | ++ | |
| 16:31:41 | bauzas | and yeah I've seen the releases patches | |
| 16:32:15 | bauzas | fwiw, I'm ok with releasing 2023.1 and 2023.2 since both have the fixes I want | |
| 16:32:25 | bauzas | Zed is still missing the RPC backports | |
| 16:32:25 | elodilles | \o/ | |
| 16:32:37 | bauzas | I'll try to look at them tomorrow | |
| 16:32:44 | bauzas | elodilles: ping me again tomorrow | |
| 16:33:11 | elodilles | bauzas: ack, thanks in advance! | |
| 16:36:09 | bauzas | ok, next is a new topic | |
| 16:36:11 | bauzas | #topic vmwareapi 3rd-party CI efforts Highlights | |
| 16:36:31 | fwiesel | Shall I? | |
| 16:36:49 | bauzas | fwiesel: shoot :) | |
| 16:36:53 | fwiesel | #Info Gerrit Event Stream connected to our CI (Argo) with personal user | |
| 16:37:27 | fwiesel | So, nothing spectacular. Events go in, they could trigger whatever we want to run later. That part is the more tricky one, and peope are working on it. | |
| 16:38:07 | fwiesel | The most complicated part is the networking, we try to cut it down as much as possible so we test only nova. | |
| 16:38:48 | fwiesel | #Info CI user needs a name. According to naming convention: SAP vmwareapi CI. Has risk of confusion, as SAP is not a vendor of the technology. Our preference simply "SAP CI" or "SAP tempest CI". | |
| 16:39:34 | fwiesel | So, the naming convention on the 3rd party CI suggests "Company Technology CI" | |
| 16:39:37 | bauzas | fwiesel: I'd prefer 'vmwareapi CI run by SAP' | |
| 16:40:02 | bauzas | something telling users what exactly has been run | |
| 16:40:24 | gibi | +1 on keep mentioning vmwareapi in the name | |
| 16:40:26 | fwiesel | That would work. But if we run another CI, say for cinder, then we end up with the same thing as the VMware nsxt CI running cinder | |
| 16:40:27 | bauzas | or yeah, "SAP vmwareapi CI" | |
| 16:41:01 | bauzas | honestly, I don't wanna bikeshed | |
| 16:41:17 | bauzas | I understand that you'd appreciate the SAP name in the CI job name | |
| 16:41:27 | fwiesel | Okay then we go with your proposal "vmwareapi CI run by SAP" | |
| 16:41:34 | bauzas | but I also want my users to know that it's testing with vmwareapi virt driver | |
| 16:41:47 | bauzas | so, please keep both names | |
| 16:41:54 | fwiesel | Fine by me. | |
| 16:41:57 | bauzas | I'm French and I'm bad at namings :) | |
| 16:42:17 | bauzas | but my gut is saying "any words with both SAP and vmwareapi are fine by me" | |
| 16:42:51 | fwiesel | Then we agree on "vmwareapi CI run by SAP"? | |
| 16:43:58 | fwiesel | #agreed The CI user will be named "vmwareapi CI run by SAP" | |
| 16:44:11 | fwiesel | That's from my side. Any questions? | |
| 16:45:21 | fwiesel | Well, then I think we are through. | |
| 16:46:23 | bauzas | ++ | |
| 16:46:38 | bauzas | thanks fwiesel for the report | |
| 16:46:40 | bauzas | :) | |
| 16:46:53 | bauzas | greatly appreciated | |
| 16:46:59 | fwiesel | You're welcome | |
| 16:47:00 | bauzas | last topic then | |
| 16:47:03 | bauzas | #topic Open discussion | |
| 16:47:12 | bauzas | the agenda is empty | |
| 16:47:19 | bauzas | anything anyone ? | |
| 16:48:10 | sean-k-mooney | nope | |
| 16:48:12 | bauzas | cool | |
| 16:48:20 | bauzas | thanks all, giving you back 12 mins | |
| 16:48:22 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-11-28-16.00.log.html | |
| 16:48:22 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-11-28-16.00.txt | |
| 16:48:22 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-11-28-16.00.html | |
| 16:48:22 | opendevmeet | Meeting ended Tue Nov 28 16:48:22 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:48:22 | bauzas | #endmeeting | |
| 16:49:57 | elodilles | thanks bauzas o- | |
| 16:50:00 | elodilles | o/ | |
| 16:50:15 | fwiesel | thanks a lot \o | |
| 19:41:08 | melwitt | old kernel bugs or people saying to run fsck and try again. not sure how to approach this, so if you have any thoughts, it would probably help | |
| 19:41:08 | melwitt | dansmith: I have noticed (anecdotally) that a large portion of gate failures have been due to guest kernel panics. I have spent some time looking at them and see for example "pci 0000:00:03.6: BAR 13: failed to assign [io size 0x1000]" and "pci 0000:00:03.6: BAR 13: no space for [io size 0x1000]" I'm not sure how to tell which thing caused the panic. and in my search for more info online I haven't found anything other than mention of | |
| 19:42:16 | melwitt | it seems weird to me how frequent this is. don't know what that could mean either. in the past guest kernel panics were a lot more rare | |
| 19:46:58 | dansmith | orly, well, that might explain the uptick lately... but yeah, I'd lean on kashyap and sean-k-mooney to look at those sorts of things | |
| 19:47:28 | dansmith | unfortunately, we've had trouble getting our kernel team to look at those things because we don't really run a supported kernel, but with the recent cirros update, we're closer to a support-able kernel I think | |
| 19:47:31 | dansmith | so might be worth trying again | |
| 19:49:03 | melwitt | ok, yeah. I checked and we are running the newest kernel that cirros supports. I did wonder if there could be a chance for improvement by using something other than cirros, like to get a different/newer kernel version | |
| 19:50:05 | melwitt | or I guess maybe building our own cirros with a newer kernel | |
| 20:07:37 | dansmith | I mean, it's possible but cirros is also tailored to work in very small environments, which is important for us | |
| 20:07:56 | dansmith | a stock distro kernel (and certainly image) won't likely work well in our super tiny guests | |
| 20:08:06 | dansmith | and if we had to increase the size of those, we'd need to make other changes | |
| 20:08:19 | dansmith | getting some kernel person to tell us what the crashes mean would be a good first start | |
| 20:08:40 | dansmith | like if it's "that should never happen" or "they're already memory constrained and hitting a bug as a result" or whatever | |
| 20:30:00 | melwitt | yeah | |
| 21:58:15 | opendevreview | Merged openstack/python-novaclient master: Fix typos https://review.opendev.org/c/openstack/python-novaclient/+/901457 | |
| 23:44:42 | sean-k-mooney[m] | melwitt: cirrus uses the ubuntu lts kernel just in case you didnt know that | |
| 23:45:48 | sean-k-mooney[m] | i really should try and make time to get my alpine serise working again but honestly havnt had time or energy to do that. | |
| 23:46:36 | melwitt | sean-k-mooney[m]: I saw on their site 🙂 but I also see the cirros we're using is also using the nearly newest kernel for jammy already so I guess maybe there's little chance a newer version would make a difference | |
| 23:46:36 | melwitt | sean-k-mooney[m]: I saw on their site 🙂 but I also see the cirros we're using is also using the nearly newest kernel for jammy already so I guess maybe there's little chance a newer version would make a difference | |
| 23:47:04 | sean-k-mooney[m] | yep 6.x is using the 22.04/jammy kernel | |
| 23:47:52 | sean-k-mooney[m] | part of the issue to dans point is that kernel does expect to be deployed in a vm with more ram then we have in our normal guest vms | |
| 23:48:06 | melwitt | sean-k-mooney[m]: this is a paste of the kernel panic if you may be able to tell what it means https://pastebin.com/raw/cVDem0Xr | |
| 23:48:32 | melwitt | I see | |
| 23:51:46 | sean-k-mooney[m] | is that q35 or pc i wonder | |
| 23:51:55 | sean-k-mooney[m] | i cant quite tell from that dmseg output | |
| 23:53:01 | sean-k-mooney[m] | its q35 | |
| 23:53:19 | sean-k-mooney[m] | pcieport 0000:00:02.0: PME: | |