| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-28 | |||
| 16:32:44 | bauzas | elodilles: ping me again tomorrow | |
| 16:33:11 | elodilles | bauzas: ack, thanks in advance! | |
| 16:36:09 | bauzas | ok, next is a new topic | |
| 16:36:11 | bauzas | #topic vmwareapi 3rd-party CI efforts Highlights | |
| 16:36:31 | fwiesel | Shall I? | |
| 16:36:49 | bauzas | fwiesel: shoot :) | |
| 16:36:53 | fwiesel | #Info Gerrit Event Stream connected to our CI (Argo) with personal user | |
| 16:37:27 | fwiesel | So, nothing spectacular. Events go in, they could trigger whatever we want to run later. That part is the more tricky one, and peope are working on it. | |
| 16:38:07 | fwiesel | The most complicated part is the networking, we try to cut it down as much as possible so we test only nova. | |
| 16:38:48 | fwiesel | #Info CI user needs a name. According to naming convention: SAP vmwareapi CI. Has risk of confusion, as SAP is not a vendor of the technology. Our preference simply "SAP CI" or "SAP tempest CI". | |
| 16:39:34 | fwiesel | So, the naming convention on the 3rd party CI suggests "Company Technology CI" | |
| 16:39:37 | bauzas | fwiesel: I'd prefer 'vmwareapi CI run by SAP' | |
| 16:40:02 | bauzas | something telling users what exactly has been run | |
| 16:40:24 | gibi | +1 on keep mentioning vmwareapi in the name | |
| 16:40:26 | fwiesel | That would work. But if we run another CI, say for cinder, then we end up with the same thing as the VMware nsxt CI running cinder | |
| 16:40:27 | bauzas | or yeah, "SAP vmwareapi CI" | |
| 16:41:01 | bauzas | honestly, I don't wanna bikeshed | |
| 16:41:17 | bauzas | I understand that you'd appreciate the SAP name in the CI job name | |
| 16:41:27 | fwiesel | Okay then we go with your proposal "vmwareapi CI run by SAP" | |
| 16:41:34 | bauzas | but I also want my users to know that it's testing with vmwareapi virt driver | |
| 16:41:47 | bauzas | so, please keep both names | |
| 16:41:54 | fwiesel | Fine by me. | |
| 16:41:57 | bauzas | I'm French and I'm bad at namings :) | |
| 16:42:17 | bauzas | but my gut is saying "any words with both SAP and vmwareapi are fine by me" | |
| 16:42:51 | fwiesel | Then we agree on "vmwareapi CI run by SAP"? | |
| 16:43:58 | fwiesel | #agreed The CI user will be named "vmwareapi CI run by SAP" | |
| 16:44:11 | fwiesel | That's from my side. Any questions? | |
| 16:45:21 | fwiesel | Well, then I think we are through. | |
| 16:46:23 | bauzas | ++ | |
| 16:46:38 | bauzas | thanks fwiesel for the report | |
| 16:46:40 | bauzas | :) | |
| 16:46:53 | bauzas | greatly appreciated | |
| 16:46:59 | fwiesel | You're welcome | |
| 16:47:00 | bauzas | last topic then | |
| 16:47:03 | bauzas | #topic Open discussion | |
| 16:47:12 | bauzas | the agenda is empty | |
| 16:47:19 | bauzas | anything anyone ? | |
| 16:48:10 | sean-k-mooney | nope | |
| 16:48:12 | bauzas | cool | |
| 16:48:20 | bauzas | thanks all, giving you back 12 mins | |
| 16:48:22 | bauzas | #endmeeting | |
| 16:48:22 | opendevmeet | Meeting ended Tue Nov 28 16:48:22 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:48:22 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-11-28-16.00.html | |
| 16:48:22 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-11-28-16.00.txt | |
| 16:48:22 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-11-28-16.00.log.html | |
| 16:49:57 | elodilles | thanks bauzas o- | |
| 16:50:00 | elodilles | o/ | |
| 16:50:15 | fwiesel | thanks a lot \o | |
| 19:41:08 | melwitt | dansmith: I have noticed (anecdotally) that a large portion of gate failures have been due to guest kernel panics. I have spent some time looking at them and see for example "pci 0000:00:03.6: BAR 13: failed to assign [io size 0x1000]" and "pci 0000:00:03.6: BAR 13: no space for [io size 0x1000]" I'm not sure how to tell which thing caused the panic. and in my search for more info online I haven't found anything other than mention of | |
| 19:41:08 | melwitt | old kernel bugs or people saying to run fsck and try again. not sure how to approach this, so if you have any thoughts, it would probably help | |
| 19:42:16 | melwitt | it seems weird to me how frequent this is. don't know what that could mean either. in the past guest kernel panics were a lot more rare | |
| 19:46:58 | dansmith | orly, well, that might explain the uptick lately... but yeah, I'd lean on kashyap and sean-k-mooney to look at those sorts of things | |
| 19:47:28 | dansmith | unfortunately, we've had trouble getting our kernel team to look at those things because we don't really run a supported kernel, but with the recent cirros update, we're closer to a support-able kernel I think | |
| 19:47:31 | dansmith | so might be worth trying again | |
| 19:49:03 | melwitt | ok, yeah. I checked and we are running the newest kernel that cirros supports. I did wonder if there could be a chance for improvement by using something other than cirros, like to get a different/newer kernel version | |
| 19:50:05 | melwitt | or I guess maybe building our own cirros with a newer kernel | |
| 20:07:37 | dansmith | I mean, it's possible but cirros is also tailored to work in very small environments, which is important for us | |
| 20:07:56 | dansmith | a stock distro kernel (and certainly image) won't likely work well in our super tiny guests | |
| 20:08:06 | dansmith | and if we had to increase the size of those, we'd need to make other changes | |
| 20:08:19 | dansmith | getting some kernel person to tell us what the crashes mean would be a good first start | |
| 20:08:40 | dansmith | like if it's "that should never happen" or "they're already memory constrained and hitting a bug as a result" or whatever | |
| 20:30:00 | melwitt | yeah | |
| 21:58:15 | opendevreview | Merged openstack/python-novaclient master: Fix typos https://review.opendev.org/c/openstack/python-novaclient/+/901457 | |
| 23:44:42 | sean-k-mooney[m] | melwitt: cirrus uses the ubuntu lts kernel just in case you didnt know that | |
| 23:45:48 | sean-k-mooney[m] | i really should try and make time to get my alpine serise working again but honestly havnt had time or energy to do that. | |
| 23:46:36 | melwitt | sean-k-mooney[m]: I saw on their site 🙂 but I also see the cirros we're using is also using the nearly newest kernel for jammy already so I guess maybe there's little chance a newer version would make a difference | |
| 23:46:36 | melwitt | sean-k-mooney[m]: I saw on their site 🙂 but I also see the cirros we're using is also using the nearly newest kernel for jammy already so I guess maybe there's little chance a newer version would make a difference | |
| 23:47:04 | sean-k-mooney[m] | yep 6.x is using the 22.04/jammy kernel | |
| 23:47:52 | sean-k-mooney[m] | part of the issue to dans point is that kernel does expect to be deployed in a vm with more ram then we have in our normal guest vms | |
| 23:48:06 | melwitt | sean-k-mooney[m]: this is a paste of the kernel panic if you may be able to tell what it means https://pastebin.com/raw/cVDem0Xr | |
| 23:48:32 | melwitt | I see | |
| 23:51:46 | sean-k-mooney[m] | is that q35 or pc i wonder | |
| 23:51:55 | sean-k-mooney[m] | i cant quite tell from that dmseg output | |
| 23:53:01 | sean-k-mooney[m] | its q35 | |
| 23:53:19 | sean-k-mooney[m] | pcieport 0000:00:02.0: PME: | |
| 23:53:33 | sean-k-mooney[m] | so i wonder if we could mitigate this by plugging less pcie ports | |
| 23:54:07 | melwitt | it's from this nova-next fail https://0c0c4e33641a15d6198e-9e6a8447d6325ce608287fa851bc53b0.ssl.cf2.rackcdn.com/892800/3/gate/nova-next/4b8b450/testr_results.html | |
| 23:55:14 | sean-k-mooney[m] | ya so nova next is where we currently test q35 | |
| 23:55:20 | sean-k-mooney[m] | and its the only job that uses q35 | |
| 23:55:35 | clarkb | sean-k-mooney[m]: I wasn't aware the linux kernel publishes minimum memory requirements | |
| 23:55:51 | clarkb | and I definitely have run linux in less memory than the ci test nodes... | |
| 23:56:10 | sean-k-mooney[m] | clarkb linux does not canonical do for there images | |
| 23:56:37 | sean-k-mooney[m] | and ya puppy linux or tinycore or alpine | |
| 23:57:00 | sean-k-mooney[m] | all support much less but i think the lowest canonical support is 256 | |
| 23:57:26 | sean-k-mooney[m] | maybe 128. that not just for the kernel of couse that for the full image with user space | |
| 23:57:46 | melwitt | it's also happening in other jobs but maybe for different reasons? example https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fe3/898188/16/check/tempest-integrated-storage/fe33fa7/testr_results.html | |
| 23:58:23 | melwitt | that ^ is not q35 | |
| 23:59:05 | JayF | We run tinycore for Ironic test VMs, and already have automation for building test images, if that's something you all have interest in. | |
| 23:59:23 | JayF | https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/tinyipa | |
| 23:59:50 | sean-k-mooney[m] | https://ubuntu.com/core/docs/system-requirements so 256mb is there minim for there smallest image | |
| 23:59:50 | JayF | Obviously we have to put more in there than you all would, but if you go that route there's no need to stop from zero. | |
| 23:59:53 | JayF | s/stop/start/ | |
| #openstack-nova - 2023-11-29 | |||
| 00:01:02 | sean-k-mooney[m] | well its not exactly form 0 since i starded it like a year ago https://review.opendev.org/q/topic:%22alpine%22 | |
| 00:01:29 | sean-k-mooney[m] | but ya we dont use much in the images an honestly i kind of wish tempest only used busybox or similar tools | |
| 00:01:47 | sean-k-mooney[m] | tineycore might be a bitt too spartan but it would be worth a try | |
| 00:02:06 | sean-k-mooney[m] | my alpine image does not have what needed for growpart | |
| 00:02:08 | clarkb | fwiw I don't expect the ubuntu kernel built for cirros to be that drastically different than another tiny kernel. They use the same source but rebuild to make it smaller and lightweight iirc | |
| 00:02:13 | sean-k-mooney[m] | so that was causing things to fail | |
| 00:02:36 | JayF | alpine does use musl libc, which can cause changes in behavior FWIW; but as I say that I realize I don't know if tinycore uses glibc :) | |
| 00:02:47 | sean-k-mooney[m] | clarkb: hum ok i tought the just unpacked the deb file without rebuilding | |