| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-28 | |||
| 16:48:22 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-11-28-16.00.html | |
| 16:48:22 | opendevmeet | Meeting ended Tue Nov 28 16:48:22 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:48:22 | bauzas | #endmeeting | |
| 16:49:57 | elodilles | thanks bauzas o- | |
| 16:50:00 | elodilles | o/ | |
| 16:50:15 | fwiesel | thanks a lot \o | |
| 19:41:08 | melwitt | old kernel bugs or people saying to run fsck and try again. not sure how to approach this, so if you have any thoughts, it would probably help | |
| 19:41:08 | melwitt | dansmith: I have noticed (anecdotally) that a large portion of gate failures have been due to guest kernel panics. I have spent some time looking at them and see for example "pci 0000:00:03.6: BAR 13: failed to assign [io size 0x1000]" and "pci 0000:00:03.6: BAR 13: no space for [io size 0x1000]" I'm not sure how to tell which thing caused the panic. and in my search for more info online I haven't found anything other than mention of | |
| 19:42:16 | melwitt | it seems weird to me how frequent this is. don't know what that could mean either. in the past guest kernel panics were a lot more rare | |
| 19:46:58 | dansmith | orly, well, that might explain the uptick lately... but yeah, I'd lean on kashyap and sean-k-mooney to look at those sorts of things | |
| 19:47:28 | dansmith | unfortunately, we've had trouble getting our kernel team to look at those things because we don't really run a supported kernel, but with the recent cirros update, we're closer to a support-able kernel I think | |
| 19:47:31 | dansmith | so might be worth trying again | |
| 19:49:03 | melwitt | ok, yeah. I checked and we are running the newest kernel that cirros supports. I did wonder if there could be a chance for improvement by using something other than cirros, like to get a different/newer kernel version | |
| 19:50:05 | melwitt | or I guess maybe building our own cirros with a newer kernel | |
| 20:07:37 | dansmith | I mean, it's possible but cirros is also tailored to work in very small environments, which is important for us | |
| 20:07:56 | dansmith | a stock distro kernel (and certainly image) won't likely work well in our super tiny guests | |
| 20:08:06 | dansmith | and if we had to increase the size of those, we'd need to make other changes | |
| 20:08:19 | dansmith | getting some kernel person to tell us what the crashes mean would be a good first start | |
| 20:08:40 | dansmith | like if it's "that should never happen" or "they're already memory constrained and hitting a bug as a result" or whatever | |
| 20:30:00 | melwitt | yeah | |
| 21:58:15 | opendevreview | Merged openstack/python-novaclient master: Fix typos https://review.opendev.org/c/openstack/python-novaclient/+/901457 | |
| 23:44:42 | sean-k-mooney[m] | melwitt: cirrus uses the ubuntu lts kernel just in case you didnt know that | |
| 23:45:48 | sean-k-mooney[m] | i really should try and make time to get my alpine serise working again but honestly havnt had time or energy to do that. | |
| 23:46:36 | melwitt | sean-k-mooney[m]: I saw on their site 🙂 but I also see the cirros we're using is also using the nearly newest kernel for jammy already so I guess maybe there's little chance a newer version would make a difference | |
| 23:46:36 | melwitt | sean-k-mooney[m]: I saw on their site 🙂 but I also see the cirros we're using is also using the nearly newest kernel for jammy already so I guess maybe there's little chance a newer version would make a difference | |
| 23:47:04 | sean-k-mooney[m] | yep 6.x is using the 22.04/jammy kernel | |
| 23:47:52 | sean-k-mooney[m] | part of the issue to dans point is that kernel does expect to be deployed in a vm with more ram then we have in our normal guest vms | |
| 23:48:06 | melwitt | sean-k-mooney[m]: this is a paste of the kernel panic if you may be able to tell what it means https://pastebin.com/raw/cVDem0Xr | |
| 23:48:32 | melwitt | I see | |
| 23:51:46 | sean-k-mooney[m] | is that q35 or pc i wonder | |
| 23:51:55 | sean-k-mooney[m] | i cant quite tell from that dmseg output | |
| 23:53:01 | sean-k-mooney[m] | its q35 | |
| 23:53:19 | sean-k-mooney[m] | pcieport 0000:00:02.0: PME: | |
| 23:53:33 | sean-k-mooney[m] | so i wonder if we could mitigate this by plugging less pcie ports | |
| 23:54:07 | melwitt | it's from this nova-next fail https://0c0c4e33641a15d6198e-9e6a8447d6325ce608287fa851bc53b0.ssl.cf2.rackcdn.com/892800/3/gate/nova-next/4b8b450/testr_results.html | |
| 23:55:14 | sean-k-mooney[m] | ya so nova next is where we currently test q35 | |
| 23:55:20 | sean-k-mooney[m] | and its the only job that uses q35 | |
| 23:55:35 | clarkb | sean-k-mooney[m]: I wasn't aware the linux kernel publishes minimum memory requirements | |
| 23:55:51 | clarkb | and I definitely have run linux in less memory than the ci test nodes... | |
| 23:56:10 | sean-k-mooney[m] | clarkb linux does not canonical do for there images | |
| 23:56:37 | sean-k-mooney[m] | and ya puppy linux or tinycore or alpine | |
| 23:57:00 | sean-k-mooney[m] | all support much less but i think the lowest canonical support is 256 | |
| 23:57:26 | sean-k-mooney[m] | maybe 128. that not just for the kernel of couse that for the full image with user space | |
| 23:57:46 | melwitt | it's also happening in other jobs but maybe for different reasons? example https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fe3/898188/16/check/tempest-integrated-storage/fe33fa7/testr_results.html | |
| 23:58:23 | melwitt | that ^ is not q35 | |
| 23:59:05 | JayF | We run tinycore for Ironic test VMs, and already have automation for building test images, if that's something you all have interest in. | |
| 23:59:23 | JayF | https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/tinyipa | |
| 23:59:50 | JayF | Obviously we have to put more in there than you all would, but if you go that route there's no need to stop from zero. | |
| 23:59:50 | sean-k-mooney[m] | https://ubuntu.com/core/docs/system-requirements so 256mb is there minim for there smallest image | |
| 23:59:53 | JayF | s/stop/start/ | |
| #openstack-nova - 2023-11-29 | |||
| 00:01:02 | sean-k-mooney[m] | well its not exactly form 0 since i starded it like a year ago https://review.opendev.org/q/topic:%22alpine%22 | |
| 00:01:29 | sean-k-mooney[m] | but ya we dont use much in the images an honestly i kind of wish tempest only used busybox or similar tools | |
| 00:01:47 | sean-k-mooney[m] | tineycore might be a bitt too spartan but it would be worth a try | |
| 00:02:06 | sean-k-mooney[m] | my alpine image does not have what needed for growpart | |
| 00:02:08 | clarkb | fwiw I don't expect the ubuntu kernel built for cirros to be that drastically different than another tiny kernel. They use the same source but rebuild to make it smaller and lightweight iirc | |
| 00:02:13 | sean-k-mooney[m] | so that was causing things to fail | |
| 00:02:36 | JayF | alpine does use musl libc, which can cause changes in behavior FWIW; but as I say that I realize I don't know if tinycore uses glibc :) | |
| 00:02:47 | sean-k-mooney[m] | clarkb: hum ok i tought the just unpacked the deb file without rebuilding | |
| 00:03:05 | clarkb | sean-k-mooney[m]: there is a whole massive build process for those images. I thought they were rebuilding the kernels. frickler can probably confirm or deny | |
| 00:03:13 | sean-k-mooney[m] | JayF: the use of musl is part of why alpine was apealing | |
| 00:03:33 | clarkb | sean-k-mooney[m]: fwiw I think the ubuntu kernel packages hosted for ubuntu are several times larger than an entire cirros image | |
| 00:03:51 | JayF | Ah. I would consider it quite the opposite since all our other testing is on glibc, and most target distributions for openstack are glibc. Seems like a place where we could find weird breakages that we might not be well-suited to fix. | |
| 00:03:55 | sean-k-mooney[m] | clarkb: ack, in this case the kernel messages seam to be indicating that we are runnign out of memeory for memroy mapped io to allocte to the pcie devices | |
| 00:03:56 | clarkb | maybe even an entire order of magnitude | |
| 00:04:16 | clarkb | JayF: also python runs poorly on musl | |
| 00:04:26 | sean-k-mooney[m] | JayF: nothign about openstack should care if the guest has glibc or not | |
| 00:04:26 | clarkb | though most of our jobs don't really need to run python in the nested VMs | |
| 00:04:33 | sean-k-mooney[m] | if it does that a problem we should fix | |
| 00:04:41 | JayF | sean-k-mooney[m]: aha, you don't run anything inside the images; Ironic does, I forgot that :D | |
| 00:05:08 | sean-k-mooney[m] | we basically use touch, ls ssh and a few other very minimal commands | |
| 00:05:29 | sean-k-mooney[m] | hench why busybox + mkfs.fat32 | |
| 00:05:34 | sean-k-mooney[m] | woudl cover most things | |
| 00:05:44 | clarkb | I think the most complicated stuff we do in the nodes is realted to networking to confirm network functionality | |
| 00:05:48 | sean-k-mooney[m] | assuming glean or cloud-init work for metadata/ssh keys | |
| 00:06:04 | JayF | sean-k-mooney[m]: you really should see how far you could get with commenting out 99% of our tinyipa build script and tossing the resulting image into a CI job | |
| 00:06:37 | JayF | I guess really the hard part is CI infra, and it wouldn't be able too reuse hte IPA-B stuff because it's not really IPA-adjacent | |
| 00:07:19 | JayF | https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/tinyipa/build-tinyipa.sh is the entrypoint for our tinyipa ramdisk build, fwiw | |
| 00:07:57 | sean-k-mooney[m] | why is it pulling from linux.dell.com? | |
| 00:08:15 | JayF | to get a source release of biosdevname, which isn't shipped in tinyipa | |
| 00:08:32 | sean-k-mooney[m] | ya but i assume that is not the upstream source repo for that | |
| 00:08:41 | JayF | you actually assume wrong, surpringly | |
| 00:08:48 | JayF | but that was basically my knee jerk review feedback too LOL | |
| 00:09:54 | sean-k-mooney[m] | hum the more you know | |
| 00:10:06 | sean-k-mooney[m] | does the ipa image have glean or cloud init? | |
| 00:10:33 | JayF | for purposes of tinyipa, I think we only test that on dhcp images, but I'm not 100% sure | |
| 00:10:37 | JayF | rpittau is the expert on those | |
| 00:13:00 | sean-k-mooney[m] | it looks like its configuing dhcp using udhcp ya | |
| 00:13:21 | JayF | sean-k-mooney[m]: yep; based on our docs I'd assume we don't glean/cloud-init it outta the box (but I'll note that TheJulia worked on upstream glean support for tinycore so it's likely just automation away): https://docs.openstack.org/ironic-python-agent-builder/latest/admin/tinyipa.html#enabling-disabling-ssh-access-to-the-ramdisk | |
| 00:13:24 | sean-k-mooney[m] | so the gap would be that iamge today does not ahve anythign to reach out and download the ssh key form metadata so that we can log in | |
| 00:13:34 | JayF | our relationship with glean in IPA is weird, bceause we run it on demand, not automatically | |
| 00:14:22 | sean-k-mooney[m] | we could fall back to password login but tempest asssumes a “cloud image” i.e. one with a cloud-init equivalent | |
| 00:14:36 | JayF | you literally just need to wait for https://review.opendev.org/c/opendev/glean/+/899219 to land | |
| 00:14:43 | JayF | and add a stanza to install glean in the image | |
| 00:15:42 | sean-k-mooney[m] | good to know. | |
| 00:15:53 | JayF | I'll summarize this all with: I don't care what you use | |
| 00:15:59 | JayF | I don't even know why we used tinycore for this image | |
| 00:16:08 | JayF | but we've used it literally for years and years, it mostly just works | |
| 00:16:25 | JayF | only real downside is the longstanding open bug we have where tinycore just does not want to host their sources over https | |
| 00:16:40 | JayF | which is infuriating, but ... acceptably infuriating for a CI-only tool | |
| 00:16:44 | TheJulia | Yeah, and that is huge sadness | |