| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-28 | |||
| 23:47:52 | sean-k-mooney[m] | part of the issue to dans point is that kernel does expect to be deployed in a vm with more ram then we have in our normal guest vms | |
| 23:48:06 | melwitt | sean-k-mooney[m]: this is a paste of the kernel panic if you may be able to tell what it means https://pastebin.com/raw/cVDem0Xr | |
| 23:48:32 | melwitt | I see | |
| 23:51:46 | sean-k-mooney[m] | is that q35 or pc i wonder | |
| 23:51:55 | sean-k-mooney[m] | i cant quite tell from that dmseg output | |
| 23:53:01 | sean-k-mooney[m] | its q35 | |
| 23:53:19 | sean-k-mooney[m] | pcieport 0000:00:02.0: PME: | |
| 23:53:33 | sean-k-mooney[m] | so i wonder if we could mitigate this by plugging less pcie ports | |
| 23:54:07 | melwitt | it's from this nova-next fail https://0c0c4e33641a15d6198e-9e6a8447d6325ce608287fa851bc53b0.ssl.cf2.rackcdn.com/892800/3/gate/nova-next/4b8b450/testr_results.html | |
| 23:55:14 | sean-k-mooney[m] | ya so nova next is where we currently test q35 | |
| 23:55:20 | sean-k-mooney[m] | and its the only job that uses q35 | |
| 23:55:35 | clarkb | sean-k-mooney[m]: I wasn't aware the linux kernel publishes minimum memory requirements | |
| 23:55:51 | clarkb | and I definitely have run linux in less memory than the ci test nodes... | |
| 23:56:10 | sean-k-mooney[m] | clarkb linux does not canonical do for there images | |
| 23:56:37 | sean-k-mooney[m] | and ya puppy linux or tinycore or alpine | |
| 23:57:00 | sean-k-mooney[m] | all support much less but i think the lowest canonical support is 256 | |
| 23:57:26 | sean-k-mooney[m] | maybe 128. that not just for the kernel of couse that for the full image with user space | |
| 23:57:46 | melwitt | it's also happening in other jobs but maybe for different reasons? example https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fe3/898188/16/check/tempest-integrated-storage/fe33fa7/testr_results.html | |
| 23:58:23 | melwitt | that ^ is not q35 | |
| 23:59:05 | JayF | We run tinycore for Ironic test VMs, and already have automation for building test images, if that's something you all have interest in. | |
| 23:59:23 | JayF | https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/tinyipa | |
| 23:59:50 | sean-k-mooney[m] | https://ubuntu.com/core/docs/system-requirements so 256mb is there minim for there smallest image | |
| 23:59:50 | JayF | Obviously we have to put more in there than you all would, but if you go that route there's no need to stop from zero. | |
| 23:59:53 | JayF | s/stop/start/ | |
| #openstack-nova - 2023-11-29 | |||
| 00:01:02 | sean-k-mooney[m] | well its not exactly form 0 since i starded it like a year ago https://review.opendev.org/q/topic:%22alpine%22 | |
| 00:01:29 | sean-k-mooney[m] | but ya we dont use much in the images an honestly i kind of wish tempest only used busybox or similar tools | |
| 00:01:47 | sean-k-mooney[m] | tineycore might be a bitt too spartan but it would be worth a try | |
| 00:02:06 | sean-k-mooney[m] | my alpine image does not have what needed for growpart | |
| 00:02:08 | clarkb | fwiw I don't expect the ubuntu kernel built for cirros to be that drastically different than another tiny kernel. They use the same source but rebuild to make it smaller and lightweight iirc | |
| 00:02:13 | sean-k-mooney[m] | so that was causing things to fail | |
| 00:02:36 | JayF | alpine does use musl libc, which can cause changes in behavior FWIW; but as I say that I realize I don't know if tinycore uses glibc :) | |
| 00:02:47 | sean-k-mooney[m] | clarkb: hum ok i tought the just unpacked the deb file without rebuilding | |
| 00:03:05 | clarkb | sean-k-mooney[m]: there is a whole massive build process for those images. I thought they were rebuilding the kernels. frickler can probably confirm or deny | |
| 00:03:13 | sean-k-mooney[m] | JayF: the use of musl is part of why alpine was apealing | |
| 00:03:33 | clarkb | sean-k-mooney[m]: fwiw I think the ubuntu kernel packages hosted for ubuntu are several times larger than an entire cirros image | |
| 00:03:51 | JayF | Ah. I would consider it quite the opposite since all our other testing is on glibc, and most target distributions for openstack are glibc. Seems like a place where we could find weird breakages that we might not be well-suited to fix. | |
| 00:03:55 | sean-k-mooney[m] | clarkb: ack, in this case the kernel messages seam to be indicating that we are runnign out of memeory for memroy mapped io to allocte to the pcie devices | |
| 00:03:56 | clarkb | maybe even an entire order of magnitude | |
| 00:04:16 | clarkb | JayF: also python runs poorly on musl | |
| 00:04:26 | clarkb | though most of our jobs don't really need to run python in the nested VMs | |
| 00:04:26 | sean-k-mooney[m] | JayF: nothign about openstack should care if the guest has glibc or not | |
| 00:04:33 | sean-k-mooney[m] | if it does that a problem we should fix | |
| 00:04:41 | JayF | sean-k-mooney[m]: aha, you don't run anything inside the images; Ironic does, I forgot that :D | |
| 00:05:08 | sean-k-mooney[m] | we basically use touch, ls ssh and a few other very minimal commands | |
| 00:05:29 | sean-k-mooney[m] | hench why busybox + mkfs.fat32 | |
| 00:05:34 | sean-k-mooney[m] | woudl cover most things | |
| 00:05:44 | clarkb | I think the most complicated stuff we do in the nodes is realted to networking to confirm network functionality | |
| 00:05:48 | sean-k-mooney[m] | assuming glean or cloud-init work for metadata/ssh keys | |
| 00:06:04 | JayF | sean-k-mooney[m]: you really should see how far you could get with commenting out 99% of our tinyipa build script and tossing the resulting image into a CI job | |
| 00:06:37 | JayF | I guess really the hard part is CI infra, and it wouldn't be able too reuse hte IPA-B stuff because it's not really IPA-adjacent | |
| 00:07:19 | JayF | https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/tinyipa/build-tinyipa.sh is the entrypoint for our tinyipa ramdisk build, fwiw | |
| 00:07:57 | sean-k-mooney[m] | why is it pulling from linux.dell.com? | |
| 00:08:15 | JayF | to get a source release of biosdevname, which isn't shipped in tinyipa | |
| 00:08:32 | sean-k-mooney[m] | ya but i assume that is not the upstream source repo for that | |
| 00:08:41 | JayF | you actually assume wrong, surpringly | |
| 00:08:48 | JayF | but that was basically my knee jerk review feedback too LOL | |
| 00:09:54 | sean-k-mooney[m] | hum the more you know | |
| 00:10:06 | sean-k-mooney[m] | does the ipa image have glean or cloud init? | |
| 00:10:33 | JayF | for purposes of tinyipa, I think we only test that on dhcp images, but I'm not 100% sure | |
| 00:10:37 | JayF | rpittau is the expert on those | |
| 00:13:00 | sean-k-mooney[m] | it looks like its configuing dhcp using udhcp ya | |
| 00:13:21 | JayF | sean-k-mooney[m]: yep; based on our docs I'd assume we don't glean/cloud-init it outta the box (but I'll note that TheJulia worked on upstream glean support for tinycore so it's likely just automation away): https://docs.openstack.org/ironic-python-agent-builder/latest/admin/tinyipa.html#enabling-disabling-ssh-access-to-the-ramdisk | |
| 00:13:24 | sean-k-mooney[m] | so the gap would be that iamge today does not ahve anythign to reach out and download the ssh key form metadata so that we can log in | |
| 00:13:34 | JayF | our relationship with glean in IPA is weird, bceause we run it on demand, not automatically | |
| 00:14:22 | sean-k-mooney[m] | we could fall back to password login but tempest asssumes a “cloud image” i.e. one with a cloud-init equivalent | |
| 00:14:36 | JayF | you literally just need to wait for https://review.opendev.org/c/opendev/glean/+/899219 to land | |
| 00:14:43 | JayF | and add a stanza to install glean in the image | |
| 00:15:42 | sean-k-mooney[m] | good to know. | |
| 00:15:53 | JayF | I'll summarize this all with: I don't care what you use | |
| 00:15:59 | JayF | I don't even know why we used tinycore for this image | |
| 00:16:08 | JayF | but we've used it literally for years and years, it mostly just works | |
| 00:16:25 | JayF | only real downside is the longstanding open bug we have where tinycore just does not want to host their sources over https | |
| 00:16:40 | JayF | which is infuriating, but ... acceptably infuriating for a CI-only tool | |
| 00:16:44 | TheJulia | Yeah, and that is huge sadness | |
| 00:18:04 | sean-k-mooney[m] | melwitt: so in nova next we allcoate 24 pci root ports https://opendev.org/openstack/nova/src/branch/master/.zuul.yaml#L414 we should drop that to something more reasonable like 12-16 | |
| 00:19:03 | sean-k-mooney[m] | that will signigincatly reduce the pci mmio space requried in the guest | |
| 00:19:12 | melwitt | sean-k-mooney[m]: ok, sweet. glad there is something we can try | |
| 00:39:09 | sean-k-mooney[m] | im not entirly sure that the actuall issue is the io space asignment. looking at the kernel trace the error seams to be coming form an irq interupt handeler but its also referencing ? exc_page_fault+0x89/0x170 so i would wonder if this is still a memory issue | |
| 00:39:13 | sean-k-mooney[m] | its saying Kernel panic - not syncing: Attempted to kill init! | |
| 00:39:22 | sean-k-mooney[m] | as the main panic message | |
| 00:39:47 | sean-k-mooney[m] | and it looks like we re loading form the initramdisk to the root file system at the time the interup was happening | |
| 00:40:26 | sean-k-mooney[m] | so im just wondering if the interupt handler is trying to allocate memroy at a time we are very close to the limit and failling ocationally | |
| 00:41:14 | sean-k-mooney[m] | if that is the case allocatedign a very small amount of swap in the nova flavor might also help but reducing the pcie root ports would still be my first step | |
| 00:41:56 | sean-k-mooney[m] | i think each unused pci root port is using like a mb of ram so 24 mb or our 128 total just for the pcie slot | |
| 00:42:15 | sean-k-mooney[m] | and we dont actully use more hten a hand full of them for volumes/ports | |
| 00:42:35 | melwitt | ok, I see | |
| 00:44:23 | sean-k-mooney[m] | the last messages before the panic are | |
| 00:44:26 | sean-k-mooney[m] | info: initramfs loading root from /dev/vda1 | |
| 00:44:26 | sean-k-mooney[m] | /sbin/init: can't load library 'libtirpc.so.3' | |
| 00:44:59 | melwitt | I noticed that too but didn't know what it means | |
| 00:46:18 | melwitt | that message is also just before the panic in the non-q35 job | |
| 00:46:24 | sean-k-mooney[m] | so this is where we are changign form runing in the kernel ramdisk to running form the actual root disk i think and in this case init (which i belive is sysvinit not systmed in this case) is loading a dynmaic lib into memory | |
| 00:46:36 | melwitt | maybe the common denominator? | |
| 00:48:00 | sean-k-mooney[m] | well if we get an interupt at that point im guessing the kernel oom killer tries to kill init. ill admit this si a bit beyond my understanding of early boot so im not sure either | |
| 00:49:17 | sean-k-mooney[m] | neutron are seeing the same error for what its worth | |
| 00:49:18 | sean-k-mooney[m] | https://bugs.launchpad.net/neutron/+bug/2039940 | |
| 00:53:20 | sean-k-mooney[m] | ya cirros has there own init script https://github.com/cirros-dev/cirros/blob/main/src/init | |
| 00:54:44 | sean-k-mooney[m] | its failing here in the switch root call l https://github.com/cirros-dev/cirros/blob/main/src/init#L86 | |
| 00:56:52 | melwitt | nice find | |
| 00:57:34 | sean-k-mooney[m] | i feel like this is busybox init | |