| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-29 | |||
| 00:04:26 | sean-k-mooney[m] | JayF: nothign about openstack should care if the guest has glibc or not | |
| 00:04:26 | clarkb | though most of our jobs don't really need to run python in the nested VMs | |
| 00:04:33 | sean-k-mooney[m] | if it does that a problem we should fix | |
| 00:04:41 | JayF | sean-k-mooney[m]: aha, you don't run anything inside the images; Ironic does, I forgot that :D | |
| 00:05:08 | sean-k-mooney[m] | we basically use touch, ls ssh and a few other very minimal commands | |
| 00:05:29 | sean-k-mooney[m] | hench why busybox + mkfs.fat32 | |
| 00:05:34 | sean-k-mooney[m] | woudl cover most things | |
| 00:05:44 | clarkb | I think the most complicated stuff we do in the nodes is realted to networking to confirm network functionality | |
| 00:05:48 | sean-k-mooney[m] | assuming glean or cloud-init work for metadata/ssh keys | |
| 00:06:04 | JayF | sean-k-mooney[m]: you really should see how far you could get with commenting out 99% of our tinyipa build script and tossing the resulting image into a CI job | |
| 00:06:37 | JayF | I guess really the hard part is CI infra, and it wouldn't be able too reuse hte IPA-B stuff because it's not really IPA-adjacent | |
| 00:07:19 | JayF | https://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/tinyipa/build-tinyipa.sh is the entrypoint for our tinyipa ramdisk build, fwiw | |
| 00:07:57 | sean-k-mooney[m] | why is it pulling from linux.dell.com? | |
| 00:08:15 | JayF | to get a source release of biosdevname, which isn't shipped in tinyipa | |
| 00:08:32 | sean-k-mooney[m] | ya but i assume that is not the upstream source repo for that | |
| 00:08:41 | JayF | you actually assume wrong, surpringly | |
| 00:08:48 | JayF | but that was basically my knee jerk review feedback too LOL | |
| 00:09:54 | sean-k-mooney[m] | hum the more you know | |
| 00:10:06 | sean-k-mooney[m] | does the ipa image have glean or cloud init? | |
| 00:10:33 | JayF | for purposes of tinyipa, I think we only test that on dhcp images, but I'm not 100% sure | |
| 00:10:37 | JayF | rpittau is the expert on those | |
| 00:13:00 | sean-k-mooney[m] | it looks like its configuing dhcp using udhcp ya | |
| 00:13:21 | JayF | sean-k-mooney[m]: yep; based on our docs I'd assume we don't glean/cloud-init it outta the box (but I'll note that TheJulia worked on upstream glean support for tinycore so it's likely just automation away): https://docs.openstack.org/ironic-python-agent-builder/latest/admin/tinyipa.html#enabling-disabling-ssh-access-to-the-ramdisk | |
| 00:13:24 | sean-k-mooney[m] | so the gap would be that iamge today does not ahve anythign to reach out and download the ssh key form metadata so that we can log in | |
| 00:13:34 | JayF | our relationship with glean in IPA is weird, bceause we run it on demand, not automatically | |
| 00:14:22 | sean-k-mooney[m] | we could fall back to password login but tempest asssumes a “cloud image” i.e. one with a cloud-init equivalent | |
| 00:14:36 | JayF | you literally just need to wait for https://review.opendev.org/c/opendev/glean/+/899219 to land | |
| 00:14:43 | JayF | and add a stanza to install glean in the image | |
| 00:15:42 | sean-k-mooney[m] | good to know. | |
| 00:15:53 | JayF | I'll summarize this all with: I don't care what you use | |
| 00:15:59 | JayF | I don't even know why we used tinycore for this image | |
| 00:16:08 | JayF | but we've used it literally for years and years, it mostly just works | |
| 00:16:25 | JayF | only real downside is the longstanding open bug we have where tinycore just does not want to host their sources over https | |
| 00:16:40 | JayF | which is infuriating, but ... acceptably infuriating for a CI-only tool | |
| 00:16:44 | TheJulia | Yeah, and that is huge sadness | |
| 00:18:04 | sean-k-mooney[m] | melwitt: so in nova next we allcoate 24 pci root ports https://opendev.org/openstack/nova/src/branch/master/.zuul.yaml#L414 we should drop that to something more reasonable like 12-16 | |
| 00:19:03 | sean-k-mooney[m] | that will signigincatly reduce the pci mmio space requried in the guest | |
| 00:19:12 | melwitt | sean-k-mooney[m]: ok, sweet. glad there is something we can try | |
| 00:39:09 | sean-k-mooney[m] | im not entirly sure that the actuall issue is the io space asignment. looking at the kernel trace the error seams to be coming form an irq interupt handeler but its also referencing ? exc_page_fault+0x89/0x170 so i would wonder if this is still a memory issue | |
| 00:39:13 | sean-k-mooney[m] | its saying Kernel panic - not syncing: Attempted to kill init! | |
| 00:39:22 | sean-k-mooney[m] | as the main panic message | |
| 00:39:47 | sean-k-mooney[m] | and it looks like we re loading form the initramdisk to the root file system at the time the interup was happening | |
| 00:40:26 | sean-k-mooney[m] | so im just wondering if the interupt handler is trying to allocate memroy at a time we are very close to the limit and failling ocationally | |
| 00:41:14 | sean-k-mooney[m] | if that is the case allocatedign a very small amount of swap in the nova flavor might also help but reducing the pcie root ports would still be my first step | |
| 00:41:56 | sean-k-mooney[m] | i think each unused pci root port is using like a mb of ram so 24 mb or our 128 total just for the pcie slot | |
| 00:42:15 | sean-k-mooney[m] | and we dont actully use more hten a hand full of them for volumes/ports | |
| 00:42:35 | melwitt | ok, I see | |
| 00:44:23 | sean-k-mooney[m] | the last messages before the panic are | |
| 00:44:26 | sean-k-mooney[m] | /sbin/init: can't load library 'libtirpc.so.3' | |
| 00:44:26 | sean-k-mooney[m] | info: initramfs loading root from /dev/vda1 | |
| 00:44:59 | melwitt | I noticed that too but didn't know what it means | |
| 00:46:18 | melwitt | that message is also just before the panic in the non-q35 job | |
| 00:46:24 | sean-k-mooney[m] | so this is where we are changign form runing in the kernel ramdisk to running form the actual root disk i think and in this case init (which i belive is sysvinit not systmed in this case) is loading a dynmaic lib into memory | |
| 00:46:36 | melwitt | maybe the common denominator? | |
| 00:48:00 | sean-k-mooney[m] | well if we get an interupt at that point im guessing the kernel oom killer tries to kill init. ill admit this si a bit beyond my understanding of early boot so im not sure either | |
| 00:49:17 | sean-k-mooney[m] | neutron are seeing the same error for what its worth | |
| 00:49:18 | sean-k-mooney[m] | https://bugs.launchpad.net/neutron/+bug/2039940 | |
| 00:53:20 | sean-k-mooney[m] | ya cirros has there own init script https://github.com/cirros-dev/cirros/blob/main/src/init | |
| 00:54:44 | sean-k-mooney[m] | its failing here in the switch root call l https://github.com/cirros-dev/cirros/blob/main/src/init#L86 | |
| 00:56:52 | melwitt | nice find | |
| 00:57:34 | sean-k-mooney[m] | i feel like this is busybox init | |
| 00:58:02 | sean-k-mooney[m] | or rather its calling into busybox init | |
| 00:58:26 | sean-k-mooney[m] | based on https://github.com/cirros-dev/cirros/blob/46a1162787f669ad8d6065cb6bbe477654b4327f/conf/busybox.config#L701 | |
| 00:58:56 | sean-k-mooney[m] | i belive switch_root is at least being provided by busybox even if we are not using busy box init directly | |
| 01:04:15 | sean-k-mooney[m] | anyway im going to go to sleep now o/ | |
| 01:05:11 | melwitt | sean-k-mooney[m]: thanks for helping with this, have a good night o/ | |
| 01:20:18 | melwitt | (leaving a message for tomorrow) bauzas, gibi, sean-k-mooney: I made a list of some of the CI failures I saw on one of dan's patches to help me keep track and linked to this ^ discussion ... https://etherpad.opendev.org/p/nova-ci-failures-minimal in case anyone might be interested | |
| 01:23:18 | melwitt | I made a new etherpad bc I have trouble parsing etherpads with larger amounts of text | |
| 08:20:59 | tkajinam | wondering if that "please report a bug" message is beneficial these days . I agree it was in the past when OpenStack was not yet common but I doubt it still is considering its current status | |
| 08:22:20 | tkajinam | recent reports are mostly caused by unrelated problems (wrong configuration, broken rabbitmq, etc) and personally I've not seen any reports related to actual logic bug in nova for a while | |
| 08:33:00 | frickler | clarkb: sean-k-mooney[m] is right, cirros is using a stock ubuntu kernel, which is why that makes up about half of the complete image size. but that's also why I'm not convinced that this is the source for the issues. did someone check for possible general OOM situations for these failures? | |
| 08:48:45 | jkulik | [ 13.568826] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00001000 | |
| 08:48:45 | jkulik | to me that kernel-panic topic sounds like something is wrong with the image or with loading it. I'd interprete it as /sbin/init running into an error and the kernel thus panicing. /sbin/init: can't load library 'libtirpc.so.3' | |
| 08:49:43 | jkulik | this bug report seems related https://bugs.launchpad.net/tempest/+bug/1888224 when ends in "It looks like rootfs image is corrupted." | |
| 08:51:05 | jkulik | I'm not sure how the test infra works, but if it only happens sporadically - could it be that some workers have a corrupt image and others don't? | |
| 09:52:34 | frickler | the cirros image should be baked into the cloud image we boot our test instances from, so if there really is some data corruption happening, it would need to be at a very low level and would likely lead to a more diverse spread of error patterns | |
| 10:00:28 | bauzas | when looking at the kernel panics, yeah I'm not sure other OSes would better work | |
| 10:27:02 | sean-k-mooney[m] | frickler: i am thinking its a general OOM issue which is why i was suggesting reducint the pcie ports we allocate and or adding swap to the tempest flavor to avoid continually adding more ram to the flavors. say add 64mb of swap in the flavor | |
| 10:42:11 | sean-k-mooney[m] | *reduceing | |
| 11:01:47 | sean-k-mooney[m] | frickler: by the way how would you feel about building busybox in statically linked mode so we can aovid dynmaiclly loading these shared objects in general? https://github.com/cirros-dev/cirros/blob/main/conf/busybox.config#L43 | |
| 11:40:41 | jkulik | hm ... yeah, could be OOM since the kernel-panic is in `exc_page_fault` ... maybe the kernel tries to load the shared library into memory but doesn't have the space? | |
| 12:01:06 | sean-k-mooney[m] | we have 128m total the kernel ramdisk was using 32mb of that and we allocated 24 pcie root devices each of which uses about 1mb or ram so we are using about half our ram just in does two. the initramfs can be unloaded once we complete the switch_root to the root block device but thats that failed. so ya it not entirely cleared if it OOM’d or not partly due to how early this is in boot but its worth exporing | |
| 12:37:51 | elodilles | bauzas: hi, you've asked me on the meeting to ping you today about the nova stable releases o:) ( https://review.opendev.org/q/project:openstack/releases+is:open+intopic:nova ) | |
| 12:39:27 | elodilles | bauzas: let me know if for any reason any of the release patches needs an update (hopefully the version bumps are correct, and hashes should be the latest) | |
| 12:40:19 | opendevreview | Danylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added. https://review.opendev.org/c/openstack/nova/+/876075 | |
| 13:03:32 | frickler | sean-k-mooney[m]: we can test that, would have to check how the effect on image size looks like | |
| 13:13:33 | sean-k-mooney | frickler: ya im not sure if it woudl go up or down (both on disk and at runtime) | |
| 13:14:29 | sean-k-mooney | frickler: are there any videos or blog posts on how cirros is built and its design goals? | |
| 13:15:36 | sean-k-mooney | i have too many other pulls on my time right now but kind of interested in either improving cirros to adress these probelems ro replacing it eventually | |
| 13:17:06 | sean-k-mooney | that way i was looking at alpine as a replacement, tinycore ala ironic ipa style would be fien too but if we can tweak cirros to make it more stable in this regrad and continue to use it then that fine in my book too | |
| 13:38:54 | dvo-plv | Hello, sean-k-mooney Are you here ? | |
| 13:39:32 | sean-k-mooney | im listenting to an internal call but yep | |
| 13:40:06 | frickler | sean-k-mooney: I'm not aware of any kind of docs except for what is inside the repo itself. if you want to talk to smoser directly, you can find us in #cirros on liberachat. iiuc the design goal is to have a minimal image for testing purposes, which matches pretty much what openstack CI does with it | |
| 13:41:23 | dvo-plv | I'm currently has a time to work with nova patch | |
| 13:41:38 | dvo-plv | I've resolve some of your comments | |
| 13:41:46 | dvo-plv | https://review.opendev.org/c/openstack/nova/+/876075/28/nova/virt/libvirt/config.py#1815 | |
| 13:41:46 | dvo-plv | but I have an open question | |
| 13:44:26 | sean-k-mooney | frickler: ack i skimed the docs breifly last night but have not actully looked at how the build system works or the current content of the image in any dept | |
| 13:44:50 | sean-k-mooney | frickler: so i was conisderign trying the static build myself | |
| 13:46:06 | sean-k-mooney | dvo-plv: so "self.driver_packed is True" is a common python mistake | |