| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-29 | |||
| 18:07:00 | melwitt | kk | |
| 18:07:00 | sean-k-mooney | we use about 4-6 by defualt and i dont think we are attaching that many volumes or prots to cause an issue | |
| 18:26:46 | opendevreview | Merged openstack/python-novaclient master: add pyproject.toml to support pip 23.1 https://review.opendev.org/c/openstack/python-novaclient/+/899950 | |
| 19:15:05 | opendevreview | Merged openstack/nova stable/2023.2: Allow enabling cpu_power_management with 0 dedicated CPUs https://review.opendev.org/c/openstack/nova/+/901656 | |
| 19:32:17 | melwitt | argh, it already failed for the non-q35 guest kernel panic https://4a4a1510e17776a8b793-89f5aa2f0368a55b3a90e6b26173438f.ssl.cf2.rackcdn.com/902175/1/check/nova-multi-cell/c2bacb0/testr_results.html | |
| 19:33:11 | melwitt | it meaning the patch, unrelated to the change itself | |
| 19:35:17 | sean-k-mooney | that is a slightly diffent panic | |
| 19:35:35 | sean-k-mooney | it paniced at the same phases switching_root | |
| 19:35:43 | melwitt | right | |
| 19:35:53 | sean-k-mooney | but instead of being in a page fault handeler it was in teh apic timer handeler | |
| 19:36:14 | sean-k-mooney | it still could be OOM related | |
| 19:37:15 | sean-k-mooney | in that case we also dont have any of the io mapping error since it machine_type=pc and using pci not pcie as a result | |
| 19:37:28 | melwitt | that would be my guess | |
| 19:37:36 | melwitt | right | |
| 19:38:03 | sean-k-mooney | so the patch will help with one of the 2 issues but perhaps we shoudl just add 64 mb of swap to the tempest flavors | |
| 19:38:44 | melwitt | I've been comparing a non kernel panic console log with a panic one and one difference I see leading up to it is that the non-panic has this line "info: copying initramfs to /dev/vda1" but the panic one does not | |
| 19:39:06 | sean-k-mooney | https://github.com/openstack/devstack/blob/master/lib/tempest#L293-L306 | |
| 19:40:30 | sean-k-mooney | so from here https://github.com/cirros-dev/cirros/blob/46a1162787f669ad8d6065cb6bbe477654b4327f/src/init#L63 | |
| 19:40:41 | melwitt | hm ok. so maybe I can try adding swap there and recheck a bunch of times to see if panic happens? | |
| 19:40:53 | melwitt | yes | |
| 19:41:22 | melwitt | so that code block is skipped in the cases where it panics. dunno why | |
| 19:41:41 | sean-k-mooney | so its failing here https://github.com/cirros-dev/cirros/blob/46a1162787f669ad8d6065cb6bbe477654b4327f/src/init#L85 | |
| 19:42:07 | melwitt | right | |
| 19:42:21 | sean-k-mooney | that after where the copy should print | |
| 19:42:42 | sean-k-mooney | do you see the GPT header messages in the one that passed | |
| 19:43:01 | melwitt | yes | |
| 19:43:03 | sean-k-mooney | [ 11.405476] GPT:Primary header thinks Alt. header is not at the end of the disk. | |
| 19:43:05 | sean-k-mooney | [ 11.406051] GPT:229375 != 2097151 | |
| 19:43:07 | sean-k-mooney | [ 11.406370] GPT:Alternate GPT header not at the end of the disk. | |
| 19:43:09 | sean-k-mooney | [ 11.406743] GPT:229375 != 2097151 | |
| 19:43:11 | sean-k-mooney | [ 11.407018] GPT: Use GNU Parted to correct GPT errors. | |
| 19:43:13 | sean-k-mooney | ok so its not realted to that | |
| 19:43:48 | sean-k-mooney | if we dont see "info: copying initramfs to /dev/vda1" | |
| 19:43:55 | sean-k-mooney | that imples $ROOT is undefiend | |
| 19:44:00 | melwitt | yeah.. I kinda wondered if the lack of "copying initramfs" is related to not being able to load 'libtirpc.so.3' | |
| 19:44:32 | sean-k-mooney | well if we have not copied the ramdisk to the root file system | |
| 19:44:42 | sean-k-mooney | the the dynmaic loader wont be able to find it | |
| 19:45:14 | sean-k-mooney | so looking at https://github.com/cirros-dev/cirros/blob/46a1162787f669ad8d6065cb6bbe477654b4327f/src/init#L44-L52 | |
| 19:45:34 | melwitt | either $ROOT is undefined of if search_for_blank "$rootspec" rw "$NEWROOT_MP" was false | |
| 19:45:56 | sean-k-mooney | ya i thik in the failing case we are taking the else branch on line 50 | |
| 19:46:43 | melwitt | I wonder how we could enable debug logging in there | |
| 19:46:59 | melwitt | debug 1 "did not find a device matching $rootspec" but we don't see it | |
| 19:47:12 | sean-k-mooney | we likely cod do ti by passing kernel command line args | |
| 19:47:23 | sean-k-mooney | but i partly wonder if that woudl fix it | |
| 19:47:57 | sean-k-mooney | we woudl have to change form the full disk image to the one with the splict out kernel and initram | |
| 19:48:10 | sean-k-mooney | i know that neutron use that in one of there jobs | |
| 19:48:22 | sean-k-mooney | to avoid some kernel panics. | |
| 19:48:29 | melwitt | oh huh | |
| 19:48:29 | sean-k-mooney | specifical this on ewith the apic timer | |
| 19:48:39 | melwitt | interesting | |
| 19:49:09 | sean-k-mooney | any unrecognised parmater on the kernel command line is set as an ENV var in the init procsss by the kernel | |
| 19:49:32 | sean-k-mooney | incase you didnt know that so if debug is based on an env var you sould jsut add that env var to the kernel command line | |
| 19:49:54 | melwitt | so theoretically the num_pcie_ports might fix the nova-next panics and the split disk image might fix the rest ... seems it would be too good to be true | |
| 19:50:17 | melwitt | ok yeah I didn't know that | |
| 19:50:22 | sean-k-mooney | well im 99% shure that the split disk will fix the apic one | |
| 19:50:52 | sean-k-mooney | the only down side to that is that we loose coverage of testign full disk images if we use that every where | |
| 19:51:10 | sean-k-mooney | ill check quickly if i can see where they enable it in there job | |
| 19:52:42 | melwitt | k. I wonder if we could get away with using the split image everywhere except nova-next | |
| 19:53:05 | opendevreview | alisafari proposed openstack/nova master: Fix traits to cpu flags mapping https://review.opendev.org/c/openstack/nova/+/902183 | |
| 19:53:15 | sean-k-mooney | proably | |
| 19:54:21 | melwitt | I feel like I'm seeing these panics all the time. not sure what changed bc they used to be so rare in the past. maybe OOM like you said or maybe if in the past everything was using the split image, that I don't know | |
| 19:56:02 | sean-k-mooney | no we were defienlty usign the whole disk image for several reelase | |
| 19:56:17 | sean-k-mooney | i dont know if we ever use d the split imate by default | |
| 19:56:18 | melwitt | ack | |
| 19:56:20 | sean-k-mooney | we may have | |
| 20:10:09 | sean-k-mooney | ah found it | |
| 20:10:10 | sean-k-mooney | https://github.com/openstack/neutron/commit/e04bd8fbdfa56320d16870b1f294b2cb62b8a828 | |
| 20:11:16 | sean-k-mooney | so if we add | |
| 20:11:20 | sean-k-mooney | CIRROS_VERSION: 0.6.2 | |
| 20:11:22 | sean-k-mooney | DEFAULT_IMAGE_NAME: cirros-0.6.2-x86_64-uec | |
| 20:11:24 | sean-k-mooney | DEFAULT_IMAGE_FILE_NAME: cirros-0.6.2-x86_64-uec.tar.gz | |
| 20:12:29 | sean-k-mooney | that shoud fix it | |
| 20:12:40 | sean-k-mooney | the other way to do that is move the job to use nested virt | |
| 20:12:43 | sean-k-mooney | https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/821067 | |
| 20:13:49 | sean-k-mooney | melwitt: what we coudl do is swap to the uec image for our default jobs | |
| 20:14:07 | sean-k-mooney | and move nova-next to nested virt with the whole disk image | |
| 20:14:59 | melwitt | sounds worth a try | |
| 20:15:02 | sean-k-mooney | although we woudl want ot use the nested virt jamy lable in our node set | |
| 20:15:04 | sean-k-mooney | https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl04.opendev.org.yaml#L41-L42 | |
| 20:17:24 | melwitt | ok | |
| 20:18:55 | sean-k-mooney | do you know how ot make those changes. im going to finish for today and get dinner but i can update the jobs tomorow if you havent and we can see if it works | |
| 20:19:34 | melwitt | I think I can do it given the examples you linked. but yeah sounds good, if I do it wrong feel free to redo it or whatever tomorrow | |
| 20:21:12 | sean-k-mooney | we in theory have nested virt capablity for 2 vexhost clouds and ovh so i dont really feel as bad about using in in nova next as i did whne it was only one provider | |
| 20:22:53 | melwitt | ah ok | |
| #openstack-nova - 2023-11-30 | |||
| 03:01:08 | opendevreview | melanie witt proposed openstack/nova master: Use split kernel/initramfs Cirros UEC image by default in jobs https://review.opendev.org/c/openstack/nova/+/902217 | |
| 06:40:30 | melwitt | sean-k-mooney: there was an attempt ... https://review.opendev.org/c/openstack/nova/+/902217 but it's super not working 😆 | |
| 06:40:30 | melwitt | sean-k-mooney: there was an attempt ... https://review.opendev.org/c/openstack/nova/+/902217 but it's super not working 😆 | |
| 07:54:29 | opendevreview | alisafari proposed openstack/nova master: Fix traits to cpu flags mapping https://review.opendev.org/c/openstack/nova/+/902183 | |
| 09:26:14 | dvo-plv | gibi, dansmith Hello, gentlemens? Maybe you will have a time to review spec and code | |
| 09:26:18 | dvo-plv | https://review.opendev.org/c/openstack/nova-specs/+/895924 | |
| 09:26:22 | dvo-plv | https://review.opendev.org/c/openstack/nova/+/876075 | |
| 09:34:12 | gibi | bauzas: https://review.opendev.org/c/openstack/nova/+/901656 merged so I bobcat point release is good to go for me :) | |
| 09:34:26 | bauzas | cool | |
| 09:34:37 | bauzas | elodilles: can you please update the bobcat release patch then , | |
| 09:36:43 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.1: Do not manage CPU0's state https://review.opendev.org/c/openstack/nova/+/901567 | |
| 09:36:56 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2023.1: Allow enabling cpu_power_management with 0 dedicated CPUs https://review.opendev.org/c/openstack/nova/+/901660 | |
| 09:37:24 | gibi | elodilles: fixed your comment in https://review.opendev.org/c/openstack/nova/+/901567 | |
| 09:50:44 | dvo-plv | gibi, thanks. Should I do something else to get +1 workflow for patch ? | |
| 09:50:47 | frickler | melwitt: yes, you need to add more things to your nodeset definition, let me update | |
| 09:51:59 | gibi | dvo-plv: I'm waiting for the spec to land | |