| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-11-29 | |||
| 16:56:32 | bauzas | pas-ha[m] change is accepting to only use one type but with a section | |
| 16:56:37 | bauzas | yeah | |
| 16:56:41 | sean-k-mooney | ya so | |
| 16:57:00 | sean-k-mooney | without that i dont think device_adderss being optional is really a good approch | |
| 16:57:04 | bauzas | so, my WIP (if we make dev_add optional) would change the existing behaviour | |
| 16:57:06 | sean-k-mooney | btu if we fix that then im fine with it | |
| 16:57:39 | bauzas | which is that if you use two types but only one got a section, then none of them use any section | |
| 16:58:08 | bauzas | not sure we could backport that | |
| 16:58:13 | sean-k-mooney | thats a bug right | |
| 16:58:22 | bauzas | not really | |
| 16:58:29 | sean-k-mooney | it kind of is | |
| 16:58:45 | sean-k-mooney | im aware of the legacy behavior for the first type | |
| 16:58:53 | bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7938-L7951 | |
| 16:59:00 | sean-k-mooney | but the second adn subsquent type are ment ot have a section | |
| 16:59:23 | bauzas | this conditional makes all types needing to have groups for each of them | |
| 16:59:23 | sean-k-mooney | and we shoudl not ignore them if they are defiend just because the first type does not ahve one | |
| 16:59:48 | bauzas | sure, I'm just changing this is a bit of a behavioural change | |
| 16:59:59 | bauzas | which couldn't be acceptable for a backport | |
| 17:00:05 | sean-k-mooney | so i do think all types shoudl have a section in general | |
| 17:00:15 | sean-k-mooney | however that only makes snese if there is somethign in it | |
| 17:00:18 | bauzas | you do or don't ? | |
| 17:00:55 | sean-k-mooney | i think we shoudl requrie a section for each mdev type unconditonally | |
| 17:00:59 | sean-k-mooney | personlaly | |
| 17:01:11 | sean-k-mooney | but wiether device_adresss is required or not is a sperate question | |
| 17:01:51 | bauzas | so you'd prefer to have https://paste.opendev.org/show/bKc2SjB6l7LILYE0onzG/ | |
| 17:02:30 | sean-k-mooney | kind of yes although i would also like to consider if max_instances is requried or not | |
| 17:02:33 | bauzas | if so, that simplifies my change and then we can backport it :) | |
| 17:02:57 | sean-k-mooney | to me https://paste.opendev.org/show/bKc2SjB6l7LILYE0onzG/ | |
| 17:03:06 | sean-k-mooney | is you intentionally not setting device_adresses | |
| 17:03:08 | bauzas | I wouldn't require max_instance or only unless device_addresses isn't set | |
| 17:03:19 | sean-k-mooney | and as a result opting into the whild card behavior | |
| 17:03:33 | bauzas | sean-k-mooney: correct, you're saying 'this is a default type for all found GPUs' | |
| 17:04:03 | bauzas | okay, then I'll change my current non-uploaded patch and I'll provide it tomorrow | |
| 17:04:11 | bauzas | thanks sean-k-mooney | |
| 17:04:30 | sean-k-mooney | ack | |
| 17:04:41 | sean-k-mooney | gibi: ^ when you have time does that work for you | |
| 17:04:46 | bauzas | sean-k-mooney: fwiw, https://review.opendev.org/c/openstack/nova-specs/+/900636 is open for reviews :) | |
| 17:05:04 | sean-k-mooney | hehe ya i need to do a review day soon | |
| 17:05:08 | sean-k-mooney | maybe tomorrow | |
| 17:05:21 | bauzas | you're lucky | |
| 17:05:28 | bauzas | my review karma is bad those days | |
| 17:06:42 | sean-k-mooney | https://www.stackalytics.io/report/contribution?module=nova-group&project_type=openstack&days=30 i mean min is not much better for thte last little bit | |
| 17:07:27 | gibi | bauzas: sean-k-mooney: sounds good to me | |
| 17:07:34 | bauzas | thanks gibi | |
| 17:07:43 | bauzas | will upload the patch quickly then | |
| 17:29:24 | opendevreview | melanie witt proposed openstack/nova master: Lower num_pcie_ports to 12 in the nova-next job https://review.opendev.org/c/openstack/nova/+/902175 | |
| 17:30:05 | melwitt | sean-k-mooney: from our convo yesterday ^ | |
| 18:06:20 | sean-k-mooney | cool lets see if that passses | |
| 18:06:34 | sean-k-mooney | it would only fial if we need more then 12 pci device in a tempest test | |
| 18:07:00 | melwitt | kk | |
| 18:07:00 | sean-k-mooney | we use about 4-6 by defualt and i dont think we are attaching that many volumes or prots to cause an issue | |
| 18:26:46 | opendevreview | Merged openstack/python-novaclient master: add pyproject.toml to support pip 23.1 https://review.opendev.org/c/openstack/python-novaclient/+/899950 | |
| 19:15:05 | opendevreview | Merged openstack/nova stable/2023.2: Allow enabling cpu_power_management with 0 dedicated CPUs https://review.opendev.org/c/openstack/nova/+/901656 | |
| 19:32:17 | melwitt | argh, it already failed for the non-q35 guest kernel panic https://4a4a1510e17776a8b793-89f5aa2f0368a55b3a90e6b26173438f.ssl.cf2.rackcdn.com/902175/1/check/nova-multi-cell/c2bacb0/testr_results.html | |
| 19:33:11 | melwitt | it meaning the patch, unrelated to the change itself | |
| 19:35:17 | sean-k-mooney | that is a slightly diffent panic | |
| 19:35:35 | sean-k-mooney | it paniced at the same phases switching_root | |
| 19:35:43 | melwitt | right | |
| 19:35:53 | sean-k-mooney | but instead of being in a page fault handeler it was in teh apic timer handeler | |
| 19:36:14 | sean-k-mooney | it still could be OOM related | |
| 19:37:15 | sean-k-mooney | in that case we also dont have any of the io mapping error since it machine_type=pc and using pci not pcie as a result | |
| 19:37:28 | melwitt | that would be my guess | |
| 19:37:36 | melwitt | right | |
| 19:38:03 | sean-k-mooney | so the patch will help with one of the 2 issues but perhaps we shoudl just add 64 mb of swap to the tempest flavors | |
| 19:38:44 | melwitt | I've been comparing a non kernel panic console log with a panic one and one difference I see leading up to it is that the non-panic has this line "info: copying initramfs to /dev/vda1" but the panic one does not | |
| 19:39:06 | sean-k-mooney | https://github.com/openstack/devstack/blob/master/lib/tempest#L293-L306 | |
| 19:40:30 | sean-k-mooney | so from here https://github.com/cirros-dev/cirros/blob/46a1162787f669ad8d6065cb6bbe477654b4327f/src/init#L63 | |
| 19:40:41 | melwitt | hm ok. so maybe I can try adding swap there and recheck a bunch of times to see if panic happens? | |
| 19:40:53 | melwitt | yes | |
| 19:41:22 | melwitt | so that code block is skipped in the cases where it panics. dunno why | |
| 19:41:41 | sean-k-mooney | so its failing here https://github.com/cirros-dev/cirros/blob/46a1162787f669ad8d6065cb6bbe477654b4327f/src/init#L85 | |
| 19:42:07 | melwitt | right | |
| 19:42:21 | sean-k-mooney | that after where the copy should print | |
| 19:42:42 | sean-k-mooney | do you see the GPT header messages in the one that passed | |
| 19:43:01 | melwitt | yes | |
| 19:43:03 | sean-k-mooney | [ 11.405476] GPT:Primary header thinks Alt. header is not at the end of the disk. | |
| 19:43:05 | sean-k-mooney | [ 11.406051] GPT:229375 != 2097151 | |
| 19:43:07 | sean-k-mooney | [ 11.406370] GPT:Alternate GPT header not at the end of the disk. | |
| 19:43:09 | sean-k-mooney | [ 11.406743] GPT:229375 != 2097151 | |
| 19:43:11 | sean-k-mooney | [ 11.407018] GPT: Use GNU Parted to correct GPT errors. | |
| 19:43:13 | sean-k-mooney | ok so its not realted to that | |
| 19:43:48 | sean-k-mooney | if we dont see "info: copying initramfs to /dev/vda1" | |
| 19:43:55 | sean-k-mooney | that imples $ROOT is undefiend | |
| 19:44:00 | melwitt | yeah.. I kinda wondered if the lack of "copying initramfs" is related to not being able to load 'libtirpc.so.3' | |
| 19:44:32 | sean-k-mooney | well if we have not copied the ramdisk to the root file system | |
| 19:44:42 | sean-k-mooney | the the dynmaic loader wont be able to find it | |
| 19:45:14 | sean-k-mooney | so looking at https://github.com/cirros-dev/cirros/blob/46a1162787f669ad8d6065cb6bbe477654b4327f/src/init#L44-L52 | |
| 19:45:34 | melwitt | either $ROOT is undefined of if search_for_blank "$rootspec" rw "$NEWROOT_MP" was false | |
| 19:45:56 | sean-k-mooney | ya i thik in the failing case we are taking the else branch on line 50 | |
| 19:46:43 | melwitt | I wonder how we could enable debug logging in there | |
| 19:46:59 | melwitt | debug 1 "did not find a device matching $rootspec" but we don't see it | |
| 19:47:12 | sean-k-mooney | we likely cod do ti by passing kernel command line args | |
| 19:47:23 | sean-k-mooney | but i partly wonder if that woudl fix it | |
| 19:47:57 | sean-k-mooney | we woudl have to change form the full disk image to the one with the splict out kernel and initram | |
| 19:48:10 | sean-k-mooney | i know that neutron use that in one of there jobs | |
| 19:48:22 | sean-k-mooney | to avoid some kernel panics. | |
| 19:48:29 | melwitt | oh huh | |
| 19:48:29 | sean-k-mooney | specifical this on ewith the apic timer | |
| 19:48:39 | melwitt | interesting | |
| 19:49:09 | sean-k-mooney | any unrecognised parmater on the kernel command line is set as an ENV var in the init procsss by the kernel | |