| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-12-06 | |||
| 17:11:08 | sean-k-mooney | form the souce node | |
| 17:11:16 | sean-k-mooney | well i guess | |
| 17:11:26 | sean-k-mooney | we can claim them | |
| 17:11:34 | sean-k-mooney | but we wont detect if the type are diffent | |
| 17:11:39 | sean-k-mooney | until after we call the souce node | |
| 17:11:45 | sean-k-mooney | so we will claim them on the test | |
| 17:11:53 | sean-k-mooney | then apass the info to the soruce with the types | |
| 17:12:00 | sean-k-mooney | and it can check if they change and abort | |
| 17:12:10 | bauzas | hum | |
| 17:13:26 | bauzas | actually, this is not possible, damn | |
| 17:13:37 | bauzas | we call the destination first, then we call the sourcer | |
| 17:13:43 | bauzas | and then we claim | |
| 17:17:02 | sean-k-mooney | right but the call to the dest coudl do the claim | |
| 17:17:20 | sean-k-mooney | then pass to the souce to confirm if the types match | |
| 17:17:22 | sean-k-mooney | actully no | |
| 17:17:30 | sean-k-mooney | we are stuck with the 3 stpes i think | |
| 17:17:46 | sean-k-mooney | lets pick up this converstaion tomorrow | |
| 17:32:03 | gibi | artom: we have a chance https://github.com/openstack-k8s-operators/tcib/blob/4aef81881de941f0251aa848515243f666b18672/container-images/tcib/base/os/tempest/tempest-extras/tempest-extras.yaml#L4 | |
| 17:32:49 | artom | Oh wow | |
| 17:32:52 | artom | jparker++ | |
| 17:32:58 | artom | I'm assuming he's to blame :D | |
| 17:33:25 | artom | (Not literal `git blame`) | |
| 17:34:58 | gibi | almost :) https://github.com/openstack-k8s-operators/tcib/commit/199178028bba73548e21df5171a620df7981b451 | |
| 17:40:39 | artom | James was pulling the strings | |
| 17:41:33 | gibi | indeed, he is still pulling it https://github.com/openstack-k8s-operators/tcib/pull/110 | |
| 17:42:14 | artom | He's pull requesting them :D | |
| 17:46:19 | gibi | exactly :D | |
| 19:04:25 | opendevreview | melanie witt proposed openstack/nova master: Set UEC image vars for jobs not defined in Nova https://review.opendev.org/c/openstack/nova/+/902809 | |
| 19:15:58 | sean-k-mooney | melwitt: oh right i guess we should do that for those too | |
| 19:17:43 | sean-k-mooney | ill try and take a look at ^ tomorrow | |
| 19:18:10 | sean-k-mooney | one thing we might want to consider is actully makeing this change in the base devstack job | |
| 19:18:34 | sean-k-mooney | nova is really the only project that likely cares about testing the full disk image code path | |
| 19:18:58 | sean-k-mooney | so defaulting to the uec image might be worth considering in the tempest/devstack base job | |
| 19:19:23 | sean-k-mooney | gmann: ^ we should proably conisder that after we have run nova on them for a few weeks | |
| 19:20:41 | gmann | sean-k-mooney: melwitt yeah, i am thinking on that, if not default at least we can do that for most of the general integrated jobs in tempest and keep a few of them for example tempest-full-py3 on full image | |
| 19:21:00 | gmann | yes, let's observe it in nova for a few weeks first | |
| 19:34:16 | melwitt | sean-k-mooney, gmann: I realized the above (jobs not defined in nova) bc I saw tempest-integrated-compute fail on the follow up patch and it was the guest kernel panic. at first I was confused how/why could that be if we've got the UEC image but then I realized it is _not_ using the image bc I had only set the vars for the jobs defined by us in our .zuul.yaml | |
| 19:35:06 | sean-k-mooney | yep i forgot about the other jobs too | |
| 19:35:34 | melwitt | sean-k-mooney: semi related to that, I had been meaning to ask you, why does using the UEC image work to avoid guest kernel panics? | |
| 19:36:15 | melwitt | just a curiosity question | |
| 19:40:09 | opendevreview | Merged openstack/nova-specs master: Add spec for PCI Groups https://review.opendev.org/c/openstack/nova-specs/+/899719 | |
| 19:41:03 | sean-k-mooney | melwitt: i dont actully know. i suspect it takign a slightly diffent path in both qemu and in the init shell script | |
| 19:41:30 | sean-k-mooney | melwitt: i know neutron had previosuly added apci=off or something like that | |
| 19:41:41 | sean-k-mooney | to disable the apic timer | |
| 19:41:47 | melwitt | ah ok | |
| 19:41:54 | sean-k-mooney | but they dont actully do that any more | |
| 19:42:07 | sean-k-mooney | basiclly one of the panics suggeted doign that and when you use the split image | |
| 19:42:24 | sean-k-mooney | nova supprot passing kernel command line args to qemu | |
| 19:42:52 | sean-k-mooney | so orginally they explored this just to follow the mesage output which said A X to the command line to turn this off | |
| 19:45:03 | sean-k-mooney | melwitt: https://bugs.launchpad.net/openstack-gate/+bug/1713162 | |
| 19:45:15 | sean-k-mooney | Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. | |
| 19:45:30 | sean-k-mooney | so ya they orgianlly tied setting noapic on the kernel commadn line | |
| 19:45:36 | melwitt | ohh I see | |
| 19:46:08 | sean-k-mooney | i "fixed" this a diffent way in devstack | |
| 19:46:39 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.libvirt_disable_apic | |
| 19:47:53 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/766043 and https://github.com/openstack/devstack/commit/1e86a25cc28e34d7f73a4c6ccbbc3fc667598d50 | |
| 19:48:11 | sean-k-mooney | lee actully enabeld it in devstack | |
| 19:48:15 | melwitt | oh, so using the split image seems to be a way to alleviate the problem without disabling the apic timer? | |
| 19:48:41 | sean-k-mooney | so i tought at one point we were addign the noapic thign somewhere | |
| 19:49:09 | sean-k-mooney | i just dont know where that is exactly | |
| 19:49:54 | sean-k-mooney | melwitt: but ya apprently the split imag eis enouch i just cant prove that | |
| 19:50:00 | melwitt | well that's confusing. it's already been set to disabled but it was still panicking? 🤔 | |
| 19:50:00 | melwitt | well that's confusing. it's already been set to disabled but it was still panicking? 🤔 | |
| 19:50:26 | sean-k-mooney | so there are two tings | |
| 19:50:30 | sean-k-mooney | the apic and ioapic | |
| 19:50:44 | sean-k-mooney | we are disabling the apic | |
| 19:50:48 | melwitt | oh | |
| 19:51:19 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1939108 | |
| 19:51:45 | sean-k-mooney | im not sure if i orgianlly got those mixed up or not | |
| 19:52:06 | sean-k-mooney | in anycase i think thats a slighly idffent kernel panic | |
| 19:52:21 | melwitt | 🤯 | |
| 19:52:21 | melwitt | 🤯 | |
| 19:52:22 | sean-k-mooney | as that speicic one was a upstrema bug in the cirros iamge kernel that was fixed in a later release | |
| 19:52:52 | sean-k-mooney | basically this specific panicn in that bug should not happen in 6.x cirros iamges | |
| 19:53:32 | melwitt | I see | |
| 19:55:36 | sean-k-mooney | for what its worth it might be interesting to see if we can disabel that workaround once we have moved to the split miage | |
| 19:55:39 | sean-k-mooney | *image | |
| 19:56:43 | sean-k-mooney | we are using cirros 6.x now so it should not eb required | |
| 19:57:14 | melwitt | gotcha | |
| 19:57:15 | sean-k-mooney | so we could perhaps deprecate it this cycle and remove it in a future release | |
| 19:57:45 | melwitt | none of the panics I have seen had the message that specifically names the APIC timer so that seems to track | |
| 19:58:08 | melwitt | (from that bug) | |
| 19:58:55 | sean-k-mooney | ya same | |
| 20:00:22 | sean-k-mooney | ok im going to call it a day | |
| 20:00:31 | sean-k-mooney | ill chekc back on your patches tomorrow o/ | |
| 20:05:16 | melwitt | thanks sean-k-mooney, seeya tomorrow o/ | |
| 20:15:44 | JayF | sean-k-mooney: I know you're interested in getting sharding in this cycle; I'm trying to land stephen's chain here https://review.opendev.org/c/openstack/nova/+/659691/38 before re-landing sharding; if you want to take a look I think Ironic cores are happy (Ironic CI passes on the head of that chain) | |
| 20:24:19 | melwitt | dpawlik: hi, thanks for confirming! it was just a misunderstanding about swift log retention vs opensearch index retention. the swift log retention is 30 days but opensearch retains less. I think this is ok :) | |
| 21:32:14 | opendevreview | Merged openstack/nova master: Fix coverage issues with eventlet https://review.opendev.org/c/openstack/nova/+/900116 | |
| #openstack-nova - 2023-12-07 | |||
| 07:14:01 | dpawlik | melwitt ack. Let me know if you would like to increase the retention time | |
| 07:25:56 | opendevreview | Amit Uniyal proposed openstack/nova-specs master: Enforce console session timeout https://review.opendev.org/c/openstack/nova-specs/+/898553 | |
| 10:41:15 | opendevreview | Vasyl Saienko proposed openstack/nova master: Fix returning empty availability zone https://review.opendev.org/c/openstack/nova/+/902875 | |
| 10:58:40 | sfinucan | sean-k-mooney: Thanks for starting reviewing that ironic SDK migration series. Given it's a priority for the ironic folks and also relatively trivial, I wonder if I could get another "review sponsor" for the series? CC: gibi melwitt dansmith bauzas gmann | |
| 10:59:43 | stephenfin | It's been long enough since I wrote them that I could review myself at this point, but I'd rather not if I can avoid it | |
| 11:07:20 | sean-k-mooney[m] | ya they are failrly simple sofar. since you rebased them yesterday it would be good to land them before they start to bit rot | |
| 11:34:19 | sean-k-mooney[m] | ok done. im +2 all the way | |
| 11:51:22 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Use clearer syntax for empty generator https://review.opendev.org/c/openstack/nova/+/903076 | |
| 11:51:32 | stephenfin | sean-k-mooney[m]: "remove dark magic" patch | |
| 11:51:37 | stephenfin | ^ | |
| 11:52:42 | sean-k-mooney[m] | thanks | |