| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-12-06 | |||
| 17:34:58 | gibi | almost :) https://github.com/openstack-k8s-operators/tcib/commit/199178028bba73548e21df5171a620df7981b451 | |
| 17:40:39 | artom | James was pulling the strings | |
| 17:41:33 | gibi | indeed, he is still pulling it https://github.com/openstack-k8s-operators/tcib/pull/110 | |
| 17:42:14 | artom | He's pull requesting them :D | |
| 17:46:19 | gibi | exactly :D | |
| 19:04:25 | opendevreview | melanie witt proposed openstack/nova master: Set UEC image vars for jobs not defined in Nova https://review.opendev.org/c/openstack/nova/+/902809 | |
| 19:15:58 | sean-k-mooney | melwitt: oh right i guess we should do that for those too | |
| 19:17:43 | sean-k-mooney | ill try and take a look at ^ tomorrow | |
| 19:18:10 | sean-k-mooney | one thing we might want to consider is actully makeing this change in the base devstack job | |
| 19:18:34 | sean-k-mooney | nova is really the only project that likely cares about testing the full disk image code path | |
| 19:18:58 | sean-k-mooney | so defaulting to the uec image might be worth considering in the tempest/devstack base job | |
| 19:19:23 | sean-k-mooney | gmann: ^ we should proably conisder that after we have run nova on them for a few weeks | |
| 19:20:41 | gmann | sean-k-mooney: melwitt yeah, i am thinking on that, if not default at least we can do that for most of the general integrated jobs in tempest and keep a few of them for example tempest-full-py3 on full image | |
| 19:21:00 | gmann | yes, let's observe it in nova for a few weeks first | |
| 19:34:16 | melwitt | sean-k-mooney, gmann: I realized the above (jobs not defined in nova) bc I saw tempest-integrated-compute fail on the follow up patch and it was the guest kernel panic. at first I was confused how/why could that be if we've got the UEC image but then I realized it is _not_ using the image bc I had only set the vars for the jobs defined by us in our .zuul.yaml | |
| 19:35:06 | sean-k-mooney | yep i forgot about the other jobs too | |
| 19:35:34 | melwitt | sean-k-mooney: semi related to that, I had been meaning to ask you, why does using the UEC image work to avoid guest kernel panics? | |
| 19:36:15 | melwitt | just a curiosity question | |
| 19:40:09 | opendevreview | Merged openstack/nova-specs master: Add spec for PCI Groups https://review.opendev.org/c/openstack/nova-specs/+/899719 | |
| 19:41:03 | sean-k-mooney | melwitt: i dont actully know. i suspect it takign a slightly diffent path in both qemu and in the init shell script | |
| 19:41:30 | sean-k-mooney | melwitt: i know neutron had previosuly added apci=off or something like that | |
| 19:41:41 | sean-k-mooney | to disable the apic timer | |
| 19:41:47 | melwitt | ah ok | |
| 19:41:54 | sean-k-mooney | but they dont actully do that any more | |
| 19:42:07 | sean-k-mooney | basiclly one of the panics suggeted doign that and when you use the split image | |
| 19:42:24 | sean-k-mooney | nova supprot passing kernel command line args to qemu | |
| 19:42:52 | sean-k-mooney | so orginally they explored this just to follow the mesage output which said A X to the command line to turn this off | |
| 19:45:03 | sean-k-mooney | melwitt: https://bugs.launchpad.net/openstack-gate/+bug/1713162 | |
| 19:45:15 | sean-k-mooney | Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. | |
| 19:45:30 | sean-k-mooney | so ya they orgianlly tied setting noapic on the kernel commadn line | |
| 19:45:36 | melwitt | ohh I see | |
| 19:46:08 | sean-k-mooney | i "fixed" this a diffent way in devstack | |
| 19:46:39 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.libvirt_disable_apic | |
| 19:47:53 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/766043 and https://github.com/openstack/devstack/commit/1e86a25cc28e34d7f73a4c6ccbbc3fc667598d50 | |
| 19:48:11 | sean-k-mooney | lee actully enabeld it in devstack | |
| 19:48:15 | melwitt | oh, so using the split image seems to be a way to alleviate the problem without disabling the apic timer? | |
| 19:48:41 | sean-k-mooney | so i tought at one point we were addign the noapic thign somewhere | |
| 19:49:09 | sean-k-mooney | i just dont know where that is exactly | |
| 19:49:54 | sean-k-mooney | melwitt: but ya apprently the split imag eis enouch i just cant prove that | |
| 19:50:00 | melwitt | well that's confusing. it's already been set to disabled but it was still panicking? 🤔 | |
| 19:50:00 | melwitt | well that's confusing. it's already been set to disabled but it was still panicking? 🤔 | |
| 19:50:26 | sean-k-mooney | so there are two tings | |
| 19:50:30 | sean-k-mooney | the apic and ioapic | |
| 19:50:44 | sean-k-mooney | we are disabling the apic | |
| 19:50:48 | melwitt | oh | |
| 19:51:19 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1939108 | |
| 19:51:45 | sean-k-mooney | im not sure if i orgianlly got those mixed up or not | |
| 19:52:06 | sean-k-mooney | in anycase i think thats a slighly idffent kernel panic | |
| 19:52:21 | melwitt | 🤯 | |
| 19:52:21 | melwitt | 🤯 | |
| 19:52:22 | sean-k-mooney | as that speicic one was a upstrema bug in the cirros iamge kernel that was fixed in a later release | |
| 19:52:52 | sean-k-mooney | basically this specific panicn in that bug should not happen in 6.x cirros iamges | |
| 19:53:32 | melwitt | I see | |
| 19:55:36 | sean-k-mooney | for what its worth it might be interesting to see if we can disabel that workaround once we have moved to the split miage | |
| 19:55:39 | sean-k-mooney | *image | |
| 19:56:43 | sean-k-mooney | we are using cirros 6.x now so it should not eb required | |
| 19:57:14 | melwitt | gotcha | |
| 19:57:15 | sean-k-mooney | so we could perhaps deprecate it this cycle and remove it in a future release | |
| 19:57:45 | melwitt | none of the panics I have seen had the message that specifically names the APIC timer so that seems to track | |
| 19:58:08 | melwitt | (from that bug) | |
| 19:58:55 | sean-k-mooney | ya same | |
| 20:00:22 | sean-k-mooney | ok im going to call it a day | |
| 20:00:31 | sean-k-mooney | ill chekc back on your patches tomorrow o/ | |
| 20:05:16 | melwitt | thanks sean-k-mooney, seeya tomorrow o/ | |
| 20:15:44 | JayF | sean-k-mooney: I know you're interested in getting sharding in this cycle; I'm trying to land stephen's chain here https://review.opendev.org/c/openstack/nova/+/659691/38 before re-landing sharding; if you want to take a look I think Ironic cores are happy (Ironic CI passes on the head of that chain) | |
| 20:24:19 | melwitt | dpawlik: hi, thanks for confirming! it was just a misunderstanding about swift log retention vs opensearch index retention. the swift log retention is 30 days but opensearch retains less. I think this is ok :) | |
| 21:32:14 | opendevreview | Merged openstack/nova master: Fix coverage issues with eventlet https://review.opendev.org/c/openstack/nova/+/900116 | |
| #openstack-nova - 2023-12-07 | |||
| 07:14:01 | dpawlik | melwitt ack. Let me know if you would like to increase the retention time | |
| 07:25:56 | opendevreview | Amit Uniyal proposed openstack/nova-specs master: Enforce console session timeout https://review.opendev.org/c/openstack/nova-specs/+/898553 | |
| 10:41:15 | opendevreview | Vasyl Saienko proposed openstack/nova master: Fix returning empty availability zone https://review.opendev.org/c/openstack/nova/+/902875 | |
| 10:58:40 | sfinucan | sean-k-mooney: Thanks for starting reviewing that ironic SDK migration series. Given it's a priority for the ironic folks and also relatively trivial, I wonder if I could get another "review sponsor" for the series? CC: gibi melwitt dansmith bauzas gmann | |
| 10:59:43 | stephenfin | It's been long enough since I wrote them that I could review myself at this point, but I'd rather not if I can avoid it | |
| 11:07:20 | sean-k-mooney[m] | ya they are failrly simple sofar. since you rebased them yesterday it would be good to land them before they start to bit rot | |
| 11:34:19 | sean-k-mooney[m] | ok done. im +2 all the way | |
| 11:51:22 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Use clearer syntax for empty generator https://review.opendev.org/c/openstack/nova/+/903076 | |
| 11:51:32 | stephenfin | sean-k-mooney[m]: "remove dark magic" patch | |
| 11:51:37 | stephenfin | ^ | |
| 11:52:42 | sean-k-mooney[m] | thanks | |
| 12:09:52 | opendevreview | Pranali Deore proposed openstack/nova master: Support glance's new location API https://review.opendev.org/c/openstack/nova/+/891036 | |
| 12:10:02 | opendevreview | Pranali Deore proposed openstack/nova master: DNM: Test glance new location api https://review.opendev.org/c/openstack/nova/+/891207 | |
| 13:36:55 | sean-k-mooney | hi can i get some eyes on https://review.opendev.org/c/openstack/devstack/+/890693 please. | |
| 13:37:06 | sean-k-mooney | oh that was for qa.. | |
| 13:40:48 | sean-k-mooney | tempest-integrated compute often seamm to get pretty close to its 2 hour timeout but there is also a lot of variance | |
| 13:40:54 | sean-k-mooney | https://zuul.opendev.org/t/openstack/builds?job_name=tempest-integrated-compute&project=openstack%2Fnova | |
| 14:10:13 | gibi | stephenfin: nice series, and now it is approved all the way up :) | |
| 14:10:40 | stephenfin | gibi: � thank you! | |
| 14:10:40 | stephenfin | gibi: 👏 thank you! | |
| 15:09:50 | bauzas | if people want to see my face live (after all, this is not like I see most of you everyday on gmeet) : https://www.youtube.com/live/thidlQGX29M?feature=shared | |
| 15:10:14 | bauzas | tl;dr: PTG recap OIL episode | |
| 16:15:07 | melwitt | gmann: if you get a moment https://review.opendev.org/c/openstack/nova/+/902809 | |
| 16:16:11 | melwitt | sean-k-mooney: I have noticed TIMED_OUT too for job running just a little over 2 hours | |
| 16:16:47 | sean-k-mooney | yep | |
| 16:16:51 | sean-k-mooney | i noticed that too | |
| 16:17:11 | sean-k-mooney | thats why i mentioned my zswap changes for devstack | |
| 16:17:15 | melwitt | I said "too" first, I saw your message in the scrollback :) | |
| 16:17:19 | sean-k-mooney | since that was ment to speed up the jobs | |
| 16:17:30 | melwitt | ah, cool, I'll check that out | |
| 16:17:43 | sean-k-mooney | it does not seam to be that common | |
| 16:57:16 | stephenfin | is it expected that adding a port will add the default security group to an instance if not already present? | |
| 16:58:06 | stephenfin | I'm guessing yes assuming a port is using the default security group | |