| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-03-18 | |||
| 16:59:10 | Uggla | #endmeeting | |
| 16:59:10 | opendevmeet | Meeting ended Tue Mar 18 16:59:10 2025 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:59:10 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2025/nova.2025-03-18-16.01.html | |
| 16:59:10 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2025/nova.2025-03-18-16.01.txt | |
| 16:59:10 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2025/nova.2025-03-18-16.01.log.html | |
| 16:59:17 | bauzas | heh, that worked well :) | |
| 16:59:18 | gibi | bauzas, Uggla thanks | |
| 16:59:20 | bauzas | thanks Uggla | |
| 16:59:27 | elodilles | thanks Uggla bauzas o/ | |
| 16:59:49 | masahito | thank you, Uggla and bauzas | |
| 16:59:53 | sean-k-mooney | ishanwar[m]: so if you look at https://github.com/openstack/nova/commit/191bdf2069086493be2a2e0351afa7f3efad7099#diff-de8fba9a58a07803b3d46cfb1f58d1b89e7cc63684f02c30297e6bdabffe870eR4373 you will see some exampel fo what the unit test should look like | |
| 16:59:58 | Uggla | bauzas, thanks | |
| 17:01:03 | ishanwar[m] | Hmm I can see mock objects sean-k-mooney , I'll have a detailed look | |
| 17:01:07 | ishanwar[m] | thanks | |
| 17:01:53 | sean-k-mooney | yep we mock the calls to neutron and assert they are called or not called depending on what should happen | |
| 17:02:18 | sean-k-mooney | ishanwar[m]: you should be able to find an exeitin tst for the validate_networks funciton | |
| 17:02:58 | sean-k-mooney | make a copy and test that it calls or does not call list_prots by mocking teh function that will check for the extension | |
| 17:07:02 | ishanwar[m] | aah i see, right, I think can use that for checking whether the function is available. | |
| 18:02:17 | cardoe | Will I be able to sneak https://review.opendev.org/c/openstack/nova/+/942019 into 2025.1 or will that have to be a backport? | |
| 18:03:28 | sean-k-mooney | cardoe: unless its a regression introduced in 2025.1 it will be a a post release backport | |
| 18:03:44 | cardoe | it's a regression introduced in 2024.1 | |
| 18:03:46 | sean-k-mooney | cardoe: ah the loggign fix | |
| 18:03:55 | sean-k-mooney | ya so that wont qualify for rc2 | |
| 18:04:08 | sean-k-mooney | and we just? curt the stable branch form rc1 | |
| 18:04:28 | sean-k-mooney | so it can merge on master but cant be backported until after 2025.1 is released | |
| 18:04:34 | cardoe | sounds good. | |
| 18:15:53 | opendevreview | sean mooney proposed openstack/nova master: Force api to exit if exceptions are uncaught https://review.opendev.org/c/openstack/nova/+/944933 | |
| 18:37:03 | opendevreview | Merged openstack/nova stable/2025.1: Update .gitreview for stable/2025.1 https://review.opendev.org/c/openstack/nova/+/944888 | |
| 18:37:09 | opendevreview | Merged openstack/nova stable/2025.1: Update TOX_CONSTRAINTS_FILE for stable/2025.1 https://review.opendev.org/c/openstack/nova/+/944889 | |
| 18:42:16 | dansmith | bauzas: Uggla are we good to go on master based on ^ ? | |
| 18:42:26 | dansmith | if so I'll +W gibi's fix series | |
| #openstack-nova - 2025-03-19 | |||
| 00:25:07 | opendevreview | sean mooney proposed openstack/nova master: Force api to exit if exceptions are uncaught https://review.opendev.org/c/openstack/nova/+/944933 | |
| 04:01:20 | opendevreview | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.opendev.org/c/openstack/nova/+/944963 | |
| 09:51:42 | opendevreview | Adam Kijak proposed openstack/nova master: Fix race condition when attaching a volume and deleting the instance https://review.opendev.org/c/openstack/nova/+/944976 | |
| 12:30:51 | opendevreview | Stephen Finucane proposed openstack/nova master: Remove tags from README https://review.opendev.org/c/openstack/nova/+/944994 | |
| 13:43:11 | opendevreview | Pierre Riteau proposed openstack/nova master: Fix missing backtick in configuration option help https://review.opendev.org/c/openstack/nova/+/931253 | |
| 13:43:44 | priteau | Hello Nova cores. This trivial doc fix has been open for a few months ^ | |
| 13:43:51 | priteau | Could you please review? | |
| 14:23:12 | opendevreview | sean mooney proposed openstack/nova master: Force api to exit if exceptions are uncaught https://review.opendev.org/c/openstack/nova/+/944933 | |
| 14:24:02 | opendevreview | Bogdan Dobrelya proposed openstack/nova master: [DNM] Hard OS exit instead of SysExit exception https://review.opendev.org/c/openstack/nova/+/945011 | |
| 14:26:42 | opendevreview | sean mooney proposed openstack/nova master: Force api to exit if exceptions are uncaught https://review.opendev.org/c/openstack/nova/+/944933 | |
| 14:31:03 | opendevreview | Bogdan Dobrelya proposed openstack/nova master: [DNM] Hard OS exit instead of SysExit exception https://review.opendev.org/c/openstack/nova/+/945011 | |
| 15:10:59 | dansmith | sean-k-mooney: so one approach is to try to exit really really hard to get apache/mod_wsgi to notice... but, | |
| 15:11:31 | dansmith | what about us making sure we never raise and just return 500s if we would have otherwise failed? | |
| 15:11:48 | sean-k-mooney | hum | |
| 15:11:50 | dansmith | even without a dedicated healthcheck that would be easy to arrange to not be healthy | |
| 15:12:09 | sean-k-mooney | we would have to kind of hack it | |
| 15:12:10 | dansmith | curl /versions or whatever, always returning 500 would leave a pod unsuitable for traffic | |
| 15:12:23 | sean-k-mooney | but we could set a module level global i guess | |
| 15:12:30 | sean-k-mooney | and check that really early | |
| 15:12:30 | dansmith | sure, we'd need some way to record it and continue logging the config error or whatever so the operator saw it in the logs | |
| 15:12:47 | dansmith | sean-k-mooney: or just make application() return a different stub object if we failed to init the real one | |
| 15:13:12 | sean-k-mooney | perhaps. the issue is it reloading that applciation into the exisitng interperter | |
| 15:13:36 | sean-k-mooney | so we keep all the module level state like that partly parsed oslo config global | |
| 15:13:45 | dansmith | do we need to? | |
| 15:14:09 | sean-k-mooney | do we need to keep that state? | |
| 15:14:19 | dansmith | we could just have a safety proxy that sits around the real one and keeps trying to load the real one, eventually proxying to it if it loads, or returning the 500s if not | |
| 15:14:23 | sean-k-mooney | i was condiring trying to reset it so that we woul parse the config and raise the excption again | |
| 15:14:48 | dansmith | I kinda don't think we need to worry about the partial state | |
| 15:15:07 | sean-k-mooney | well that why its getting to the microverison endpoint when its reloaded | |
| 15:15:27 | dansmith | I imagine everyone doing this in a modern way is going to fix the config and restart a container when they do, so just having it be a terminal state if config is broken seems fine to me | |
| 15:15:27 | sean-k-mooney | just enough fo the state is thare that we get passed where the excption woudl be raised | |
| 15:15:55 | sean-k-mooney | right im just wondering how to make it terminal | |
| 15:16:29 | dansmith | I mean if mod_wsgi reloads us and we have to try initializing again and still fail, who cares? | |
| 15:16:35 | sean-k-mooney | im totally fine with changing the guts of the decorator to do something like record if it ever raised an excption | |
| 15:16:46 | sean-k-mooney | and then keep raising that if invoked in the future | |
| 15:16:51 | sean-k-mooney | that would be pretty clean i think | |
| 15:17:17 | sean-k-mooney | dansmith: right now when we are reloaded we dont fail on the init the second time | |
| 15:17:20 | dansmith | yeah, just set a global if we failed, stop trying, just return all the 500s, wait for them to fix config and restart | |
| 15:17:21 | sean-k-mooney | becuase of the leaked state | |
| 15:17:36 | dansmith | sean-k-mooney: ahh, I see, then yeah, some global to prevent us from trying | |
| 15:17:49 | sean-k-mooney | dansmith: lets wait for the current ci job to complete | |
| 15:18:05 | sean-k-mooney | ill refactor to raise 500 if we ever raised before and see what that looks like | |
| 15:18:07 | dansmith | I just feel like trying to sys.exit() harder and harder is sort of missing the point | |
| 15:18:12 | bogdando | we also have one testing the os._exit "fork" | |
| 15:18:25 | sean-k-mooney | ya i dont like calling that ^ | |
| 15:18:35 | dansmith | we should just detect, log, and signal (which for a web application is a 500 healthcheck, IMHO) | |
| 15:18:48 | dansmith | sean-k-mooney: agree about os._exit, that's what I mean by "harder" :) | |
| 15:19:23 | sean-k-mooney | ya so i dont know if you looked at my most recent version wich follows the reload docs | |
| 15:19:36 | sean-k-mooney | i choose to use sig kill instead of sig interupt | |
| 15:19:51 | sean-k-mooney | but either i think woudl work | |
| 15:20:12 | bogdando | sys exit is trying the door, but apache won't let us out | |
| 15:20:20 | sean-k-mooney | dansmith: i think your suggestion is worth trying as it shoudl be pretty clean. | |
| 15:20:26 | bogdando | os.exit - trying a window | |
| 15:20:38 | bogdando | 500 code - sitting in the room arguing with the door guard | |
| 15:21:04 | sean-k-mooney | 500s are more polite | |
| 15:21:13 | dansmith | sean-k-mooney: to me, the kill/reload thing is what we should do if we need to retry, but for bad config or almost all of our situations, we don't | |
| 15:21:20 | dansmith | so doing kill there just means we crashloop | |
| 15:21:44 | dansmith | starting up and returning 500 will signal that we're not happy instead of constantly looking like we're restarting | |
| 15:21:49 | sean-k-mooney | yep | |
| 15:22:02 | dansmith | I much prefer the "unhealthy" marker instead of the constant "starting..." status where you're not sure if you're still supposed to wait | |
| 15:22:10 | sean-k-mooney | long term either implemeting helthchecks or hooking the system ready socket is cleaner | |
| 15:22:26 | bogdando | the problem is we need a hard stop, no half measures | |
| 15:22:27 | sean-k-mooney | i.e. for knowing if we started properly | |
| 15:22:43 | bogdando | letting the service operating half-configured even a fraction of a second could be destructive | |
| 15:23:15 | sean-k-mooney | well if my current patch works it wont | |
| 15:23:21 | bogdando | (if we got unlucky enough) | |
| 15:23:25 | sean-k-mooney | and if we do the 500 thing it wont either | |
| 15:23:30 | dansmith | it's not going to operate half-configured either way if we never return application | |
| 15:23:32 | dansmith | right | |