| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-22 | |||
| 16:54:51 | bauzas | I just hope this meeting won't transform into some CI meeting | |
| 16:54:57 | chateaulav | artom: nova zuul has an example of weekly periodic now | |
| 16:55:02 | bauzas | if we start adding more periodics | |
| 16:55:22 | artom | I mean, feel free to nack the idea entirely :) | |
| 16:55:29 | sean-k-mooney | bauzas: well it should just be (are they green no we shoudl look at X after the meeting) | |
| 16:55:31 | artom | I'll obviously try to debate/convince you | |
| 16:55:37 | bauzas | artom: nah, I like the idea, I just want us to buy it | |
| 16:55:47 | artom | But if we think whitebox doesn't bring value to Nova CI, let's just not do it :) | |
| 16:56:04 | bauzas | we're approaching meeting's end time | |
| 16:56:14 | artom | End times are nigh | |
| 16:56:18 | bauzas | any other item to mention before we close ? | |
| 16:56:19 | sean-k-mooney | :) | |
| 16:56:33 | sean-k-mooney | ah i actully had two blueprints i wanted to raise | |
| 16:56:47 | sean-k-mooney | we defered updating the defaults for allcoation ratios | |
| 16:57:00 | bauzas | sean-k-mooney: oh I forgot to mention I changed Launchpad to reflect zed as the active series | |
| 16:57:16 | sean-k-mooney | shall we proceed with that or discuss at ptg | |
| 16:57:29 | sean-k-mooney | also kasyaps blueprint for usign the new libvirt apis | |
| 16:57:35 | bauzas | we're a bit short in time for reapproving specless bps by now | |
| 16:57:37 | sean-k-mooney | can we retarget both to zed | |
| 16:57:45 | sean-k-mooney | ack | |
| 16:57:52 | bauzas | but we can look at them during next meeting | |
| 16:57:52 | sean-k-mooney | we can disucss it next week or at ptg | |
| 16:58:03 | bauzas | well, Zed is open | |
| 16:58:13 | bauzas | I'm OK with approving things by now | |
| 16:58:19 | bauzas | and the specs repo is ready | |
| 16:58:50 | bauzas | sean-k-mooney: just propose your two blueprints for the next meeting so we'll reapprove them (unless concerns of course) | |
| 16:59:04 | sean-k-mooney | ack | |
| 16:59:29 | bauzas | fwiw, I leave the non-implemented blueprints in Deferred state | |
| 17:00:01 | bauzas | once we start reapproving some, I'd change back their state | |
| 17:00:11 | bauzas | but anyway, we're on time | |
| 17:00:13 | bauzas | thanks all | |
| 17:00:16 | bauzas | #endmeeting | |
| 17:00:16 | opendevmeet | Meeting ended Tue Mar 22 17:00:16 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 17:00:16 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-03-22-16.00.html | |
| 17:00:16 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-03-22-16.00.txt | |
| 17:00:16 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-03-22-16.00.log.html | |
| 17:00:25 | elodilles | thanks bauzas o/ | |
| 17:00:34 | bauzas | was a productive meeting, after all | |
| 17:03:30 | EugenMayer | Is there any 'good way' to set the task-state of an instance that has been stuck in 'image backup' due to an issue in glance? so the field OS-EXT-STS:task_state is on "image_backup" | |
| 17:05:07 | EugenMayer | i see there is 'nova set --state' or 'nova reset-state' but both seeem to operate on the instance-power-state (OS-EXT-STS:power_state) or OS-EXT-STS:vm_state - but not the task-state | |
| 17:07:20 | zigo | sean-k-mooney: Yeah, this was an evacuate operation. | |
| 17:07:47 | dansmith | zigo: I thought you said live migrate? | |
| 17:07:50 | sean-k-mooney | zigo: ok the reason this breaks is for evacuation we only have 1 allocation in placemnt against both hosts | |
| 17:08:17 | sean-k-mooney | and since the souce host is over capstiy because you reduce the allocate ration the entire allcoation is considered invlaid | |
| 17:08:34 | sean-k-mooney | we disussed this at the ptg 1 or 2 ptgs ago | |
| 17:09:13 | sean-k-mooney | i cant recall if we said we should fix this after consumer types but i dont think we had a workaround other then tempoarly increase the allcoation ratio so its nolonger over commited | |
| 17:09:55 | dansmith | sean-k-mooney: we could also solve it the way we do for cold migration, which is hold the allocation on the source with the migration uuid right? | |
| 17:10:13 | sean-k-mooney | dansmith: yes we could that was on eof the options | |
| 17:10:50 | sean-k-mooney | im trying to find the launchpad bug | |
| 17:11:44 | bauzas | dansmith: sean-k-mooney: yeah, the Migration uuid for evacuate seems the better and cleaner approach | |
| 17:12:04 | sean-k-mooney | bauzas: that is what we were proposing doing | |
| 17:12:26 | sean-k-mooney | but i dont think anyone has worked on it since | |
| 17:13:39 | bauzas | :-) | |
| 17:13:47 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1943191 | |
| 17:13:49 | sean-k-mooney | that might be it | |
| 17:13:58 | EugenMayer | I'am looking on https://wiki.openstack.org/wiki/CrashUp/Recover_From_Nova_Uncontrolled_Operations to understand how to recover from the crashed task state 'image_backup' but i'am not sure how to actual act upon that. Should i use the nova api? | |
| 17:14:03 | sean-k-mooney | and https://bugs.launchpad.net/nova/+bug/1924123 | |
| 17:14:37 | bauzas | sean-k-mooney: some people expect bugs to be fixed automatically :) | |
| 17:14:55 | bauzas | we don't have yet AI bots smart enough to close the gaps | |
| 17:14:58 | sean-k-mooney | EugenMayer: the wiki is basicaly unmaintained | |
| 17:15:09 | EugenMayer | i see. Thank you | |
| 17:16:03 | sean-k-mooney | in the early days of openstack we used the wiki for sepc and project created docs(docs not by the docs team) | |
| 17:16:14 | EugenMayer | I'am really not sure hot to again recover from the failed task the proper way. The only way i yet know, which is huge is: reset the state, then restart the compute the vm is hosted so thee state is somewhat recovered | |
| 17:17:13 | sean-k-mooney | there is not way to recover form it really beyond that | |
| 17:17:29 | sean-k-mooney | we dont provide a api to allow taskt to be restarted | |
| 17:17:53 | dansmith | reset state and reboot the vm is what I'd try first, | |
| 17:18:00 | sean-k-mooney | yep same | |
| 17:18:03 | dansmith | not restarting the compute I'd hope | |
| 17:18:16 | sean-k-mooney | ya that normally shoudl not be required | |
| 17:18:25 | sean-k-mooney | i guess it woudl depend on why it failed | |
| 17:18:43 | dansmith | definitely not expected for anything like a glance thing | |
| 17:18:50 | EugenMayer | trying that. AFAIR i had to restart the entire compute last time. Anyway, trying that | |
| 17:19:17 | sean-k-mooney | do you recall way? | |
| 17:19:20 | sean-k-mooney | *why | |
| 17:19:23 | EugenMayer | dansmith well this happens the 4th time. A stuck glance image backup task leaves the task_state of the instance in a broken state | |
| 17:19:26 | dansmith | honestly restarting the compute shouldn't even do anything, AFAIK | |
| 17:19:59 | sean-k-mooney | i wonder if the main thread of the compute agent was blocked on an io operations | |
| 17:20:16 | sean-k-mooney | that is the only thing i can think of that would be fixed by an agent restart | |
| 17:20:41 | sean-k-mooney | we were not using a thread pool on some of the older release for those | |
| 17:20:53 | dansmith | sean-k-mooney: compute is the thing that "consumes" the task_state and turns it into a vm_state, so to speak, so maybe we clear task_state in init_host in some cases? | |
| 17:20:58 | EugenMayer | well i'am on xena, so not really old | |
| 17:21:13 | dansmith | but either way, reset_state to error is supposed to let you clear everything by enabling force reboot I think | |
| 17:21:17 | dansmith | or that's the intent | |
| 17:21:18 | sean-k-mooney | dansmith: i think we do yes but not sure about this case | |
| 17:22:01 | EugenMayer | dansmith it is clear, swt wise, that there is more then one misconception in the microservice and task callstack. I'am not sure if glance is required to call a webhook on success or error (not sure how the result is propagated) but this is simply not the right design. | |
| 17:22:30 | EugenMayer | should the task crash on glance, neither success nor error is called (ever) and there seems nothing to recover from that | |
| 17:22:42 | dansmith | EugenMayer: none of that :) | |
| 17:22:58 | dansmith | everything is nova->glance | |
| 17:23:05 | sean-k-mooney | i belive this is a blocking call to do the upload to glace | |
| 17:23:15 | sean-k-mooney | if its async then either nova would poll | |
| 17:23:22 | sean-k-mooney | or we woudl get an external event form glance | |
| 17:23:26 | dansmith | so depending on the failure, nova should clean up whatever it can.. an upload to glance for sure should be recoverable on our end, so that's likely it's own bug if we're missing something | |
| 17:23:28 | sean-k-mooney | but i think image upload if blocking | |
| 17:23:35 | dansmith | sean-k-mooney: none of that with glance | |
| 17:24:00 | sean-k-mooney | right we dont do polling or external event right | |
| 17:24:07 | sean-k-mooney | we just do two blocking calls | |
| 17:24:09 | EugenMayer | if it is a blocking task, well the blocking should cleanup - which it seem to not do | |
| 17:24:15 | sean-k-mooney | one for creating the image and the second for the data upload | |
| 17:24:34 | dansmith | EugenMayer: if you can repro the problem that's definitely a bug candidate | |
| 17:24:42 | sean-k-mooney | EugenMayer: yes it should clean up if we get an error form glance | |