| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-01 | |||
| 16:15:05 | Uggla | #link Blog: https://gibizer.github.io/categories/eventlet/ | |
| 16:15:14 | gibi | o/ | |
| 16:15:14 | Uggla | #link nova-scheduler series is ready for core review, starting at https://review.opendev.org/c/openstack/nova/+/947966 | |
| 16:15:19 | Uggla | gibi the mic is yours | |
| 16:15:32 | gibi | so the scatter gather refactor patch has been landed | |
| 16:15:37 | gibi | thanks for the reviews | |
| 16:15:40 | Uggla | \o/ | |
| 16:16:01 | gibi | the next two patches are also approved but the gate was / is pretty flaky in the recent days so they are not landed yet | |
| 16:16:12 | gibi | (hybrid-plug and ceph issuees) | |
| 16:16:47 | gibi | I fixed the comments form bauzas on the next patch in the series about a possible race condition. Please take a look | |
| 16:17:09 | gibi | On the top of the series I started an effor to run unit tests without eventlet | |
| 16:17:29 | gibi | the effort starts here https://review.opendev.org/c/openstack/nova/+/953436/5 | |
| 16:18:07 | gibi | couple of patches later in that series we have a job that runs most of our unit tests without monkey patching and with threading oslo.service and oslo.messaging backend | |
| 16:18:19 | gibi | there is a list of excluded test I have to work through | |
| 16:18:57 | gibi | I'm tracking to type of problems. Sqlaclhemy nonsense errors and simple test case hangs | |
| 16:19:04 | gibi | s/to/two/ | |
| 16:19:15 | gibi | you can see them noted in the test exclude list | |
| 16:19:37 | gibi | I think we can make the new zuul job voting pretty soon | |
| 16:19:45 | gibi | (while keeping some excludes) | |
| 16:20:23 | gibi | that is it from me | |
| 16:20:51 | sean-k-mooney | o/ sorry had this behind anohter window | |
| 16:21:48 | Uggla | gibi do you have some kind of "urgent" review anymore ? | |
| 16:22:01 | bauzas | I'll continue to review the series for sure | |
| 16:22:25 | gibi | Uggla: the bottom of the series is as urgent as the eventlet removal :) | |
| 16:22:51 | gibi | bauzas: hanks | |
| 16:22:54 | gibi | thanks | |
| 16:24:23 | gibi | I think we can move on :) | |
| 16:24:30 | Uggla | BTW we have Nova Eventlet removal sync tomorrow too. | |
| 16:25:11 | Uggla | 14:30 UTC | |
| 16:25:12 | gibi | yepp we have the call | |
| 16:25:14 | gibi | as planned | |
| 16:25:32 | Uggla | thanks gibi, next topic | |
| 16:25:39 | Uggla | #topic Open discussion | |
| 16:25:47 | Uggla | #topic (fwiesel) Evacuate Action & InstanceInvalidState | |
| 16:26:20 | Uggla | fwiesel please go ahead. | |
| 16:26:28 | fwiesel | So, we are currently looking into doing HA by using the evacuate action on instances, and were running into various states, where it was not possible. | |
| 16:27:00 | fwiesel | From our side, evacuation usually happens when the host is down so can happen at any time. | |
| 16:27:39 | fwiesel | And I was wondering, if it wouldn't make sense to adjust the code so it can evacuate an instance in as many sitations as possible. | |
| 16:28:47 | fwiesel | So, raising an InstanceInvalidState is merely a reflection of what has been implemented. What do you say? | |
| 16:29:44 | gibi | do you have specific examples of states you want to evacuate from? | |
| 16:29:48 | cardoe | So I'll say internally we've got extra stuff which we don't see nova ever being able to handle. | |
| 16:29:59 | cardoe | Like even human involved state | |
| 16:30:37 | fwiesel | Let me think. For starters: Verify-Resize. | |
| 16:31:13 | fwiesel | But let's say, we have a live-migration, and that host falls down. I still would like to do a failover | |
| 16:31:57 | gibi | verify-resize: so while you could evacuate from that state you also need to implement a proper cleanup in the nova-compute coming up in the source node of the original resize | |
| 16:32:00 | sean-k-mooney | so evacuate requires you the admin | |
| 16:32:11 | sean-k-mooney | to assert that the vm is not running on any host | |
| 16:32:31 | sean-k-mooney | so allowing it in verify resize and othe rstates is questionable | |
| 16:32:52 | sean-k-mooney | espically since in that state your in the middle of an ongoing migration | |
| 16:32:55 | cardoe | Yeah there's a lot of squirrely cases there which need to be monitored for. | |
| 16:33:07 | sean-k-mooney | and the flavor that is in use is kind of ill defiend | |
| 16:33:47 | sean-k-mooney | its technialy the new one in a real reaise but it has nto beeen confirm so you woudl want ot evacuate to the old one, but hte numa toplogy and ohter things are all in a weird hybrid state | |
| 16:34:28 | sean-k-mooney | what you likely should be able to do is revert the resize in verify-resize | |
| 16:34:29 | gibi | so in general it would be nice to support these but in practic it seems like a bunch of cans of worms | |
| 16:34:37 | sean-k-mooney | even if the dest host is down | |
| 16:34:41 | sean-k-mooney | then evacuate it | |
| 16:34:43 | sean-k-mooney | if needed | |
| 16:35:13 | cardoe | We've got a whole thing out of band that's specific to the hypervisor in use for these items. | |
| 16:35:31 | gibi | yeah allowing revert-resize in souce host down is one way, even though a weird one. Who wants to revert a resize that will lead to the VM runing nowhere :) | |
| 16:35:33 | fwiesel | But technically, an evacuate is practically a rebuild. So, I can start from "scratch" (nova wise) | |
| 16:35:54 | sean-k-mooney | gibi: well it would revert reisze when dest down and conrim resize when source down | |
| 16:35:56 | fwiesel | I just have to build the VM up somewhere fitting the spec | |
| 16:36:05 | gibi | yeah starting is easy, cleaning up the remnants is messy | |
| 16:36:12 | sean-k-mooney | fwiesel: yes and no | |
| 16:36:13 | cardoe | yeah but you need to make sure that gets fully cleaned up. | |
| 16:36:20 | fwiesel | Of course | |
| 16:36:25 | sean-k-mooney | we have some api contract about preserving storage in some cases | |
| 16:36:40 | sean-k-mooney | please there are the neutorn/cidner state clean up that we need to do | |
| 16:36:48 | gibi | sean-k-mooney: revert-resize: ahh right | |
| 16:36:48 | fwiesel | Exactly | |
| 16:37:07 | fwiesel | Right now, I have the choice to implement all that outside of nova. Or within nova | |
| 16:37:25 | sean-k-mooney | to do this in nova woudl need a very detailed spec | |
| 16:37:41 | fwiesel | Well, I would not tackle all cases in one go. | |
| 16:37:42 | sean-k-mooney | and you woudl have to consier things liek cross cell resize | |
| 16:38:01 | fwiesel | And maybe then not resize :) | |
| 16:38:15 | cardoe | its a worthy goal | |
| 16:38:18 | sean-k-mooney | right but even if you take an incremntal approch you need to have each incremantal chagne be valid | |
| 16:38:55 | sean-k-mooney | i think for rezie it woubld be siiler to allwo the migration to compelte or revert based on which end is alive | |
| 16:38:57 | cardoe | I'll say when I did this with xen it was much easier because xapi had an object version field so I used that as a monotonic number which also then was stored in the nova DB | |
| 16:39:03 | sean-k-mooney | and if they are both dead well your kind of out of luck | |
| 16:39:10 | fwiesel | Of course. I was more fearing that it will be difficult to test that in the CI, and adds complexity. Which might be areason you might not want it at all. | |
| 16:39:26 | cardoe | So each time the instance got updated the monotonic number increased and that got pushed down into xapi | |
| 16:39:28 | gibi | fwiesel: start with a simpler state first, write up a spec how would you make sure we end up in the consistent state after the a successful evac and a recovered compute (or an failed evac and a recovered compute, of a failed evac and a retried evac and recovered compute, ...) | |
| 16:39:31 | cardoe | But libvirt doesn't have that | |
| 16:39:47 | cardoe | So when we'd go to clean up, anything that wasn't at the current version could get nuked. | |
| 16:39:59 | sean-k-mooney | fwiesel: so testing that in tempest would be hard, but it would be more doable in the functional suite | |
| 16:40:03 | cardoe | But there's all kinds of dangling references to storage and what not that really are a pain to fix | |
| 16:40:36 | gibi | yepp use functional test for sure | |
| 16:40:51 | fwiesel | Okay, I'll take that with me. Thanks. | |
| 16:40:52 | gibi | we have a limited evact test somewhere after tempest | |
| 16:40:57 | sean-k-mooney | cardoe: so we dont need montic object version in general becasue we assocate the instance with the comptue service that is currently manging it | |
| 16:40:58 | gibi | in a palybook | |
| 16:41:11 | sean-k-mooney | in the post test hook in nova-next | |
| 16:41:16 | cardoe | For us we implemented this more as monitoring that an unexpected object was on a host. | |
| 16:41:17 | sean-k-mooney | and i think the cepch job? | |
| 16:42:03 | sean-k-mooney | cardoe: ya so that kind of approch is not very comaptible with the nova desing although we do have some per host perodic that are intened ot clean up these types of issues | |
| 16:42:08 | cardoe | sean-k-mooney: when you perform a live migrate and the dest goes down at the very end during the resume | |
| 16:42:09 | sean-k-mooney | esplcially on compute agent start up | |
| 16:42:53 | sean-k-mooney | cardoe: if the vm dies before resume libvirt will abort the migraiot | |
| 16:43:01 | sean-k-mooney | if we got to post live migrate | |