| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-08-27 | |||
| 14:29:01 | sean-k-mooney | dansmith: the answer to https://review.opendev.org/c/openstack/placement/+/945464/comment/0189c002_387845d8/ was just the test is doing to diffent things so i woudl have made it two tests | |
| 14:29:38 | sean-k-mooney | ill just ack tthat comment and we can leave it as is | |
| 14:32:14 | sean-k-mooney | the repoducer looks fine ill need to spen a littel more time on the actual fix but ill do that later today | |
| 14:32:17 | ralonsoh | sean-k-mooney, yes, you can rely on this flag instead of the port binding | |
| 14:32:28 | ralonsoh | we are setting now in both places (the field and port binding) | |
| 14:32:39 | ralonsoh | we'll remove the last one (port binding) once accepted in Nova | |
| 14:33:36 | sean-k-mooney | ack. ill add nova ot the bug and check with other if we are ok to fix and backport that as a security hardening bug | |
| 14:33:55 | sean-k-mooney | if not then ill propose doing it as a specless blueprint next cycle | |
| 14:42:49 | dansmith | gibi: I was going to ping our PTL, but ... https://review.opendev.org/c/openstack/nova/+/957424 | |
| 14:43:00 | gibi | dansmith: I'm +2 on both patch in placement about the overallocation. Thanks both was pretty clear. | |
| 14:43:01 | dansmith | I wonder if we should point them to a specific outlet (or outlets) to report success | |
| 14:43:07 | dansmith | gibi: sweet thanks | |
| 14:44:04 | gibi | dansmith: I can ask them to drop a mail to the ML or show up on IRC. | |
| 14:44:41 | gibi | probably ML is easier for them | |
| 14:44:45 | dansmith | yeah the former is what I was thinking.. I know some may not even go that far, so I wish we had another venue but.. at least asking for success reporting seems like a good idea | |
| 14:45:01 | gibi | I will respin and add it | |
| 14:45:12 | gibi | (and in the same time I will remove the nova-conductor from the series) | |
| 14:45:18 | dansmith | maybe put bauzas' personal email and cell number (as our closest TC rep) in there if they don't want to join the ML :D | |
| 14:45:30 | gibi | lol :) | |
| 14:45:32 | bauzas | hmmm, WAT ? | |
| 14:45:33 | dansmith | gibi: also why not just squash those renos into one? | |
| 14:45:54 | bauzas | $context ? | |
| 14:46:00 | gibi | dansmith: about the renos there. We already have two from two independent patch but yeah I can try to merge them in that commit and see what our doc gen says about a deleted reno | |
| 14:46:21 | dansmith | not a big deal but when I read the second one I thought it was accidental duplication until I flipped back and forth a couple times to see | |
| 14:46:33 | dansmith | gibi: oh right, duh, okay nevermind then | |
| 14:47:41 | dansmith | usually renos in patches are new and I was reading it as if you were adding those.. no need to fix now I think | |
| 14:47:50 | gibi | OK | |
| 15:01:08 | gibi | Uggla: ping :) | |
| 15:02:21 | Uggla | gibi pong ! :) | |
| 15:02:48 | gibi | probably a netspilt happened between the IRC servers but those should be reported to us. At least in the univesity days that was somewhat frequent and that was a way to steal nicks :) | |
| 15:11:59 | bauzas | gibi: dansmith: what was the problem with reno ? sorry, hardly following | |
| 15:12:10 | dansmith | yeah I think after some netsplit my nick list just didn't recover.. I restarted my client and it's back now | |
| 15:12:13 | dansmith | bauzas: no problem | |
| 15:12:16 | bauzas | also, I'd appreciate if some cores could give a look at tkajinam's series on SEV ES | |
| 15:12:53 | bauzas | I reviewed that, Uggla found some typos that tkajinam will update but apart from that, we're in a very good spot to support next generation of AMD SEV :) | |
| 15:12:57 | dansmith | bauzas: as I said, I'm not reviewing that before tomorrow.. I'm totally blank on it and don't think I'd be doing a good job in such a short time frame... just FYI | |
| 15:13:22 | bauzas | dansmith: well, it was a general reminder to all of us, not particularly you who already said that indeed, cool | |
| 15:13:35 | dansmith | yep, I know, just sayin' :) | |
| 15:14:30 | bauzas | there is also a quite simple feature patch worth reviewing, which is to copy our provider.yaml, I started reviewing it and I have an open question : https://review.opendev.org/c/openstack/nova/+/948304 | |
| 15:14:42 | bauzas | we could fix that in a FUP if other cores like it | |
| 15:15:41 | gmaan | dansmith: RE 957578: ah it merge conflict, I will rebase it. it was all good until last night. testing and everything done for this and it is good to review (once i rebase) | |
| 15:15:48 | dansmith | ack | |
| 15:16:24 | bauzas | gmaan: I can give it a shot | |
| 15:16:40 | bauzas | fwiw, the etherpad is updated with my findings https://etherpad.opendev.org/p/nova-2025.2-status | |
| 15:16:50 | bauzas | people can use it for directing reviews | |
| 15:29:18 | lajoskatona | bauzas, Uggla: Hi, perhaps you can help me out, we have a bug originally from a downstream customer: https://bugs.launchpad.net/nova/+bug/2051685 for migration | |
| 15:30:22 | lajoskatona | bauzas, Uggla: it hit us few cycles back and than rubasov made an upstream reproduction which you can read in the bug description but he hit some walls in understanding the depths of Nova as you can read at the end of the report :-) | |
| 15:31:59 | lajoskatona | bauzas, Uggla: if we should have a pointer if the issue is really an issue and can be fixed would be enough help to keep working on it but we lost how to dig deeper and if it is really something that can be / worth to be fixed | |
| 15:33:27 | Uggla | lajoskatona, at least I can add this bug for review in our next bug scrubbing. | |
| 15:34:10 | lajoskatona | Uggla: Thanks for your time, if any more info is needed we can check and add o course to the LP | |
| 15:35:21 | Uggla | lajoskatona, I'll try to discuss it in next nova meeting. But I can't promise we will have time, depending of the agenda. | |
| 15:36:14 | lajoskatona | Uggla: no problem, thanks | |
| 16:15:21 | sean-k-mooney | lajoskatona: so use reset state after a migration error si never corect | |
| 16:15:31 | sean-k-mooney | so if this only happens when you do that its not really a valid bug | |
| 16:16:47 | sean-k-mooney | lajoskatona: also im surpesed we accepted --availability-zone :devstack0 | |
| 16:17:05 | sean-k-mooney | in new micorversion you ment to pass --host | |
| 16:17:16 | sean-k-mooney | if you jsut want to slect a host | |
| 16:18:01 | sean-k-mooney | i tought the az was intended to be requried and just passing :<host> was not intended to be supprot but perhaps that worked by acidnet | |
| 16:18:08 | lajoskatona | sean-k-mooney: reset state was only used to show that in devstack the issue can happen, as it happened in some customer site without proper logs and reproduction | |
| 16:18:12 | sean-k-mooney | we never really encurraged that usage | |
| 16:18:50 | sean-k-mooney | right but reset state will break novas ablity ot properly clean up | |
| 16:19:31 | sean-k-mooney | as will restarting the nova-comapute durign the mighration | |
| 16:20:31 | lajoskatona | the issue is that when something happens with the destination host during migration the migration can stuck in error case | |
| 16:20:54 | sean-k-mooney | well yo say stuck in error but i tough erro was a valid end state | |
| 16:21:04 | sean-k-mooney | so its not stuck its in the terminal state | |
| 16:21:10 | sean-k-mooney | or one of the terminal states | |
| 16:21:20 | lajoskatona | as rubasov added to the last lines as a thought experiment: "I don't see it yet, how this should be fixed. Maybe a migration should never be stuck indefinitely in status=error...." | |
| 16:21:27 | sean-k-mooney | failed models a diffent endsate then error | |
| 16:22:12 | sean-k-mooney | error is something went wrong that we cant recover form gracefully liek the dest host gets restarted | |
| 16:22:27 | lajoskatona | yes of course | |
| 16:22:29 | sean-k-mooney | failed is the end state when it somethig we can properly recover form | |
| 16:22:38 | sean-k-mooney | like we try to migrate and the cpu is incompatible | |
| 16:23:13 | sean-k-mooney | so the migration status being in error for ever is nto a bug to me | |
| 16:23:51 | sean-k-mooney | the orignal issue was caused by "overcommit on dedicated PCPUs and CPUPinningInvalid exception" | |
| 16:24:23 | sean-k-mooney | I.E. there was already invlaid sate in one or boht of the compute nodes | |
| 16:24:52 | lajoskatona | and the update_available_resource periodically fetches these also and tries to apply the migration that ended in error state but the repeated migration for the same server to the same host succeded so it should be find by the method that the migration is really done by a new migration (server uuid - dest host pair) | |
| 16:24:54 | sean-k-mooney | i.e. vms not runningn on the right core or the oepration modfied the nova config to change the core set | |
| 16:26:41 | lajoskatona | so the question is if we can say that ahhh, the same VM to the same host migration operation is finished so we can forget to stress on the previous migration (same vm uuid same host) that failed ? | |
| 16:28:13 | sean-k-mooney | in general no | |
| 16:28:17 | lajoskatona | so it is really a corner case as I see not error due to the things you mentioned (pcpu config or such) but like issue with the compute itself and after the fix the migration can be repeated | |
| 16:28:36 | sean-k-mooney | but we liekly do have a bug in the logic | |
| 16:29:00 | sean-k-mooney | i dont thnk it woudl be correct for the erroed migration to be updated to failed | |
| 16:29:06 | sean-k-mooney | is shoudl stay in error | |
| 16:29:27 | sean-k-mooney | but if the vm is not on this host we shoudl not process the old migration on new iterations | |
| 16:29:39 | sean-k-mooney | of the update_aviable_resouces perodic | |
| 16:29:49 | sean-k-mooney | movign it to fialed might achive that | |
| 16:29:54 | lajoskatona | that's fair to rethink it and make it clear and we can document it for customers in such case in worst case | |
| 16:30:07 | sean-k-mooney | but im not sure we can alwasy assume that is safe to do | |
| 16:30:29 | lajoskatona | sounds great to think at least about it | |
| 16:31:10 | sean-k-mooney | you were testing bfv was it using ceph by the way? | |
| 16:31:48 | sean-k-mooney | if so the already exist erro might eb related to https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.ensure_libvirt_rbd_instance_dir_cleanup | |
| 16:31:54 | lajoskatona | to tell the truth I am not sure but would say no (have to double check) | |
| 16:32:53 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1414895 | |
| 16:32:59 | sean-k-mooney | is one of the rrefenece bugs | |
| 16:33:59 | sean-k-mooney | we dont generally delete the instnace storage on the dest on start up because we dont want ot loose data | |
| 16:34:25 | sean-k-mooney | i.e. incase the live migration actuly commpelted at teh libvirt level while the agent was being restarted | |
| 16:34:53 | lajoskatona | thanks I check this issue also if can be related | |
| 16:34:55 | sean-k-mooney | when an migration ends in error it means an operator might need to fix the db | |
| 16:35:21 | sean-k-mooney | to update the host to bring nova back in sync with where the vm is. or carfully move the vm back ot the souce host | |
| 16:35:40 | sean-k-mooney | if a migration end in failed there shoudl be no action needed by an operator to fix anything | |