Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-08-27
15:16:24 bauzas gmaan: I can give it a shot
15:16:40 bauzas fwiw, the etherpad is updated with my findings https://etherpad.opendev.org/p/nova-2025.2-status
15:16:50 bauzas people can use it for directing reviews
15:29:18 lajoskatona bauzas, Uggla: Hi, perhaps you can help me out, we have a bug originally from a downstream customer: https://bugs.launchpad.net/nova/+bug/2051685 for migration
15:30:22 lajoskatona bauzas, Uggla: it hit us few cycles back and than rubasov made an upstream reproduction which you can read in the bug description but he hit some walls in understanding the depths of Nova as you can read at the end of the report :-)
15:31:59 lajoskatona bauzas, Uggla: if we should have a pointer if the issue is really an issue and can be fixed would be enough help to keep working on it but we lost how to dig deeper and if it is really something that can be / worth to be fixed
15:33:27 Uggla lajoskatona, at least I can add this bug for review in our next bug scrubbing.
15:34:10 lajoskatona Uggla: Thanks for your time, if any more info is needed we can check and add o course to the LP
15:35:21 Uggla lajoskatona, I'll try to discuss it in next nova meeting. But I can't promise we will have time, depending of the agenda.
15:36:14 lajoskatona Uggla: no problem, thanks
16:15:21 sean-k-mooney lajoskatona: so use reset state after a migration error si never corect
16:15:31 sean-k-mooney so if this only happens when you do that its not really a valid bug
16:16:47 sean-k-mooney lajoskatona: also im surpesed we accepted --availability-zone :devstack0
16:17:05 sean-k-mooney in new micorversion you ment to pass --host
16:17:16 sean-k-mooney if you jsut want to slect a host
16:18:01 sean-k-mooney i tought the az was intended to be requried and just passing :<host> was not intended to be supprot but perhaps that worked by acidnet
16:18:08 lajoskatona sean-k-mooney: reset state was only used to show that in devstack the issue can happen, as it happened in some customer site without proper logs and reproduction
16:18:12 sean-k-mooney we never really encurraged that usage
16:18:50 sean-k-mooney right but reset state will break novas ablity ot properly clean up
16:19:31 sean-k-mooney as will restarting the nova-comapute durign the mighration
16:20:31 lajoskatona the issue is that when something happens with the destination host during migration the migration can stuck in error case
16:20:54 sean-k-mooney well yo say stuck in error but i tough erro was a valid end state
16:21:04 sean-k-mooney so its not stuck its in the terminal state
16:21:10 sean-k-mooney or one of the terminal states
16:21:20 lajoskatona as rubasov added to the last lines as a thought experiment: "I don't see it yet, how this should be fixed. Maybe a migration should never be stuck indefinitely in status=error...."
16:21:27 sean-k-mooney failed models a diffent endsate then error
16:22:12 sean-k-mooney error is something went wrong that we cant recover form gracefully liek the dest host gets restarted
16:22:27 lajoskatona yes of course
16:22:29 sean-k-mooney failed is the end state when it somethig we can properly recover form
16:22:38 sean-k-mooney like we try to migrate and the cpu is incompatible
16:23:13 sean-k-mooney so the migration status being in error for ever is nto a bug to me
16:23:51 sean-k-mooney the orignal issue was caused by "overcommit on dedicated PCPUs and CPUPinningInvalid exception"
16:24:23 sean-k-mooney I.E. there was already invlaid sate in one or boht of the compute nodes
16:24:52 lajoskatona and the update_available_resource periodically fetches these also and tries to apply the migration that ended in error state but the repeated migration for the same server to the same host succeded so it should be find by the method that the migration is really done by a new migration (server uuid - dest host pair)
16:24:54 sean-k-mooney i.e. vms not runningn on the right core or the oepration modfied the nova config to change the core set
16:26:41 lajoskatona so the question is if we can say that ahhh, the same VM to the same host migration operation is finished so we can forget to stress on the previous migration (same vm uuid same host) that failed ?
16:28:13 sean-k-mooney in general no
16:28:17 lajoskatona so it is really a corner case as I see not error due to the things you mentioned (pcpu config or such) but like issue with the compute itself and after the fix the migration can be repeated
16:28:36 sean-k-mooney but we liekly do have a bug in the logic
16:29:00 sean-k-mooney i dont thnk it woudl be correct for the erroed migration to be updated to failed
16:29:06 sean-k-mooney is shoudl stay in error
16:29:27 sean-k-mooney but if the vm is not on this host we shoudl not process the old migration on new iterations
16:29:39 sean-k-mooney of the update_aviable_resouces perodic
16:29:49 sean-k-mooney movign it to fialed might achive that
16:29:54 lajoskatona that's fair to rethink it and make it clear and we can document it for customers in such case in worst case
16:30:07 sean-k-mooney but im not sure we can alwasy assume that is safe to do
16:30:29 lajoskatona sounds great to think at least about it
16:31:10 sean-k-mooney you were testing bfv was it using ceph by the way?
16:31:48 sean-k-mooney if so the already exist erro might eb related to https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.ensure_libvirt_rbd_instance_dir_cleanup
16:31:54 lajoskatona to tell the truth I am not sure but would say no (have to double check)
16:32:53 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1414895
16:32:59 sean-k-mooney is one of the rrefenece bugs
16:33:59 sean-k-mooney we dont generally delete the instnace storage on the dest on start up because we dont want ot loose data
16:34:25 sean-k-mooney i.e. incase the live migration actuly commpelted at teh libvirt level while the agent was being restarted
16:34:53 lajoskatona thanks I check this issue also if can be related
16:34:55 sean-k-mooney when an migration ends in error it means an operator might need to fix the db
16:35:21 sean-k-mooney to update the host to bring nova back in sync with where the vm is. or carfully move the vm back ot the souce host
16:35:40 sean-k-mooney if a migration end in failed there shoudl be no action needed by an operator to fix anything
16:55:02 opendevreview Ghanshyam proposed openstack/nova master: Add service role in Nova policy https://review.opendev.org/c/openstack/nova/+/957578
16:56:25 gmaan bauzas: thanks, rebased it ^^ dansmith sean-k-mooney ^^
16:56:33 dansmith ack
16:57:25 bauzas gmaan: I'm a bit out of steam now, I'll look at it tomorrow morning
16:57:35 gmaan no issue.
16:59:46 sean-k-mooney gmaan: i did a test in cidner to see if i coudl make it work with the glance revert
16:59:57 sean-k-mooney but id dint work and i dont know why
17:00:18 gmaan sean-k-mooney: you mean nova change did not work with glance revert?
17:00:28 gmaan or glance revert things only
17:00:28 sean-k-mooney no the cider oen i crfeated
17:00:48 sean-k-mooney i looke at how nova creates our glance client nad tried to do the same in cidner
17:00:50 gmaan ohk, i can check that. cinder to send right roles in nova, I can check for glance
17:01:12 gmaan i checked all those files recently so it will be quick, looking..
17:02:31 sean-k-mooney https://review.opendev.org/c/openstack/cinder/+/958582
17:02:43 gmaan sean-k-mooney: here is issue, cinder does not send the confiugured service user to glance instead send the user token (context) + service token so user token does not have 'service' role - https://github.com/openstack/cinder/blob/89709ef589b74506885c72e42cad57eb8dedfb56/cinder/image/glance.py#L134C45-L134C52
17:03:34 gmaan here cinder should have first try to get the configured service user and that auth plugin (load from keystoneauth) they can send along with service token
17:03:48 opendevreview Balazs Gibizer proposed openstack/nova master: Ask for pre-prod testing for native threading https://review.opendev.org/c/openstack/nova/+/957424
17:03:49 opendevreview Balazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet https://review.opendev.org/c/openstack/nova/+/953436
17:03:49 opendevreview Balazs Gibizer proposed openstack/nova master: Run unit test with threading mode https://review.opendev.org/c/openstack/nova/+/953475
17:03:50 opendevreview Balazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively https://review.opendev.org/c/openstack/nova/+/953815
17:03:50 opendevreview Balazs Gibizer proposed openstack/nova master: [CI]Make nova-tox-py312-threading voting https://review.opendev.org/c/openstack/nova/+/955791
17:03:51 opendevreview Balazs Gibizer proposed openstack/nova master: Do not yield in threading mode https://review.opendev.org/c/openstack/nova/+/950994
17:03:51 opendevreview Balazs Gibizer proposed openstack/nova master: Make RBD Tpool usage conditional https://review.opendev.org/c/openstack/nova/+/956089
17:03:52 opendevreview Balazs Gibizer proposed openstack/nova master: Make libvirt Tpool proxying conditional https://review.opendev.org/c/openstack/nova/+/956090
17:03:52 opendevreview Balazs Gibizer proposed openstack/nova master: Fix ProviderTree copying with threading Lock https://review.opendev.org/c/openstack/nova/+/956091
17:03:54 opendevreview Balazs Gibizer proposed openstack/nova master: [test]Further categorization of disabled unit tests https://review.opendev.org/c/openstack/nova/+/956092
17:03:54 opendevreview Balazs Gibizer proposed openstack/nova master: [vncproxy]Handle ssl.wrap_socket removal in py312 https://review.opendev.org/c/openstack/nova/+/955915
17:03:56 opendevreview Balazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor https://review.opendev.org/c/openstack/nova/+/952666
17:04:14 sean-k-mooney gmaan that what i was tryign to do here https://review.opendev.org/c/openstack/cinder/+/958582/1/cinder/image/glance.py#145
17:04:45 sean-k-mooney let me update that quickly to get rid of the pep8 errors so its readable
17:05:53 gmaan sean-k-mooney: but you need to load service user token from keystoneauth and pass it to https://review.opendev.org/c/openstack/cinder/+/958582/1/cinder/image/glance.py#148
17:05:57 gibi dansmith: fixed the doc pre-prod testing doc patch to ask for a mail.
17:06:06 dansmith gibi: already approved
17:06:07 dansmith thanks
17:06:12 gibi dansmith: thanks!
17:06:47 gibi sean-k-mooney: you threading unit test finding was valid, I missed that in our exclude list in the first patch and only added later. Now it is excluded from the begining
17:06:50 gmaan sean-k-mooney: otherwise auth going to keystone sessions is still user token (with no service role) + service token - https://review.opendev.org/c/openstack/cinder/+/958582/1/cinder/image/glance.py#154
17:07:54 sean-k-mooney gmaan: that what i was tryign to do by having the session object constucted form teh config
17:08:16 sean-k-mooney im obvioulsy missing a step but im tring to use the user token form the config not the actual user
17:09:15 gmaan I think keystonemiddleware do fetch the headers from auth plugin and populate the context with those data but let me check
17:10:13 opendevreview melanie witt proposed openstack/nova stable/2025.1: libvirt: Get info with abs path, rebase with rel path https://review.opendev.org/c/openstack/nova/+/958674
17:10:16 gmaan sean-k-mooney: but yes if you can cleanup those pep8, i can read it more clearly

Earlier   Later