Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-08-27
16:21:10 sean-k-mooney or one of the terminal states
16:21:20 lajoskatona as rubasov added to the last lines as a thought experiment: "I don't see it yet, how this should be fixed. Maybe a migration should never be stuck indefinitely in status=error...."
16:21:27 sean-k-mooney failed models a diffent endsate then error
16:22:12 sean-k-mooney error is something went wrong that we cant recover form gracefully liek the dest host gets restarted
16:22:27 lajoskatona yes of course
16:22:29 sean-k-mooney failed is the end state when it somethig we can properly recover form
16:22:38 sean-k-mooney like we try to migrate and the cpu is incompatible
16:23:13 sean-k-mooney so the migration status being in error for ever is nto a bug to me
16:23:51 sean-k-mooney the orignal issue was caused by "overcommit on dedicated PCPUs and CPUPinningInvalid exception"
16:24:23 sean-k-mooney I.E. there was already invlaid sate in one or boht of the compute nodes
16:24:52 lajoskatona and the update_available_resource periodically fetches these also and tries to apply the migration that ended in error state but the repeated migration for the same server to the same host succeded so it should be find by the method that the migration is really done by a new migration (server uuid - dest host pair)
16:24:54 sean-k-mooney i.e. vms not runningn on the right core or the oepration modfied the nova config to change the core set
16:26:41 lajoskatona so the question is if we can say that ahhh, the same VM to the same host migration operation is finished so we can forget to stress on the previous migration (same vm uuid same host) that failed ?
16:28:13 sean-k-mooney in general no
16:28:17 lajoskatona so it is really a corner case as I see not error due to the things you mentioned (pcpu config or such) but like issue with the compute itself and after the fix the migration can be repeated
16:28:36 sean-k-mooney but we liekly do have a bug in the logic
16:29:00 sean-k-mooney i dont thnk it woudl be correct for the erroed migration to be updated to failed
16:29:06 sean-k-mooney is shoudl stay in error
16:29:27 sean-k-mooney but if the vm is not on this host we shoudl not process the old migration on new iterations
16:29:39 sean-k-mooney of the update_aviable_resouces perodic
16:29:49 sean-k-mooney movign it to fialed might achive that
16:29:54 lajoskatona that's fair to rethink it and make it clear and we can document it for customers in such case in worst case
16:30:07 sean-k-mooney but im not sure we can alwasy assume that is safe to do
16:30:29 lajoskatona sounds great to think at least about it
16:31:10 sean-k-mooney you were testing bfv was it using ceph by the way?
16:31:48 sean-k-mooney if so the already exist erro might eb related to https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.ensure_libvirt_rbd_instance_dir_cleanup
16:31:54 lajoskatona to tell the truth I am not sure but would say no (have to double check)
16:32:53 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1414895
16:32:59 sean-k-mooney is one of the rrefenece bugs
16:33:59 sean-k-mooney we dont generally delete the instnace storage on the dest on start up because we dont want ot loose data
16:34:25 sean-k-mooney i.e. incase the live migration actuly commpelted at teh libvirt level while the agent was being restarted
16:34:53 lajoskatona thanks I check this issue also if can be related
16:34:55 sean-k-mooney when an migration ends in error it means an operator might need to fix the db
16:35:21 sean-k-mooney to update the host to bring nova back in sync with where the vm is. or carfully move the vm back ot the souce host
16:35:40 sean-k-mooney if a migration end in failed there shoudl be no action needed by an operator to fix anything
16:55:02 opendevreview Ghanshyam proposed openstack/nova master: Add service role in Nova policy https://review.opendev.org/c/openstack/nova/+/957578
16:56:25 gmaan bauzas: thanks, rebased it ^^ dansmith sean-k-mooney ^^
16:56:33 dansmith ack
16:57:25 bauzas gmaan: I'm a bit out of steam now, I'll look at it tomorrow morning
16:57:35 gmaan no issue.
16:59:46 sean-k-mooney gmaan: i did a test in cidner to see if i coudl make it work with the glance revert
16:59:57 sean-k-mooney but id dint work and i dont know why
17:00:18 gmaan sean-k-mooney: you mean nova change did not work with glance revert?
17:00:28 sean-k-mooney no the cider oen i crfeated
17:00:28 gmaan or glance revert things only
17:00:48 sean-k-mooney i looke at how nova creates our glance client nad tried to do the same in cidner
17:00:50 gmaan ohk, i can check that. cinder to send right roles in nova, I can check for glance
17:01:12 gmaan i checked all those files recently so it will be quick, looking..
17:02:31 sean-k-mooney https://review.opendev.org/c/openstack/cinder/+/958582
17:02:43 gmaan sean-k-mooney: here is issue, cinder does not send the confiugured service user to glance instead send the user token (context) + service token so user token does not have 'service' role - https://github.com/openstack/cinder/blob/89709ef589b74506885c72e42cad57eb8dedfb56/cinder/image/glance.py#L134C45-L134C52
17:03:34 gmaan here cinder should have first try to get the configured service user and that auth plugin (load from keystoneauth) they can send along with service token
17:03:48 opendevreview Balazs Gibizer proposed openstack/nova master: Ask for pre-prod testing for native threading https://review.opendev.org/c/openstack/nova/+/957424
17:03:49 opendevreview Balazs Gibizer proposed openstack/nova master: Run unit test with threading mode https://review.opendev.org/c/openstack/nova/+/953475
17:03:49 opendevreview Balazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet https://review.opendev.org/c/openstack/nova/+/953436
17:03:50 opendevreview Balazs Gibizer proposed openstack/nova master: [CI]Make nova-tox-py312-threading voting https://review.opendev.org/c/openstack/nova/+/955791
17:03:50 opendevreview Balazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively https://review.opendev.org/c/openstack/nova/+/953815
17:03:51 opendevreview Balazs Gibizer proposed openstack/nova master: Make RBD Tpool usage conditional https://review.opendev.org/c/openstack/nova/+/956089
17:03:51 opendevreview Balazs Gibizer proposed openstack/nova master: Do not yield in threading mode https://review.opendev.org/c/openstack/nova/+/950994
17:03:52 opendevreview Balazs Gibizer proposed openstack/nova master: Fix ProviderTree copying with threading Lock https://review.opendev.org/c/openstack/nova/+/956091
17:03:52 opendevreview Balazs Gibizer proposed openstack/nova master: Make libvirt Tpool proxying conditional https://review.opendev.org/c/openstack/nova/+/956090
17:03:54 opendevreview Balazs Gibizer proposed openstack/nova master: [vncproxy]Handle ssl.wrap_socket removal in py312 https://review.opendev.org/c/openstack/nova/+/955915
17:03:54 opendevreview Balazs Gibizer proposed openstack/nova master: [test]Further categorization of disabled unit tests https://review.opendev.org/c/openstack/nova/+/956092
17:03:56 opendevreview Balazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor https://review.opendev.org/c/openstack/nova/+/952666
17:04:14 sean-k-mooney gmaan that what i was tryign to do here https://review.opendev.org/c/openstack/cinder/+/958582/1/cinder/image/glance.py#145
17:04:45 sean-k-mooney let me update that quickly to get rid of the pep8 errors so its readable
17:05:53 gmaan sean-k-mooney: but you need to load service user token from keystoneauth and pass it to https://review.opendev.org/c/openstack/cinder/+/958582/1/cinder/image/glance.py#148
17:05:57 gibi dansmith: fixed the doc pre-prod testing doc patch to ask for a mail.
17:06:06 dansmith gibi: already approved
17:06:07 dansmith thanks
17:06:12 gibi dansmith: thanks!
17:06:47 gibi sean-k-mooney: you threading unit test finding was valid, I missed that in our exclude list in the first patch and only added later. Now it is excluded from the begining
17:06:50 gmaan sean-k-mooney: otherwise auth going to keystone sessions is still user token (with no service role) + service token - https://review.opendev.org/c/openstack/cinder/+/958582/1/cinder/image/glance.py#154
17:07:54 sean-k-mooney gmaan: that what i was tryign to do by having the session object constucted form teh config
17:08:16 sean-k-mooney im obvioulsy missing a step but im tring to use the user token form the config not the actual user
17:09:15 gmaan I think keystonemiddleware do fetch the headers from auth plugin and populate the context with those data but let me check
17:10:13 opendevreview melanie witt proposed openstack/nova stable/2025.1: libvirt: Get info with abs path, rebase with rel path https://review.opendev.org/c/openstack/nova/+/958674
17:10:16 gmaan sean-k-mooney: but yes if you can cleanup those pep8, i can read it more clearly
17:10:37 sean-k-mooney im doing that as we speak
17:10:58 sean-k-mooney gibi: ack cool i was hoping it was just that
17:11:26 sean-k-mooney gibi: so its still exclude and or passing in the later patches ya?
17:12:16 opendevreview Balazs Gibizer proposed openstack/nova master: Switch nova-conductor to use ThreadPoolExecutor https://review.opendev.org/c/openstack/nova/+/957088
17:12:17 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-conductor in native threading mode https://review.opendev.org/c/openstack/nova/+/958575
17:12:36 gibi sean-k-mooney: yepp it was excluded later, but now it is excluded from the first patch :) thanks for catching it
17:13:01 gibi the conductor series is now move top of master out from the unit test series
17:13:18 sean-k-mooney ack ill loop back over that shortly
17:13:37 gibi thanks. I'm filing the futurist bug now and the end my day
17:13:39 sean-k-mooney i need to actully finish at leat the first patch in the sev series
17:14:16 sean-k-mooney im going to be aroudn for amybe another hour doign review possible a little more
17:14:47 sean-k-mooney gmaan: i didnt have a chance to revew the dnms but did you get clean runs on the service role the last time
17:15:47 gmaan sean-k-mooney: yes, all good. stable job on tempest change was failing in experimental pipeline so I fixed tests to continue using admin on stable and service on master
17:15:50 sean-k-mooney i was mostly happy with your nova change last time i looked so i think we can likely land that today or tomorrow
17:16:11 sean-k-mooney did you see my quetion on the tempest change
17:16:14 gmaan I fixed your doc/release notes comment too
17:16:21 gmaan oh, not yet, checking
17:16:28 sean-k-mooney baout wither the primary role could be service?
17:16:32 sean-k-mooney instead of admin
17:16:45 gmaan sean-k-mooney: oh, that one i replied yes
17:17:16 sean-k-mooney https://review.opendev.org/c/openstack/cinder/+/958582 now passes pep8
17:17:26 gmaan its not admin. actually 'credential' var in tests is little weird and unreadable
17:18:13 gmaan credentials = ['primary', 'admin', ['service_user', 'admin', 'service']] means: it create three users - 1. user with member role (primary), 2. user with admin (admin), 3. user with admin and saervice role (service_user)

Earlier   Later