| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-08-27 | |||
| 16:21:27 | sean-k-mooney | failed models a diffent endsate then error | |
| 16:22:12 | sean-k-mooney | error is something went wrong that we cant recover form gracefully liek the dest host gets restarted | |
| 16:22:27 | lajoskatona | yes of course | |
| 16:22:29 | sean-k-mooney | failed is the end state when it somethig we can properly recover form | |
| 16:22:38 | sean-k-mooney | like we try to migrate and the cpu is incompatible | |
| 16:23:13 | sean-k-mooney | so the migration status being in error for ever is nto a bug to me | |
| 16:23:51 | sean-k-mooney | the orignal issue was caused by "overcommit on dedicated PCPUs and CPUPinningInvalid exception" | |
| 16:24:23 | sean-k-mooney | I.E. there was already invlaid sate in one or boht of the compute nodes | |
| 16:24:52 | lajoskatona | and the update_available_resource periodically fetches these also and tries to apply the migration that ended in error state but the repeated migration for the same server to the same host succeded so it should be find by the method that the migration is really done by a new migration (server uuid - dest host pair) | |
| 16:24:54 | sean-k-mooney | i.e. vms not runningn on the right core or the oepration modfied the nova config to change the core set | |
| 16:26:41 | lajoskatona | so the question is if we can say that ahhh, the same VM to the same host migration operation is finished so we can forget to stress on the previous migration (same vm uuid same host) that failed ? | |
| 16:28:13 | sean-k-mooney | in general no | |
| 16:28:17 | lajoskatona | so it is really a corner case as I see not error due to the things you mentioned (pcpu config or such) but like issue with the compute itself and after the fix the migration can be repeated | |
| 16:28:36 | sean-k-mooney | but we liekly do have a bug in the logic | |
| 16:29:00 | sean-k-mooney | i dont thnk it woudl be correct for the erroed migration to be updated to failed | |
| 16:29:06 | sean-k-mooney | is shoudl stay in error | |
| 16:29:27 | sean-k-mooney | but if the vm is not on this host we shoudl not process the old migration on new iterations | |
| 16:29:39 | sean-k-mooney | of the update_aviable_resouces perodic | |
| 16:29:49 | sean-k-mooney | movign it to fialed might achive that | |
| 16:29:54 | lajoskatona | that's fair to rethink it and make it clear and we can document it for customers in such case in worst case | |
| 16:30:07 | sean-k-mooney | but im not sure we can alwasy assume that is safe to do | |
| 16:30:29 | lajoskatona | sounds great to think at least about it | |
| 16:31:10 | sean-k-mooney | you were testing bfv was it using ceph by the way? | |
| 16:31:48 | sean-k-mooney | if so the already exist erro might eb related to https://docs.openstack.org/nova/latest/configuration/config.html#workarounds.ensure_libvirt_rbd_instance_dir_cleanup | |
| 16:31:54 | lajoskatona | to tell the truth I am not sure but would say no (have to double check) | |
| 16:32:53 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1414895 | |
| 16:32:59 | sean-k-mooney | is one of the rrefenece bugs | |
| 16:33:59 | sean-k-mooney | we dont generally delete the instnace storage on the dest on start up because we dont want ot loose data | |
| 16:34:25 | sean-k-mooney | i.e. incase the live migration actuly commpelted at teh libvirt level while the agent was being restarted | |
| 16:34:53 | lajoskatona | thanks I check this issue also if can be related | |
| 16:34:55 | sean-k-mooney | when an migration ends in error it means an operator might need to fix the db | |
| 16:35:21 | sean-k-mooney | to update the host to bring nova back in sync with where the vm is. or carfully move the vm back ot the souce host | |
| 16:35:40 | sean-k-mooney | if a migration end in failed there shoudl be no action needed by an operator to fix anything | |
| 16:55:02 | opendevreview | Ghanshyam proposed openstack/nova master: Add service role in Nova policy https://review.opendev.org/c/openstack/nova/+/957578 | |
| 16:56:25 | gmaan | bauzas: thanks, rebased it ^^ dansmith sean-k-mooney ^^ | |
| 16:56:33 | dansmith | ack | |
| 16:57:25 | bauzas | gmaan: I'm a bit out of steam now, I'll look at it tomorrow morning | |
| 16:57:35 | gmaan | no issue. | |
| 16:59:46 | sean-k-mooney | gmaan: i did a test in cidner to see if i coudl make it work with the glance revert | |
| 16:59:57 | sean-k-mooney | but id dint work and i dont know why | |
| 17:00:18 | gmaan | sean-k-mooney: you mean nova change did not work with glance revert? | |
| 17:00:28 | gmaan | or glance revert things only | |
| 17:00:28 | sean-k-mooney | no the cider oen i crfeated | |
| 17:00:48 | sean-k-mooney | i looke at how nova creates our glance client nad tried to do the same in cidner | |
| 17:00:50 | gmaan | ohk, i can check that. cinder to send right roles in nova, I can check for glance | |
| 17:01:12 | gmaan | i checked all those files recently so it will be quick, looking.. | |
| 17:02:31 | sean-k-mooney | https://review.opendev.org/c/openstack/cinder/+/958582 | |
| 17:02:43 | gmaan | sean-k-mooney: here is issue, cinder does not send the confiugured service user to glance instead send the user token (context) + service token so user token does not have 'service' role - https://github.com/openstack/cinder/blob/89709ef589b74506885c72e42cad57eb8dedfb56/cinder/image/glance.py#L134C45-L134C52 | |
| 17:03:34 | gmaan | here cinder should have first try to get the configured service user and that auth plugin (load from keystoneauth) they can send along with service token | |
| 17:03:48 | opendevreview | Balazs Gibizer proposed openstack/nova master: Ask for pre-prod testing for native threading https://review.opendev.org/c/openstack/nova/+/957424 | |
| 17:03:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Allow to start unit test without eventlet https://review.opendev.org/c/openstack/nova/+/953436 | |
| 17:03:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run unit test with threading mode https://review.opendev.org/c/openstack/nova/+/953475 | |
| 17:03:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: [test]RPC using threading or eventlet selectively https://review.opendev.org/c/openstack/nova/+/953815 | |
| 17:03:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: [CI]Make nova-tox-py312-threading voting https://review.opendev.org/c/openstack/nova/+/955791 | |
| 17:03:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Do not yield in threading mode https://review.opendev.org/c/openstack/nova/+/950994 | |
| 17:03:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make RBD Tpool usage conditional https://review.opendev.org/c/openstack/nova/+/956089 | |
| 17:03:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Make libvirt Tpool proxying conditional https://review.opendev.org/c/openstack/nova/+/956090 | |
| 17:03:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix ProviderTree copying with threading Lock https://review.opendev.org/c/openstack/nova/+/956091 | |
| 17:03:54 | opendevreview | Balazs Gibizer proposed openstack/nova master: [test]Further categorization of disabled unit tests https://review.opendev.org/c/openstack/nova/+/956092 | |
| 17:03:54 | opendevreview | Balazs Gibizer proposed openstack/nova master: [vncproxy]Handle ssl.wrap_socket removal in py312 https://review.opendev.org/c/openstack/nova/+/955915 | |
| 17:03:56 | opendevreview | Balazs Gibizer proposed openstack/nova master: Warn on long task wait time for executor https://review.opendev.org/c/openstack/nova/+/952666 | |
| 17:04:14 | sean-k-mooney | gmaan that what i was tryign to do here https://review.opendev.org/c/openstack/cinder/+/958582/1/cinder/image/glance.py#145 | |
| 17:04:45 | sean-k-mooney | let me update that quickly to get rid of the pep8 errors so its readable | |
| 17:05:53 | gmaan | sean-k-mooney: but you need to load service user token from keystoneauth and pass it to https://review.opendev.org/c/openstack/cinder/+/958582/1/cinder/image/glance.py#148 | |
| 17:05:57 | gibi | dansmith: fixed the doc pre-prod testing doc patch to ask for a mail. | |
| 17:06:06 | dansmith | gibi: already approved | |
| 17:06:07 | dansmith | thanks | |
| 17:06:12 | gibi | dansmith: thanks! | |
| 17:06:47 | gibi | sean-k-mooney: you threading unit test finding was valid, I missed that in our exclude list in the first patch and only added later. Now it is excluded from the begining | |
| 17:06:50 | gmaan | sean-k-mooney: otherwise auth going to keystone sessions is still user token (with no service role) + service token - https://review.opendev.org/c/openstack/cinder/+/958582/1/cinder/image/glance.py#154 | |
| 17:07:54 | sean-k-mooney | gmaan: that what i was tryign to do by having the session object constucted form teh config | |
| 17:08:16 | sean-k-mooney | im obvioulsy missing a step but im tring to use the user token form the config not the actual user | |
| 17:09:15 | gmaan | I think keystonemiddleware do fetch the headers from auth plugin and populate the context with those data but let me check | |
| 17:10:13 | opendevreview | melanie witt proposed openstack/nova stable/2025.1: libvirt: Get info with abs path, rebase with rel path https://review.opendev.org/c/openstack/nova/+/958674 | |
| 17:10:16 | gmaan | sean-k-mooney: but yes if you can cleanup those pep8, i can read it more clearly | |
| 17:10:37 | sean-k-mooney | im doing that as we speak | |
| 17:10:58 | sean-k-mooney | gibi: ack cool i was hoping it was just that | |
| 17:11:26 | sean-k-mooney | gibi: so its still exclude and or passing in the later patches ya? | |
| 17:12:16 | opendevreview | Balazs Gibizer proposed openstack/nova master: Switch nova-conductor to use ThreadPoolExecutor https://review.opendev.org/c/openstack/nova/+/957088 | |
| 17:12:17 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-conductor in native threading mode https://review.opendev.org/c/openstack/nova/+/958575 | |
| 17:12:36 | gibi | sean-k-mooney: yepp it was excluded later, but now it is excluded from the first patch :) thanks for catching it | |
| 17:13:01 | gibi | the conductor series is now move top of master out from the unit test series | |
| 17:13:18 | sean-k-mooney | ack ill loop back over that shortly | |
| 17:13:37 | gibi | thanks. I'm filing the futurist bug now and the end my day | |
| 17:13:39 | sean-k-mooney | i need to actully finish at leat the first patch in the sev series | |
| 17:14:16 | sean-k-mooney | im going to be aroudn for amybe another hour doign review possible a little more | |
| 17:14:47 | sean-k-mooney | gmaan: i didnt have a chance to revew the dnms but did you get clean runs on the service role the last time | |
| 17:15:47 | gmaan | sean-k-mooney: yes, all good. stable job on tempest change was failing in experimental pipeline so I fixed tests to continue using admin on stable and service on master | |
| 17:15:50 | sean-k-mooney | i was mostly happy with your nova change last time i looked so i think we can likely land that today or tomorrow | |
| 17:16:11 | sean-k-mooney | did you see my quetion on the tempest change | |
| 17:16:14 | gmaan | I fixed your doc/release notes comment too | |
| 17:16:21 | gmaan | oh, not yet, checking | |
| 17:16:28 | sean-k-mooney | baout wither the primary role could be service? | |
| 17:16:32 | sean-k-mooney | instead of admin | |
| 17:16:45 | gmaan | sean-k-mooney: oh, that one i replied yes | |
| 17:17:16 | sean-k-mooney | https://review.opendev.org/c/openstack/cinder/+/958582 now passes pep8 | |
| 17:17:26 | gmaan | its not admin. actually 'credential' var in tests is little weird and unreadable | |
| 17:18:13 | gmaan | credentials = ['primary', 'admin', ['service_user', 'admin', 'service']] means: it create three users - 1. user with member role (primary), 2. user with admin (admin), 3. user with admin and saervice role (service_user) | |
| 17:18:24 | gibi | here is the futurist bug https://bugs.launchpad.net/futurist/+bug/2121545 | |
| 17:18:36 | sean-k-mooney | this was a case of "i dont understand how to read that list" so i was just asking why primary was admin for what its worth | |