| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-01-27 | |||
| 07:39:19 | gokhan | pre_live_migration fails: I’ve observed that many live migration failures (like the one where a volume is in 'backing-up' status) could potentially be caught much earlier. Currently, if a volume is busy or in an unstable state, the failure often occurs during pre_live_migration, triggering the complex rollback logic we've been refining. Would it make sense to implement a proactive volume status check at the Nova API level (or early in th | |
| 07:39:19 | gokhan | hello folks, I want to get your thoughts about this patch https://review.opendev.org/c/openstack/nova/+/973750 . It seems https://github.com/openstack/nova/commit/5a55a78d510b86975f0f4f8f43ee1feef7206244#diff-47eb12598e353b9e0689707d7b477353200d0aa3ed13045ffd3d017ee7d9e753 commit resolves live migration pre migrate failure. Regarding the fixes in Bug #1899835 (Avoid volume rollback mismatches), which handles cleanup on destination when | |
| 07:39:20 | gokhan | e Conductor) before the RPC call to the destination node? | |
| 07:47:47 | gibi | gokhan: I agree that if there is a way to catch "bad" volume states early in the complex live migration process then that can decrease the likelyhood of a problematic rollback. | |
| 07:56:18 | gokhan | thanks gibi We have submitted a patch (https://review.opendev.org/c/openstack/nova/+/973750) for catching in api side. We would like to confirm if this is the recommended approach. | |
| 08:02:32 | gokhan | Since melwitt has previously worked on similar issues—specifically the 'Avoid volume rollback mismatches' patches—I would highly value her insights and comments on this matter to ensure we're aligned with the existing logic. | |
| 08:03:50 | gibi | gokhan: yeah, I'm reading you patch now... | |
| 08:04:47 | gokhan | thanks gibi :) | |
| 08:10:58 | gibi | gokhan: I think the direction is OK. I left comments about the test coverage, and pinged melwitt in the review | |
| 08:15:56 | gokhan | Thanks for the review gibi, we will address your comments and upload a new patch set soon. | |
| 08:17:14 | opendevreview | Seyeong Kim proposed openstack/nova master: libvirt: Support boot_index for multiple block devices https://review.opendev.org/c/openstack/nova/+/963665 | |
| 11:04:54 | tobias-urdin | can somebody refresh my memory, when doing live migration for an instance with cpu_policy=dedicated and cpu_thread_policy=require and cpu_dedicated_set set, does scheduler take into account remapping PCPU for example on source cores 0,40 to something like 1,41 if 0,40 was not free on destination compute? | |
| 11:05:38 | tobias-urdin | gibi: ^ iirc it should work? (caracal) | |
| 11:12:16 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron floating IPs https://review.opendev.org/c/openstack/nova/+/962604 | |
| 11:12:16 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 11:24:37 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron networks https://review.opendev.org/c/openstack/nova/+/928022 | |
| 11:24:59 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron subnets https://review.opendev.org/c/openstack/nova/+/962190 | |
| 11:25:09 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 11:30:28 | opendevreview | Esra Ozkan proposed openstack/nova master: Fix Concurrent VM Live Migrate - Volume Backup Error https://review.opendev.org/c/openstack/nova/+/973750 | |
| 11:32:26 | opendevreview | Max proposed openstack/nova master: fix: delete attachments after rescheduling delete https://review.opendev.org/c/openstack/nova/+/974832 | |
| 11:32:26 | opendevreview | Max proposed openstack/nova master: Add regression test for bug #2137366 https://review.opendev.org/c/openstack/nova/+/974831 | |
| 11:36:19 | opendevreview | Esra Ozkan proposed openstack/nova master: Fix Concurrent VM Live Migrate - Volume Backup Error https://review.opendev.org/c/openstack/nova/+/973750 | |
| 11:36:23 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron floating IPs https://review.opendev.org/c/openstack/nova/+/962604 | |
| 12:04:37 | gibi | tobias-urdin: yeah I think it should work but obviously we need more data. | |
| 12:09:23 | gibi | tobias-urdin: I think this is the funtional test that at least partially covering it https://github.com/openstack/nova/blob/d840c63a18ad951382be821f90a6b28c627911fa/nova/tests/functional/libvirt/test_numa_live_migration.py#L197 | |
| 12:09:59 | sean-k-mooney | tobias-urdin: it does in any release the properly supprot numa live migration which i think was added in train or wallaby | |
| 12:10:30 | sean-k-mooney | ya it was train | |
| 12:10:49 | sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/numa-aware-live-migration.html | |
| 12:33:17 | tobias-urdin | thanks, not sure why i had a vague memory of it having to be equal on both sides, probably pre-train days something then :) | |
| 12:33:57 | sean-k-mooney | yep pre-train the live migraiton woudl be allowed but the vms xml would not be updated until it was hard rebboted | |
| 12:34:14 | sean-k-mooney | asumign the cores existed if they didnt the live migraiton woudl fail but only in that case | |
| 12:34:34 | sean-k-mooney | we put a block in place and backported pre train with a workaround to reenable it as part of supprotign it properly | |
| 12:35:00 | sean-k-mooney | basiclly pre train we said only do live migration with numa if your usign hugepage without cpu pinning | |
| 12:42:30 | opendevreview | Stephen Finucane proposed openstack/nova master: Follow-up for I85a7729ee65f8987ed0239f80d8d0082a414ab8f https://review.opendev.org/c/openstack/nova/+/974843 | |
| 13:53:26 | opendevreview | Merged openstack/nova master: api: Simplify servers views (1/3) https://review.opendev.org/c/openstack/nova/+/956231 | |
| 14:00:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445 | |
| 14:00:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM:Test with oslo.vmware + compute eventlet removal patches https://review.opendev.org/c/openstack/nova/+/973468 | |
| 14:00:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467 | |
| 14:00:53 | opendevreview | Balazs Gibizer proposed openstack/nova master: SubclassSignatureTestCase to use NoDBTestCase as base https://review.opendev.org/c/openstack/nova/+/974861 | |
| 14:11:25 | gibi | sean-k-mooney: bauzas: gmaan: https://review.opendev.org/c/openstack/nova/+/974445 this is ready for serious review now. I have couple of remaining todos but nothing is expected to change in the core of the patch. In the other hand the core of the patch needs deep review | |
| 14:12:47 | bauzas | ack, I definitely need a coffee for reviewing it | |
| 14:13:28 | gibi | don't worry if it take two :) | |
| 14:13:34 | opendevreview | Merged openstack/nova master: api: Simplify servers views (2/3) https://review.opendev.org/c/openstack/nova/+/956232 | |
| 14:17:54 | opendevreview | Merged openstack/nova master: api: Simplify servers views (3/3) https://review.opendev.org/c/openstack/nova/+/956233 | |
| 14:23:37 | sean-k-mooney | gibi: ok ill try an take a look today or tomorrow | |
| 14:56:47 | gibi | thanks both of you :) | |
| 15:13:54 | MaxLamprecht[m] | Hey folks, I'm currently debugging many out of sync cases between nova and cinder BDMs/attachments in our environment. Mostly these are caused by parallel operations/lock-contention or other reasons. I added a comment... (full message at <https://matrix.org/oftc/media/v1/media/download/AYgeZZCT43vzk9WSn7ySGLYl0AqCdodcgtuHH4PEsRyizpNZ9dfQRaY5IGWjRMbtweIwNYxNRndku0zt8aS2xE9CecSB29CAAG1hdHJpeC5vcmcvb0drQk95eXZDTkVTUmphRGJPSk5GYlRZ>) | |
| 15:17:42 | sean-k-mooney | MaxLamprecht[m] multi line messages on matrix only rednere the first line on irc with a link to the full message so its best to avoid multi line messages | |
| 15:19:17 | sean-k-mooney | MaxLamprecht[m]: it would be good to add those to https://etherpad.opendev.org/p/nova-2026.1-status#L76 | |
| 15:19:39 | sean-k-mooney | add a new line per bug with 2 sub bullet point for the regression and fix reviews | |
| 15:20:47 | sean-k-mooney | MaxLamprecht[m]: ohter then that the best thing you can do to get review input is to bring it up here or in the open discuss section of the weekly irc meeting | |
| 15:20:55 | MaxLamprecht[m] | ack. Good to know :) | |
| 15:20:58 | MaxLamprecht[m] | thx, I will add these MRs | |
| 15:21:09 | sean-k-mooney | the fact you have created the repoducer and fix sepreatly ectra will definly help with the reivew | |
| 15:22:39 | gmaan | gibi: ack | |
| 15:27:36 | sean-k-mooney | MaxLamprecht[m]: 2 of the serise refence the same bug so you have 3 bug fix/regresion serise bvut only 2 bugs | |
| 15:28:34 | sean-k-mooney | on actully no you have 3 | |
| 15:28:37 | sean-k-mooney | just similar numbers | |
| 15:29:23 | MaxLamprecht[m] | haha. I was already checking if I messed something up :D | |
| 15:29:33 | sean-k-mooney | MaxLamprecht[m]: can you make sure you assign https://bugs.launchpad.net/nova/+bug/2088066 https://bugs.launchpad.net/nova/+bug/2137366 and https://bugs.launchpad.net/nova/+bug/2139135 to your self in lanchpad | |
| 15:30:06 | sean-k-mooney | MaxLamprecht[m]:it was the 66 that was confusing me | |
| 15:30:14 | MaxLamprecht[m] | sean-k-mooney: done | |
| 15:31:16 | sean-k-mooney | all 3 sound liek valid bugs so i quickly tiragers all 3 as medium and added the approate tags | |
| 15:41:58 | MaxLamprecht[m] | perfect. I'm currently trying to resolve/find root causes for such bdm/attachment inconcistencies in our environments. Maybe I will find additional bugs while digging into this. | |
| 15:42:12 | MaxLamprecht[m] | I have added another small db index MR to the etherpad that is mostly relevant for large scale deployments | |
| 15:43:34 | sean-k-mooney | MaxLamprecht[m]: its still proably a good idea to summerise this in a mailing list post or in the next irc meetign so that folks are aware of the related work your are trying to progress | |
| 15:43:56 | sean-k-mooney | its not striclty requried but the more visiablity your patches have the more likely it will get reviewd | |
| 15:48:50 | MaxLamprecht[m] | sean-k-mooney: ack. I will try to join the next irc meeting | |
| 15:50:56 | rm_work[m] | this and any name will work? | |
| 15:50:56 | rm_work[m] | hopefully quick q about nova vendordata dynamicJSON... I can't find it in the docs anywhere but it seems like I need to configure the `vendordata_dynamic_targets` option with the target named `cloud-init@<address>` or else cloud-init doesn't actually return anything for it. is that expected? and how can I use multiple targets if they need to be named that way? or did I maybe get confused from some other bug I had when I was testing | |
| 15:58:58 | sean-k-mooney | rm_work[m]: so yes and no | |
| 15:59:13 | sean-k-mooney | rm_work[m]: we have no upstream supoprt for usign it as a data souce for cloud-init | |
| 15:59:29 | sean-k-mooney | they have some documentation that suggest you could use it that way | |
| 15:59:37 | sean-k-mooney | but ti has never been officaly supproted in nova | |
| 16:00:08 | sean-k-mooney | rm_work[m]: https://github.com/canonical/cloud-init/issues/5221 | |
| 16:01:47 | sean-k-mooney | rm_work[m]: in https://github.com/canonical/cloud-init/issues/5221#issuecomment-2086560281 i provide a example of how to hack it to work by doing | |
| 16:01:51 | sean-k-mooney | [API] | |
| 16:01:53 | sean-k-mooney | vendordata_dynamic_targets=['cloud-init:string@http://127.0.0.1:123'] | |
| 16:04:35 | sean-k-mooney | there was a spec that was not implemtned this https://review.opendev.org/c/openstack/nova-specs/+/917109/6/specs/2024.2/approved/dynamicjson-vendordata-cloud-config.rst https://blueprints.launchpad.net/nova/+spec/dynamicjson-vendordata-cloud-config | |
| 16:05:27 | sean-k-mooney | rm_work[m]: so yes if your dynmaic backend return a single string that is the cloud init info i think it can work but since it not offially supproted if ti does not its not a nova bug | |
| 16:06:00 | sean-k-mooney | its a new feature as this was never intended to work with cloud init automaticly like that | |
| 16:36:38 | rm_work[m] | I mean I don't need it to RUN as part of cloud-init, I just need cloud-init to properly fetch the contents into the vendordata2.json file | |
| 16:37:00 | rm_work[m] | like, VM boots, user checks the json file and it contains stuff | |
| 16:37:20 | rm_work[m] | using a colon is a neat trick though, ok will try that | |
| 16:38:07 | sean-k-mooney | so it will be present in the the config drive and via the metadata api | |
| 16:38:33 | sean-k-mooney | you just wont be abel to execute scrips in the guest vms via vender data automaticly without the hack | |
| 16:38:48 | sean-k-mooney | so if you just want to make the file avaiable you do not need the hack | |
| 16:39:23 | sean-k-mooney | if you want to install packages ectra then you do but as i noted its not offically supproted so just be aware of that | |
| 16:59:23 | rm_work[m] | yeah it looks like I CAN use any name after all, I think I had an encoding bug originally and fixed that at the same time as trying switching to cloud-init as the key, and assumed I needed that name | |
| 17:01:05 | rm_work[m] | nova doesn't like if the target returns anything other than a string with 200 code | |
| 17:01:23 | rm_work[m] | I was trying to return JSON originally... seeing as how it's called DynamicJSON... but nope lol | |
| 17:01:43 | rm_work[m] | THAT seems like a bug to me but i assume there's a reason | |
| 17:10:27 | sean-k-mooney | nova is expectign a json respocne but a single stirng is also a vaild json responce | |
| 17:10:42 | sean-k-mooney | rm_work[m]: we are doing json.loads on the responce | |
| 17:11:54 | sean-k-mooney | rm_work[m]: https://www.madebymikal.com/nova-vendordata-deployment-an-excessively-detailed-guide/ | |
| 17:12:09 | rm_work[m] | hmm | |
| 17:12:26 | rm_work[m] | yeah that was the guide I was originally following | |
| 17:13:26 | sean-k-mooney | yep so the responce format should be the same as the static json file | |
| 17:13:29 | rm_work[m] | ok I'll try again again again | |
| 17:13:56 | sean-k-mooney | the hack of using a single string with the file contents for cloud init config is abusign the fact that a string is a valid json document | |