| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-08-05 | |||
| 16:18:38 | jssfr | so we're currently looking into two topics which are somewhat related: | |
| 16:18:41 | jssfr | first we're rebasing https://review.opendev.org/c/openstack/nova/+/621646 | |
| 16:18:53 | jssfr | second I proposed a draft patch for a vTPM data preservation issue a couple weeks ago: https://review.opendev.org/c/openstack/nova/+/955657 | |
| 16:19:22 | sean-k-mooney | o/ sorry was doign something else | |
| 16:19:26 | jssfr | o/ | |
| 16:19:38 | jssfr | I also think that sean-k-mooney proposed something similar to the vTPM draft for UEFI NVRAM preservation | |
| 16:20:00 | jssfr | we'd like to hear whether we're heading the right direction by rebasing #621646 regarding UEFI, or whether there might be a better way. | |
| 16:20:17 | sean-k-mooney | i was suggeing bring it up in this meetign yes | |
| 16:20:20 | melwitt | oh, good. this was the topic I wanted heh (vTPM data) | |
| 16:20:24 | sean-k-mooney | or rather that melwitt bring it up | |
| 16:20:35 | jssfr | I also saw only now that you replied to #955657, because I had my inbox sorting messed up, reading that now. | |
| 16:21:00 | sean-k-mooney | melwitt: do you want to summerize | |
| 16:21:08 | jssfr | read it | |
| 16:21:12 | melwitt | sure | |
| 16:21:33 | melwitt | we are looking for input and consensus about handling the recently raised issue in the community regarding vTPM data that is lost across guest reboots https://bugs.launchpad.net/nova/+bug/2118888 and https://review.opendev.org/c/openstack/nova/+/955657 the tl;dr is that this is a latent problem that originates in libvirt | |
| 16:22:08 | melwitt | libvirt added a flag for this VIR_DOMAIN_UNDEFINE_KEEP_TPM a couple of years after vTPM was added in Nova and jssfr has proposed a patch for using the flag ^ | |
| 16:22:20 | jssfr | (which will need a bit of work, such as adding tests) | |
| 16:22:30 | jssfr | (but I can do that) | |
| 16:22:53 | sean-k-mooney | there is a related but seperate issue that related to the nvram data | |
| 16:23:08 | sean-k-mooney | again there are now flag to supprot keeping or removing the nvram file | |
| 16:23:32 | jssfr | (UEFI NVRAM bug is https://bugs.launchpad.net/nova/+bug/1785123, patch-with-merge-conflicts is https://review.opendev.org/c/openstack/nova/+/621646 ) | |
| 16:23:37 | dansmith | so if we do that, we need to do our own deleting of the TPM file when we undefine to actually delete the instance? | |
| 16:23:51 | jssfr | dansmith, no, because we can just not pass that flag when we truly want to delete | |
| 16:23:57 | sean-k-mooney | dansmith: no. we can pass the keep or delte flag as approtiate | |
| 16:23:58 | jssfr | in my draft patch, I tied it to the delete_storage bool IIRC | |
| 16:24:01 | dansmith | oh it's a flag _to_ undefine? | |
| 16:24:04 | jssfr | yes :) | |
| 16:24:06 | melwitt | yes | |
| 16:24:07 | dansmith | okay | |
| 16:24:26 | dansmith | ah I see yes, okay | |
| 16:24:57 | sean-k-mooney | so we have a number of placess we undefine the vm today wher the data shoudl not be deleted, hard-reboot is one but also some volume operations require use to temporlly undefien the domain | |
| 16:25:47 | melwitt | we are thinking about, how to fix this for older versions. the flag addition could be technically seen as a "feature" but IMHO the impact is quite significant and personally I would think about backporting the flag fix to older versions that could be reasonably expected to use libvirt >= 8.9.0 where the flag was introduced | |
| 16:26:13 | melwitt | wondering what others think of this | |
| 16:26:37 | dansmith | shifting behaviors in a library that we have to account for is not a feature, IMHO | |
| 16:26:37 | jssfr | I think a backport would be quite appropriate | |
| 16:26:44 | jssfr | and we are going to backport to at least 2024.1 anyway | |
| 16:27:15 | jssfr | so we could contribute those backported patches at least. | |
| 16:27:27 | sean-k-mooney | dansmith: the wrinkel is libvirt uncondtionally delete the data when we added vtpm supprot and the option to preseve it was added 2 years later | |
| 16:27:44 | dansmith | yeah understand | |
| 16:28:04 | sean-k-mooney | with that said the didnt really document that which is partly why we missed it but i agre we shoudl not require a feature to "do the write thign" | |
| 16:28:05 | jssfr | sean-k-mooney, so you're saying it's not shifting behaviour in libvirt, but simply a bug which existed since forever but which nobody noticed? | |
| 16:28:30 | sean-k-mooney | jssfr: yes, however that does not mean i belive we shoudl consider it a feature | |
| 16:28:51 | sean-k-mooney | i supprot the idea of fixing this condtional on the libvirt verison | |
| 16:28:54 | dansmith | it's shifting behavior and our failure to account for it is a bug not a feature | |
| 16:28:55 | jssfr | (I get the impression that "feature" versus "bug" is an important distinction here, but I'm not sure why. backporting/support policies?) | |
| 16:29:05 | sean-k-mooney | and possibel backporting that based on dicusssion with teh stable team | |
| 16:29:20 | melwitt | I agree with dansmith | |
| 16:29:41 | sean-k-mooney | dansmith: well the default never changed in libvirt. it still deleted on undefine by defualt. they just added the ablity to keep it in later release | |
| 16:29:56 | sean-k-mooney | so do we agree to treat this a logic bug in nova | |
| 16:30:03 | sean-k-mooney | i.e. that we didnt account for how this actully works | |
| 16:30:24 | sean-k-mooney | and if so are we ok with usign the flags to adress that | |
| 16:30:58 | sean-k-mooney | jssfr: yes the stable plociy does not allow feature backports upstream | |
| 16:31:05 | jssfr | aha, thanks! | |
| 16:31:11 | sean-k-mooney | jssfr: there are very limited excptions to that | |
| 16:31:52 | jssfr | noted, thanks. | |
| 16:32:25 | melwitt | I support backporting use of the flags (with libvirt version conditional) | |
| 16:32:40 | sean-k-mooney | im +1 on treating it as a but and fixing it by approatbly passing the flags to undefine based on if we are deleting the instance or just undefining for some other reason | |
| 16:33:31 | NicolaiRuckel | this applies to both vTPM and UEFI, right? | |
| 16:33:33 | sean-k-mooney | s/as a but/as a bug/ | |
| 16:33:49 | sean-k-mooney | so its the same patther and the same logic bug | |
| 16:34:07 | sean-k-mooney | so i think we could apply the same direction to both | |
| 16:34:34 | melwitt | +1 | |
| 16:34:39 | jssfr | sean-k-mooney, does this only apply for the accidental deletion on undefine for UEFI, or also for the loss of NVRAM data during migrations? | |
| 16:34:39 | sean-k-mooney | the specific libvirt verions may differ for the relevent flags | |
| 16:34:46 | sean-k-mooney | so we shoudl fix both as seperate patches | |
| 16:35:17 | sean-k-mooney | jssfr: nvram data is actully copied during live migration i belive | |
| 16:35:38 | sean-k-mooney | cold migration is sperate. we shoudl focus on the non move operations cases first | |
| 16:36:00 | jssfr | NicolaiRuckel, any evidence of nvram preservation during live migrations when initially rebased https://review.opendev.org/c/openstack/nova/+/621646 ? | |
| 16:36:15 | sean-k-mooney | im not sure which subset of opertions loose nvram data but fixing all fo them is not going to be a singel bug | |
| 16:36:27 | jssfr | understood, thanks. | |
| 16:36:31 | sean-k-mooney | jssfr: i belive its copied by qemu | |
| 16:36:51 | jssfr | aha | |
| 16:37:13 | jssfr | NicolaiRuckel, we'll have to double-check that in the devstack, without the rebased patches. | |
| 16:37:23 | NicolaiRuckel | yeah, I didn't try it yet | |
| 16:37:41 | NicolaiRuckel | There were so many failing test cases that I wanted to clean that up first. | |
| 16:38:02 | sean-k-mooney | i would sugget fixign one bug first then the other | |
| 16:38:18 | sean-k-mooney | i dont really have a prefence in order but proably vtpm might be less work | |
| 16:38:49 | sean-k-mooney | for nvram there is existing tech debt that need to be unwornd related to rebuild | |
| 16:38:53 | jssfr | sean-k-mooney, okay, I'll get to work on making the vTPM patch ready for merge this week then. | |
| 16:39:22 | jssfr | probably not going to *finish* it this week though. What is the time we need to submit the polished patch (incl. tests) for it to be realistically merged for Flamingo? | |
| 16:39:48 | sean-k-mooney | you have about 3 weeks | |
| 16:39:54 | jssfr | okay, so 1w before FF | |
| 16:39:55 | jssfr | noted | |
| 16:40:03 | sean-k-mooney | well you have until FF | |
| 16:40:14 | jssfr | oh I thought that was in 4 weeks. | |
| 16:40:26 | Uggla | A bit less than 4 weeks | |
| 16:40:37 | jssfr | right | |
| 16:40:38 | sean-k-mooney | after which it woudl need a backport but its not a regressin intoduced in this cycle so between FF and RC1 we dont fix a lot of latent bugs | |
| 16:40:54 | sean-k-mooney | after rc 1 is cut the branch woudl be reopen for bugfixes again | |
| 16:41:00 | Uggla | FF will be August 28th | |
| 16:41:15 | jssfr | it should be doable to finish it in two weeks, so I'll aim for a submission before or on 2025-08-19 | |
| 16:41:18 | sean-k-mooney | yep that why i said about 3 weeks | |
| 16:41:38 | NicolaiRuckel | I'll continue with rebasing/fixing the UEFI patch in the meantime. | |
| 16:44:45 | sean-k-mooney | i think we can move on if there are other topic unless there are any final objections? | |
| 16:45:24 | jssfr | I'd probably jump out of the meeting then. I'll stay reachable here as usual though if there's any later remarks. Thanks for all the input! | |
| 16:46:15 | Uggla | ok I guess melwitt point was covered ^ | |
| 16:46:33 | Uggla | anything else to discuss ? | |
| 16:47:40 | jssfr | okay, text you all later, thanks! | |
| 16:52:59 | Uggla | As we are almost on the top of the hour, I'll skip bug scrubbing. | |
| 16:53:44 | Uggla | Thanks all, see you next week. | |