| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-03-06 | |||
| 19:38:55 | sean-k-mooney | it looks like the barbican user in the service ocnfigs has the creator roles https://github.com/openstack-archive/tripleo-heat-templates/blob/stable/wallaby/deployment/barbican/barbican-api-container-puppet.yaml#L259-L274 | |
| 19:39:34 | sean-k-mooney | anyway with that in mind do we still want ot have a check for this in the api | |
| 19:39:53 | sean-k-mooney | given we shoudl be using the new defaults in 2024.1 across all services | |
| 19:41:42 | sean-k-mooney | looking at https://github.com/openstack/governance/blob/48d090643e7faf138b8f4e19f4a2915a20e1a873/goals/selected/consistent-and-secure-rbac.rst#20231-release-timeline | |
| 19:42:11 | sean-k-mooney | barbica had supprot for enforce_new_defaults=True in antelope | |
| 19:42:26 | sean-k-mooney | and in 2024.1 oslo is ment to remvoe enforce_scope | |
| 19:42:29 | sean-k-mooney | https://github.com/openstack/governance/blob/48d090643e7faf138b8f4e19f4a2915a20e1a873/goals/selected/consistent-and-secure-rbac.rst#20241-release-timeline | |
| 19:43:23 | sean-k-mooney | so as of this release and certenly for 2024.1 we shoudl be able to assume barbican has the behavior of enforce_scope=true | |
| 19:46:01 | sean-k-mooney | *2024.2 | |
| 19:48:16 | melwitt | sean-k-mooney: yeah.. I guess, I'm thinking of the upgrade scenario of a deployment that's been around for awhile and has enforce_scope = False set explicitly in policy? not sure how common that may be | |
| 19:48:40 | sean-k-mooney | melwitt: it looks like oslo has not deprecate the option or change the default yet | |
| 19:48:42 | sean-k-mooney | https://github.com/openstack/oslo.policy/blob/master/oslo_policy/opts.py#L27-L36 | |
| 19:48:44 | JayF | I suspect it's reasonably common | |
| 19:48:57 | melwitt | same.. | |
| 19:48:58 | JayF | Ironic was configured that way (enforce_scope=False) by default until this cycle | |
| 19:49:38 | sean-k-mooney | JayF: right so it looks like we did not make progress on changing the default and removing the option this cycle | |
| 19:49:48 | melwitt | sean-k-mooney: yeah oslo_policy hasn't yet but barbican defaulting it https://github.com/openstack/barbican/blob/master/barbican/common/config.py#L357-L361 | |
| 19:49:51 | sean-k-mooney | i remember this commign up in the last ptg | |
| 19:50:51 | sean-k-mooney | melwitt: so we can do soemthing like this | |
| 19:51:09 | sean-k-mooney | we can either use our new default value to infer barbicans | |
| 19:51:28 | sean-k-mooney | and check for creator based on that or just catch the excption late | |
| 19:51:48 | sean-k-mooney | once htis cant be turned off in barbican | |
| 19:51:52 | sean-k-mooney | we can drop the check in nova | |
| 19:51:59 | sean-k-mooney | and just rely on member | |
| 19:52:23 | melwitt | I was gonna say that we could check [oslo_policy]enforce_scope and if it's False do the check to barbican API but that would be assuming barbican is deployed on the same host as nova-api. which may not be a safe assumption | |
| 19:52:24 | sean-k-mooney | what that will actully mean is by default we wont need to check | |
| 19:53:13 | sean-k-mooney | it might pass the api check when it shoudl not but we will catch the api error in the compute manager when we try to create a key | |
| 19:53:25 | melwitt | er, or it would be assuming that barbican is configured the same. which it may not | |
| 19:53:47 | sean-k-mooney | so its better then nothign for the small subset of pepole who diverge form the defaults | |
| 19:54:08 | sean-k-mooney | right alghough you are not really ment to mix and match enfoce_scope | |
| 19:54:48 | sean-k-mooney | i mean i guess it depends on how expensive the check is | |
| 19:55:01 | sean-k-mooney | i dont know if oslo midelware gives us a list of all the roles on the token already | |
| 19:55:08 | sean-k-mooney | if it does and its just a map lookup | |
| 19:55:14 | sean-k-mooney | then it cheap and we can alwasy do it | |
| 19:55:39 | sean-k-mooney | if its an api call to keystone on every spawn with this feature that when we might want to optimise | |
| 19:57:04 | melwitt | I was thinking the same thing. I had been thinking it would have to be an api call but maybe we could do a lookup in the RequestContext or something | |
| 19:59:03 | sean-k-mooney | ya im hoping that oslo.midelway or keystone auth is just giving use a list of roles when it doing the token validation in keystone and we can jsut check in that | |
| 20:42:03 | gmann | sean-k-mooney: melwitt: we are not at the stage to remove/change defaults of enforce_scope in oslo.policy as many other project might be impacted but this config default can be overriden per service | |
| 20:42:22 | gmann | for nova, it is enabled by default, this is where we override the default https://github.com/openstack/nova/blob/13ccaf75f61a19ba11f39b63052bcef5dfbce91f/nova/policy.py#L51 | |
| 20:43:52 | sean-k-mooney | gmann: yep i know we have it enabled | |
| 20:43:52 | gmann | We enabled them by default since Antelope release | |
| 20:44:07 | sean-k-mooney | right but all project were ment to be enbaled by default last cycle | |
| 20:44:17 | sean-k-mooney | i remember that we disucssed this last ptg | |
| 20:44:22 | sean-k-mooney | and that they had not got tha tfar | |
| 20:44:32 | sean-k-mooney | so i tought we were trying to do that for 2024.1 | |
| 20:46:00 | gmann | it is still not there, I am picking one by one project. Tacker is my goal this cycle and will pick a few of them in next but progress from other projects is slow | |
| 20:47:16 | sean-k-mooney | it might be better at this point to flip the default in olso and make project explcity opt out | |
| 20:48:07 | gmann | yeah, I am also thinking that way. We can discuss it in PTG if no objection It think this is right way to proceed | |
| 20:48:52 | sean-k-mooney | ack | |
| 20:48:54 | gmann | sean-k-mooney: just in case you have not seen, this is updated version of testing runtime adding 3.12 in this doc as non-mandatory to test https://review.opendev.org/c/openstack/governance/+/908862 | |
| 20:49:21 | sean-k-mooney | we coudl turn it on by default and deprecate it in oslo, and since its a non slurp we need to keep the option until at least 2025.2 | |
| 20:49:52 | sean-k-mooney | gmann: thanks | |
| 20:51:07 | gmann | I will say not to rush on remove as project adding scope in their policy will have it enabled by default. but other point is that we do not have system scope and it will be project scoped only so it does not break the things just improve the error msg if system scoped are used. | |
| 20:51:08 | sean-k-mooney | gmann: https://review.opendev.org/c/openstack/governance/+/908862/6/reference/runtimes/2024.2.rst#61 minior nit but over all this looks fine to me | |
| 20:51:26 | sean-k-mooney | gmann: well we cant basedon the slurp process | |
| 20:51:35 | sean-k-mooney | gmann: oslo have not deprecated the options yet | |
| 20:51:42 | gmann | yes | |
| 20:52:05 | sean-k-mooney | so we cant remove until the deprecateion goes out in 2025.1 so 2025.2 woudl be the absolute earliest it coudl be removed now | |
| 20:52:30 | gmann | yeah, we need it deprecated at least for 2025.1 | |
| 20:53:07 | gmann | I will preapre the rbac PTG sessions etherpad to add this just in case we forget. I schedule session on Monday 13 UTC though | |
| 22:08:27 | opendevreview | melanie witt proposed openstack/nova-specs master: Re-propose specs for ephemeral encryption https://review.opendev.org/c/openstack/nova-specs/+/907654 | |
| #openstack-nova - 2024-03-07 | |||
| 05:06:55 | opendevreview | Rajesh Tailor proposed openstack/python-novaclient master: Bump microversion to 2.96 https://review.opendev.org/c/openstack/python-novaclient/+/911575 | |
| 12:00:26 | opendevreview | Rajesh Tailor proposed openstack/python-novaclient master: Bump microversion to 2.96 https://review.opendev.org/c/openstack/python-novaclient/+/911575 | |
| 18:04:59 | dansmith | melwitt: so I created an encrypted instance, cirros image, booted fine | |
| 18:05:15 | dansmith | I snapshotted it, the snap has the new credential uuid | |
| 18:05:55 | dansmith | I then booted a new instance from that snap, saw that it ran qemu-img convert to (I think) unencrypt it before it uploaded to ceph to then create a vm image from, | |
| 18:06:06 | dansmith | but the disk is unreadable in the new instance.. like fails in BIOS | |
| 18:06:38 | dansmith | since this is ceph it's hard to examine the actual disk to see what the state is.. I should try to repro with qcow because I can poke at it easier | |
| 18:07:03 | dansmith | but (a) that's expected to work, right? | |
| 18:07:46 | dansmith | and (b) I'm thinking if a user does this, they don't realize that you just stripped the encryption from their snapshot image to upload it | |
| 18:08:16 | dansmith | or maybe I'm wrong about what was happening - perhaps we were just re-encrypting it with the new key? if had used qcow it'd be easier for me to see what happened | |
| 18:08:23 | dansmith | maybe it's in the logs actually | |
| 18:08:53 | melwitt | dansmith: hm, well it shouldn't be unencrypting it, convert is used to make a copy with a new key. but as we discussed I have to change all that and haven't yet | |
| 18:09:40 | dansmith | okay, I thought that maybe the "no support yet for encrypted backing files" meant we were de-keying it because it's the image we're based on | |
| 18:10:07 | melwitt | and yeah, it was expected to work, to be able to boot a new instance from the image. there might be something that got messed up when I refactored the things needed from review feedback on the bottom patches | |
| 18:10:37 | melwitt | well, the rbd patch is after the encrypted backing file support one | |
| 18:11:13 | dansmith | okay so, there's a command in the log which is what I was seeing when I watched it, where we convert _to_ luks _from_ raw | |
| 18:11:20 | dansmith | so I assumed that meant we had raw'd it | |
| 18:12:20 | melwitt | and that's the root disk, not swap or ephemeral? | |
| 18:14:02 | dansmith | we wouldn't run convert on ephemeral or swap because we're creating from scratch would we/ | |
| 18:14:51 | melwitt | oh, right. in earlier revisions I had it creating as raw blank and then mkfs and then encrypted it. but yeah I think I changed it to create luks off the bat | |
| 18:16:16 | melwitt | so it sounds like there must be a bug there that happened in the intervening time. I need to look into it | |
| 18:17:59 | melwitt | converting from raw to luks should only happen if the source image is unencrypted. and it would be interesting to see if it's erroneously getting converted to raw after pulling it down from ceph. I wonder if maybe i messed something up in the fetch_to_raw function. it's supposed to be a no-op if the source image is luks | |
| 18:18:30 | melwitt | that was something that got changed during addressing review feedback | |
| 18:21:24 | melwitt | I'll investigate | |
| 18:22:21 | dansmith | there's a raw disk in _base, but I think that might be the original cirros I created from | |
| 18:22:40 | dansmith | but yeah, I probably need to take ceph out of the equation here so I can grok what's going on | |
| 18:22:55 | dansmith | I'm trying to look at the log and I don't see anything with "-O raw" which is good | |
| 18:24:31 | dansmith | yeah I'm not sure maybe I misread a command line as a convert when it was something else | |
| 18:26:18 | melwitt | it would be 'qemu-img convert' in that case | |
| 18:27:22 | dansmith | yeah | |
| 18:27:40 | dansmith | we are doing a lot of qemu-img info execs with no input format, which is a problem, | |
| 18:27:49 | dansmith | not sure if that's old or new code though | |
| 18:28:30 | dansmith | okay let me redo this with no ceph so I can try to figure out they boot-of-a-snap is dead | |
| 18:29:16 | dansmith | or see if it's a ceph thing I guess | |
| 18:29:18 | melwitt | should be old, for qemu-img info | |
| 18:30:12 | dansmith | ack, I thought we had fixed those | |
| 18:31:06 | melwitt | I'm double checking. maybe I'm forgetting something | |
| 18:33:10 | melwitt | info itself didn't change but passing the format is optional, it's possible I added a call and didn't pass the format. I don't remember off the top of my head but just saying there's a possibility. I'll go back and fix that if I find it | |
| 18:36:25 | dansmith | oh I know it's optional, it's just a security bug to not provide the format | |