| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-07 | |||
| 20:22:10 | mikal | But that's the key point. OpenStack tries hard not to break existing behaviour. cloud-init apparently does not. | |
| 20:22:39 | mikal | I think this is the strongest argument I've seen as to why people should move to glean, especially because glean implies that config drive will not in fact one day be removed. | |
| 20:23:14 | sean-k-mooney | there are other light wight alternitives too. the problem with glean is partly its streght | |
| 20:23:22 | sean-k-mooney | it does not supprot metadtaa only config drive | |
| 20:23:30 | sean-k-mooney | which si why glean is so reliable | |
| 20:24:15 | sean-k-mooney | but since we cant update config dirve it only really good for first boot but honestly if your doing day two thinbgs use ansible | |
| 20:25:04 | fungi | i don't think it's entirely fair to say that cloud-init doesn't try to preserve existing behaviors, just in this particular case they judged the existing behavior to be a greater risk than the impact from breaking it (whether they judged that accurately is another matter) | |
| 20:25:51 | sean-k-mooney | i guess we will see if we get a spike in reports of "my ssh key was not injected" or similar | |
| 20:26:05 | fungi | they're typically pretty good about not breaking functionality, and in this particular case the bug was open in private for over a year while they debated the risks of doing so | |
| 20:27:15 | sean-k-mooney | on a related not the thing i whish we had time to do was add supprot for jwt or applciation creditials via config dirve. i.e. provide a way simialr to k8s for you to provide and authentication token to the workloads | |
| 20:27:37 | fungi | but yes, the complexity and expansive scope of cloud-init, as well as their choice to rely on lots of different non-stdlib python libs, is why we made and use glean for our ci nodes | |
| 20:28:02 | sean-k-mooney | its something that aws and all the other cloud already do but providing a way to do athestation or the server and auth is a gap we have today. | |
| 20:28:30 | fungi | zuul has jwt support and can do oidc claims too | |
| 20:29:08 | sean-k-mooney | fungi: we are only using glean for the nodepool images right? cirros has its own mini implemation in bash that it uses instead | |
| 20:29:11 | fungi | though i'm not sure if we could leverage that to authenticate metadata server responses | |
| 20:29:24 | fungi | sean-k-mooney: correct | |
| 20:29:44 | clarkb | fungi: the zuul implementation requires the third party set up zuul as a trusted oidc provider | |
| 20:29:46 | fungi | we don't (currently) build cirros images ourselves afaik | |
| 20:30:32 | fungi | clarkb: yeah, i was trying to think if there was a way client-side to leverage that, but the booting server instance would be the client in that case and needs to be reachable before zuul can interact with it | |
| 20:30:35 | sean-k-mooney | this is there code https://github.com/cirros-dev/cirros/blob/main/src/usr/bin/ec2metadata and not we dont we use the upstream images and back them into the ci images to avoid downloadeing it in the job | |
| 20:31:38 | fungi | yeah, we download the upstream cirros images and ship convenience copies in the cache in our test nodes | |
| 20:31:54 | clarkb | note that is a cache though. we can and do remove old versions or lag on adding new versions | |
| 20:32:12 | sean-k-mooney | clarkb: well new version very rarely happen | |
| 20:32:36 | sean-k-mooney | the last cirrors release was 26th of september 2024 | |
| 20:32:48 | fungi | though i expect we're overdue for pruning some of the current list of cirros images we include | |
| 20:33:15 | sean-k-mooney | i think there were still some jobs using 0.5.x last time i checked | |
| 20:33:24 | sean-k-mooney | they shoudl all be on 0.6.3 at this point but ... | |
| 20:33:28 | clarkb | sean-k-mooney: I'm calling it out because someone noticed a cirros image wasn't baked in recently and while it was a bug it was also a buggy job not falling back to download it | |
| 20:33:40 | sean-k-mooney | i belive there is a 0.7.0 in the works btu that has been true for a while now | |
| 20:33:48 | clarkb | and yes they were using 0.5.x and I indicated that those old versions should probably be considered removable too | |
| 20:34:24 | fungi | if the jobs are run infrequently, e.g. on aging unmaintained branches, we usually consider it safe to drop them from the cache since those jobs can still fetch the files at runtime when they occasionally happen to run | |
| 20:34:42 | sean-k-mooney | so the kernel panics we hit in 0.6.3 rarely are much more common on 0.5.x becase there is at least 1 addional kernle bug that is not fixed there. | |
| 20:55:26 | mikal | sean-k-mooney: a less terrible spiffe arrestor would probably meet your needs while pleasing the hipsters. It's on my mental list of things to one day look at, possibly with some TPM sprinkled in. | |
| 20:56:16 | mikal | fungi: they had it open for a year and never thought to ask a Nova core? | |
| 20:57:30 | sean-k-mooney | i mean it was a private "secuirty issue" so i would not nessisarly expiect them to | |
| 20:57:49 | sean-k-mooney | alhtough if they were unsure they coudl have. i guess fungi was at least somewhat aware | |
| 20:58:08 | sean-k-mooney | so i asusme they asked the vmt? | |
| 20:58:55 | fungi | jamespage convinced them on adding me to the bug as a heads up, though technically not for a vmt perspective since the vulnerability doesn't exist in openstack clouds, it was more of a question about the potential impact of the breaking change they were considering | |
| 20:58:57 | mikal | I don't see that on the bug? I might have missed it because it only skimmed, but they seem to have spent much more like talking to Canonical than the affected platform team. | |
| 20:59:32 | mikal | (The VMT being asked that is) | |
| 21:01:22 | fungi | and since it wasn't a report for an openstack project, i didn't feel like i had the authority to pull more people into it | |
| 21:02:41 | fungi | i was absolutely out of my depth on whether cloud-init's reliance on reading dmi data was a good choice, but also that seemed to already be happening in prior versions | |
| 21:05:09 | sean-k-mooney | fungi: ya i dont expect you to knwo those details | |
| 21:05:27 | sean-k-mooney | it just sad that the chice is to break eveyr nova virt driver that is not libvirt | |
| 21:06:24 | sean-k-mooney | well not break but the dmi interface was only supproted in the libvirt driver so the auto detection wont work genericlly | |
| 21:06:58 | mikal | sean-k-mooney: one issue with machine credentials is that OpenStack lacks an IAM role system, so what credentials would you give the machine and with what permissions? Oslo policy is such a mess Red that won't take support calls if you use it, and there really is nothing else. | |
| 21:07:40 | mikal | sean-k-mooney: in other clouds that machine credential would be of the form "can read from this one specific object store bucket and this one special user". We can't do that right now. | |
| 21:08:15 | sean-k-mooney | mikal: i wanted to supprot passign an applciation credetial that you precreate or askign for nova to create one that on the users behalf | |
| 21:08:44 | sean-k-mooney | and we dont supprot custom policy because oslo.plicy is a mess | |
| 21:08:46 | sean-k-mooney | its not | |
| 21:09:01 | sean-k-mooney | we dont supprot it by default because custoemr kept breaking thing by incorrectly configuring it | |
| 21:09:38 | sean-k-mooney | so we say if you want to have custom policy file a supprot exception to at least get a cursory ok form someone who shoudl under stand it | |
| 21:10:46 | fungi | it just seems like a catch-22 since you'd need some way to pass data into a server booted from a generic image so it knows what to validate the signature with (public key for example) | |
| 21:11:24 | fungi | and if you already had a secure communication channel, you could just use that for the metadata too | |
| 21:12:10 | mikal | Well, qemu supports secure communications channels, we just don't use them. | |
| 21:12:59 | sean-k-mooney | well metadta was ment to be the secure channel but ya... | |
| 21:13:39 | sean-k-mooney | mikal: we wanted it to be mostly one way, and we intentally dont allow direct db access form the compute agent | |
| 21:13:55 | sean-k-mooney | fungi: the problem is that it does nto mattter how secure we make this in neutron | |
| 21:14:20 | sean-k-mooney | its ever othe rcloud that woudl need ot be secure for them to feel ok with this being enabled by default | |
| 21:16:06 | sean-k-mooney | like on the nuetorn side we already trap the reqest based on that ip and then use the soruce mac/ip to proxy it to nova with the neutron proxy authentiication using a shared secret | |
| 21:16:30 | sean-k-mooney | neutron also does not allow other vms to respond to the requeest eith its anti spoffing rules | |
| 21:16:56 | sean-k-mooney | but unless azure and every other cloud also did that they would consider any well knwo address to be insecure | |
| 21:17:20 | sean-k-mooney | i can kind of get where they are commign form it just how cloud images are expected to work | |
| 21:18:16 | sean-k-mooney | if they dont supprot metadta by defualt it not really a cloud iamge anymore since tha was kind fo the defining factor at least in the context of cloud-init | |
| 21:18:26 | sean-k-mooney | anyways i better call it a day o/ | |
| 21:19:42 | clarkb | mikal: you actually can configure a bucket (swift container) that only a specific one off account can edit | |
| 21:20:07 | clarkb | its a bit under documented and the tooling is rough. infra uses it and we end up doing direct curl stuff to make it happen iirc | |
| 21:33:57 | fungi | sean-k-mooney: by "secure communication channel" i meant one which the booted operating system can somehow verify is trustworthy. a trusted third party (like a well-known public certificate authority) is one possible solution, but probably a terrible one | |
| 21:37:17 | fungi | basically include a reference in the metadata blob to the location of a signature and a certificate corresponding to the key that signed the metadata, where that certificate is issued by a trusted third party that cloud-init or whatever knows and can validate it against | |
| 21:39:48 | fungi | as soon as you eliminate the ttp, i think you're stuck having to inject some custom trust information into the server, and the trustworthy injection problem becomes essentially the same as the the trustworthy metadata problem | |
| 21:40:16 | fungi | but it's likely there are common solutions i'm just not aware of | |
| 21:41:45 | sean-k-mooney | fungi: your thinking of something liek the grub efi shim | |
| 21:42:36 | fungi | yes, that's the same sort of implementation | |
| 21:43:19 | fungi | and faces effectively the same challenges | |
| 21:43:44 | sean-k-mooney | so with something like config drive youy can sort of do that in the sense tha tyou could provdie a ca cert to use to authenticate but at that point you might as well just provide the metadta as well | |
| 21:43:55 | fungi | my point exactly | |
| 21:44:39 | sean-k-mooney | we also try to aovid injecting things directly into the image now if we can avoid it | |
| 21:45:07 | fungi | bootstrapping trust is a nontrivial problem | |
| 21:45:41 | sean-k-mooney | yep and the trusted compute folks would go as far as saying proving nvram or other uefi stores is not secure enough | |
| 22:02:35 | mikal | clarkb: other clouds let you do it effectively for any API call. | |
| 22:03:02 | clarkb | ya I know. Its just that is the one example where you can do it with openstack | |
| 22:04:58 | mikal | fungi: this "credential zero" problem is literally what SPIFFE is aimed at. Its just the OpenStack attestor is meh. One day someone should fix it. | |
| 22:05:25 | mikal | Like the hypervisor could produce a certificate for the instance at boot and ram it into the TPM, to my current understanding of TPMs. | |
| 22:09:35 | fungi | yeah, and then the tpm is your implicitly trusted communication channel, albeit one that probably isn't suitable for ramming all the metadata through | |
| 22:09:58 | fungi | but can at least bootstrap a more general-purpose source at that point | |
| 22:13:39 | sean-k-mooney | mikal: so injecting things into tpm would generally not eb considered secure | |
| 22:14:10 | sean-k-mooney | i know you could add a root of turst to teh tpm | |
| 22:14:38 | sean-k-mooney | but if we supprot that soemoen woudl ask to be able to lock that out so that only there root of trust form barbiacn could be injected | |
| 22:16:05 | sean-k-mooney | that woudl not be so terrible i gues.. allow them ot updload somethign to barbican and pass a refence ot it when booting and have nova inject it into the tpm usign the users token ot retirve it | |
| 22:16:08 | mikal | sean-k-mooney: noting that I've never managed to find a good guide to TPM stuff and am therefore a noob, why would the hypervisor generating a unique certificate that identifies the instance and storing it in the TPM so inside the instance can sign things but not actually get to the certificate not be secure? Isn't that almost exactly what the | |
| 22:16:08 | mikal | firmware is doing, just with a CA for the cloud instead of a CA for jus the TPM? | |
| 22:17:12 | mikal | sean-k-mooney: understanding TPMs is also on my todo list of things I'll never get around to. | |
| 22:18:03 | sean-k-mooney | main thing to under stand to day is we cant live migrate with tpms to day because we encypt them with a secreate stored in barbican that only the user that create the vm can acess :) so tpms equal pain | |
| 22:19:05 | sean-k-mooney | we coudl likely generate a cert to "sign" or attest the identiy of the vm, and maybe put that in the config drive or tpm for you to atteset | |
| 22:19:27 | sean-k-mooney | but im not sure we could use that for auth | |
| 22:20:14 | sean-k-mooney | that was one of the jwt propsoals that was made not too long ago. inject a jwt token purly for attestation of the identiy of the sever but not for actual keystone auth usage | |
| 22:21:53 | sean-k-mooney | mikal: https://github.com/bbc/nova/commit/382984a3a23032c96089cb5877a55e425db7cee4 | |
| 22:25:13 | mikal | sean-k-mooney: so that injected JWT has a name in SPIFFE that I've forgotten, but its a definitely a thing they do. Then again, how do you handle the JWT expiring for long lived VMs? | |
| 22:27:33 | sean-k-mooney | mikal: no idea the bbc impleaiton was just an experiment and it never landed upstream | |
| 22:28:41 | mikal | https://kasm.stillhq.com/#/session/9606cb74-10e3-4e1e-8990-0779495f79dd -- SPIFFE calls it a SVID apparently. | |