| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-07 | |||
| 21:06:58 | mikal | sean-k-mooney: one issue with machine credentials is that OpenStack lacks an IAM role system, so what credentials would you give the machine and with what permissions? Oslo policy is such a mess Red that won't take support calls if you use it, and there really is nothing else. | |
| 21:07:40 | mikal | sean-k-mooney: in other clouds that machine credential would be of the form "can read from this one specific object store bucket and this one special user". We can't do that right now. | |
| 21:08:15 | sean-k-mooney | mikal: i wanted to supprot passign an applciation credetial that you precreate or askign for nova to create one that on the users behalf | |
| 21:08:44 | sean-k-mooney | and we dont supprot custom policy because oslo.plicy is a mess | |
| 21:08:46 | sean-k-mooney | its not | |
| 21:09:01 | sean-k-mooney | we dont supprot it by default because custoemr kept breaking thing by incorrectly configuring it | |
| 21:09:38 | sean-k-mooney | so we say if you want to have custom policy file a supprot exception to at least get a cursory ok form someone who shoudl under stand it | |
| 21:10:46 | fungi | it just seems like a catch-22 since you'd need some way to pass data into a server booted from a generic image so it knows what to validate the signature with (public key for example) | |
| 21:11:24 | fungi | and if you already had a secure communication channel, you could just use that for the metadata too | |
| 21:12:10 | mikal | Well, qemu supports secure communications channels, we just don't use them. | |
| 21:12:59 | sean-k-mooney | well metadta was ment to be the secure channel but ya... | |
| 21:13:39 | sean-k-mooney | mikal: we wanted it to be mostly one way, and we intentally dont allow direct db access form the compute agent | |
| 21:13:55 | sean-k-mooney | fungi: the problem is that it does nto mattter how secure we make this in neutron | |
| 21:14:20 | sean-k-mooney | its ever othe rcloud that woudl need ot be secure for them to feel ok with this being enabled by default | |
| 21:16:06 | sean-k-mooney | like on the nuetorn side we already trap the reqest based on that ip and then use the soruce mac/ip to proxy it to nova with the neutron proxy authentiication using a shared secret | |
| 21:16:30 | sean-k-mooney | neutron also does not allow other vms to respond to the requeest eith its anti spoffing rules | |
| 21:16:56 | sean-k-mooney | but unless azure and every other cloud also did that they would consider any well knwo address to be insecure | |
| 21:17:20 | sean-k-mooney | i can kind of get where they are commign form it just how cloud images are expected to work | |
| 21:18:16 | sean-k-mooney | if they dont supprot metadta by defualt it not really a cloud iamge anymore since tha was kind fo the defining factor at least in the context of cloud-init | |
| 21:18:26 | sean-k-mooney | anyways i better call it a day o/ | |
| 21:19:42 | clarkb | mikal: you actually can configure a bucket (swift container) that only a specific one off account can edit | |
| 21:20:07 | clarkb | its a bit under documented and the tooling is rough. infra uses it and we end up doing direct curl stuff to make it happen iirc | |
| 21:33:57 | fungi | sean-k-mooney: by "secure communication channel" i meant one which the booted operating system can somehow verify is trustworthy. a trusted third party (like a well-known public certificate authority) is one possible solution, but probably a terrible one | |
| 21:37:17 | fungi | basically include a reference in the metadata blob to the location of a signature and a certificate corresponding to the key that signed the metadata, where that certificate is issued by a trusted third party that cloud-init or whatever knows and can validate it against | |
| 21:39:48 | fungi | as soon as you eliminate the ttp, i think you're stuck having to inject some custom trust information into the server, and the trustworthy injection problem becomes essentially the same as the the trustworthy metadata problem | |
| 21:40:16 | fungi | but it's likely there are common solutions i'm just not aware of | |
| 21:41:45 | sean-k-mooney | fungi: your thinking of something liek the grub efi shim | |
| 21:42:36 | fungi | yes, that's the same sort of implementation | |
| 21:43:19 | fungi | and faces effectively the same challenges | |
| 21:43:44 | sean-k-mooney | so with something like config drive youy can sort of do that in the sense tha tyou could provdie a ca cert to use to authenticate but at that point you might as well just provide the metadta as well | |
| 21:43:55 | fungi | my point exactly | |
| 21:44:39 | sean-k-mooney | we also try to aovid injecting things directly into the image now if we can avoid it | |
| 21:45:07 | fungi | bootstrapping trust is a nontrivial problem | |
| 21:45:41 | sean-k-mooney | yep and the trusted compute folks would go as far as saying proving nvram or other uefi stores is not secure enough | |
| 22:02:35 | mikal | clarkb: other clouds let you do it effectively for any API call. | |
| 22:03:02 | clarkb | ya I know. Its just that is the one example where you can do it with openstack | |
| 22:04:58 | mikal | fungi: this "credential zero" problem is literally what SPIFFE is aimed at. Its just the OpenStack attestor is meh. One day someone should fix it. | |
| 22:05:25 | mikal | Like the hypervisor could produce a certificate for the instance at boot and ram it into the TPM, to my current understanding of TPMs. | |
| 22:09:35 | fungi | yeah, and then the tpm is your implicitly trusted communication channel, albeit one that probably isn't suitable for ramming all the metadata through | |
| 22:09:58 | fungi | but can at least bootstrap a more general-purpose source at that point | |
| 22:13:39 | sean-k-mooney | mikal: so injecting things into tpm would generally not eb considered secure | |
| 22:14:10 | sean-k-mooney | i know you could add a root of turst to teh tpm | |
| 22:14:38 | sean-k-mooney | but if we supprot that soemoen woudl ask to be able to lock that out so that only there root of trust form barbiacn could be injected | |
| 22:16:05 | sean-k-mooney | that woudl not be so terrible i gues.. allow them ot updload somethign to barbican and pass a refence ot it when booting and have nova inject it into the tpm usign the users token ot retirve it | |
| 22:16:08 | mikal | firmware is doing, just with a CA for the cloud instead of a CA for jus the TPM? | |
| 22:16:08 | mikal | sean-k-mooney: noting that I've never managed to find a good guide to TPM stuff and am therefore a noob, why would the hypervisor generating a unique certificate that identifies the instance and storing it in the TPM so inside the instance can sign things but not actually get to the certificate not be secure? Isn't that almost exactly what the | |
| 22:17:12 | mikal | sean-k-mooney: understanding TPMs is also on my todo list of things I'll never get around to. | |
| 22:18:03 | sean-k-mooney | main thing to under stand to day is we cant live migrate with tpms to day because we encypt them with a secreate stored in barbican that only the user that create the vm can acess :) so tpms equal pain | |
| 22:19:05 | sean-k-mooney | we coudl likely generate a cert to "sign" or attest the identiy of the vm, and maybe put that in the config drive or tpm for you to atteset | |
| 22:19:27 | sean-k-mooney | but im not sure we could use that for auth | |
| 22:20:14 | sean-k-mooney | that was one of the jwt propsoals that was made not too long ago. inject a jwt token purly for attestation of the identiy of the sever but not for actual keystone auth usage | |
| 22:21:53 | sean-k-mooney | mikal: https://github.com/bbc/nova/commit/382984a3a23032c96089cb5877a55e425db7cee4 | |
| 22:25:13 | mikal | sean-k-mooney: so that injected JWT has a name in SPIFFE that I've forgotten, but its a definitely a thing they do. Then again, how do you handle the JWT expiring for long lived VMs? | |
| 22:27:33 | sean-k-mooney | mikal: no idea the bbc impleaiton was just an experiment and it never landed upstream | |
| 22:28:41 | mikal | https://kasm.stillhq.com/#/session/9606cb74-10e3-4e1e-8990-0779495f79dd -- SPIFFE calls it a SVID apparently. | |
| 22:30:37 | mikal | Ugh, wrong url. | |
| 22:30:53 | mikal | https://github.com/spiffe/spiffe/blob/main/standards/JWT-SVID.md this one. | |
| #openstack-nova - 2025-07-08 | |||
| 08:37:37 | mhen | Hi! #openstack-placement seems to be empty. Is this the right channel for Placement? | |
| 09:19:07 | gibi | mhen: yes it is the right one | |
| 09:20:37 | mhen | I've got a question regarding Placement: | |
| 09:20:45 | mhen | is my interpretation correct that Placement offers no means of adjusting its WSGI pipeline? It seems hardcoded here: https://github.com/openstack/placement/blob/ce677d96f2a5e18dcc5e10a2046cac9780a67cde/placement/deploy.py#L108-L117 | |
| 09:20:47 | opendevreview | Kamil Sambor proposed openstack/nova master: Switch from Eventlet timeout https://review.opendev.org/c/openstack/nova/+/954324 | |
| 09:21:00 | mhen | For other services, there is usually an api-paste.ini or something similar. | |
| 09:21:12 | mhen | So the audit middleware (https://docs.openstack.org/keystonemiddleware/latest/audit.html) cannot be used with Placement? | |
| 09:23:37 | gibi | mhen: good question let me double check | |
| 09:24:34 | gibi | mhen: it seems you are right we have a hard coded pipeline with no api-paste configuratbility | |
| 09:27:50 | mhen | alright, thanks for confirming | |
| 12:51:48 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2115905 https://review.opendev.org/c/openstack/nova/+/954336 | |
| 12:51:48 | opendevreview | Balazs Gibizer proposed openstack/nova master: [pci]Keep used dev in Placement regardless of dev_spec https://review.opendev.org/c/openstack/nova/+/954149 | |
| 13:25:51 | sean-k-mooney | gibi: i reviewed the first couple of patches in your eventlet work and im back up to where i left off before (the stats priting one) for the most part they look fine to me | |
| 13:27:16 | sean-k-mooney | i also looked at kamils patch, https://review.opendev.org/c/openstack/nova/+/949754/17 i think ThreadingEventWithResult should just be a future not a event with a result, i woudl be interested to know if you agree but i left that as a comment in gerrit so no rush | |
| 13:56:05 | zigo | I tried using an older version of pymysql (the one in Bookworm), it didn't help. I tried doing a gc.disable() to check if that was another instance of the GC issue in Python 3.13, and it didn't help either. Anyone has an idea of what's going on? | |
| 13:56:05 | zigo | Hi there! I filled-up a new bug: https://bugs.launchpad.net/nova/+bug/2116186 | |
| 13:59:07 | gibi | sean-k-mooney: on the ThreadingEventWithResult: I wanted a solution that changes as minimal in the current flow of the code as possible. Hence the direct replication of the Eventlet.Event behavior with threading.Event + a variable approach. The Future based approach has the drawback in my mind that the concurrent Future interface explicitly says that setting the result of the Future should happen by | |
| 13:59:13 | gibi | the implementation of an Executor. So it seems using a Future in that nova code would create a lot more code churn if it is even possible. | |
| 13:59:50 | sean-k-mooney | am so future are usabel with raw thread | |
| 13:59:58 | sean-k-mooney | so an executor is not stritly required | |
| 14:00:14 | gibi | https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Future.set_result | |
| 14:00:33 | gibi | This method should only be used by Executor implementations and unit tests. | |
| 14:01:11 | sean-k-mooney | hum | |
| 14:01:22 | sean-k-mooney | that does not conform to what i would normally consider a future then | |
| 14:02:17 | sean-k-mooney | we would have to use a queue isnterad which is less obvious | |
| 14:03:43 | gibi | we started with a queue actually :) If you look back of previous PSs it is there | |
| 14:04:17 | sean-k-mooney | to me a future is concpetiulaly the same as a a queue of lenth 1 with blocking or non blocking gets dependign on if you pass a timeout to result | |
| 14:04:18 | gibi | then we learned that eventlet.Event has the behavior that it sends the same piece of result to all waiters so a queue with a lenght of 1 is not optimal | |
| 14:04:34 | sean-k-mooney | ah i see | |
| 14:04:40 | gibi | as it requires requeueing the same result :/ | |
| 14:05:02 | sean-k-mooney | well that fan out or multi subsciber behaivor seams non standard too | |
| 14:05:07 | gibi | yep it is not | |
| 14:05:13 | sean-k-mooney | are we actuly reallying on that | |
| 14:05:27 | sean-k-mooney | i think for now im ok with the current approch then | |
| 14:05:34 | gibi | I'm not sure but I'm not willing to risk | |
| 14:05:40 | sean-k-mooney | but we may want to see if we can simplicy in the future | |
| 14:05:50 | gibi | yepp simplifying in the future is the way | |
| 14:06:05 | gibi | first have something that works and then optimize | |
| 14:10:37 | sean-k-mooney | so the asyncio future does nto have that limiation for what its worth | |
| 14:10:39 | sean-k-mooney | https://docs.python.org/3/library/asyncio-future.html#asyncio.Future.set_result | |
| 14:12:47 | sean-k-mooney | i ws not expectign thos to diverge in that regard | |
| 14:16:55 | gibi | yeah it is strange | |
| 14:22:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2115905 https://review.opendev.org/c/openstack/nova/+/954336 | |