| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-03-12 | |||
| 08:11:02 | bauzas | sure, I'll review those patches | |
| 08:11:06 | bauzas | (I'm used to :) ) | |
| 09:48:16 | bauzas | sean-k-mooney: fwiw, I started to peek at numa aware vswitches numa affinity problems and I created functests here for reproducing : https://review.opendev.org/c/openstack/nova/+/979999/ | |
| 09:48:59 | bauzas | at the moment, I'm not seeing any issue by the fixtures, as the scheduler provides limits to the compute claim | |
| 09:49:43 | bauzas | that's only when the filter doesn't pass the limits or if the reqspec is modified that we no longer get the limits for the claim and then the compute doesn't look at the numa nodes affinities | |
| 10:10:48 | opendevreview | Dominik proposed openstack/nova master: NUMA Topology with Resource Providers: Libvirt NUMA Migrate https://review.opendev.org/c/openstack/nova/+/971177 | |
| 11:51:30 | sean-k-mooney | bauzas: the probelm is not in the schduler | |
| 11:51:39 | sean-k-mooney | bauzas: so in the schduler it works its on the compute node | |
| 11:52:13 | sean-k-mooney | bauzas: when we reproduced it in a devstack enve we saw the schduelr do the right thing but when the compute agent called the same code it did not | |
| 11:53:03 | sean-k-mooney | bauzas: by the way the limits form the schdluer are not provided to the compute agenet | |
| 11:53:45 | sean-k-mooney | we reconsudct the instance numa toplogy blob in the compute and recompute the pining ectra we dont pass it over rpc today | |
| 11:54:14 | sean-k-mooney | bauzas: my guess its this si really dumb and we are just not passing a parmater somewhere | |
| 11:56:12 | sean-k-mooney | bauzas: as an asside if the numa shculder filter is not used numa is not suprpoted at all but but ti ssond liek you may have fount part of the issue network_metadata is missgin? | |
| 11:58:05 | sean-k-mooney | the compute agent shoudl enfoce numa regardelss fothe schduler config obvioulsy but the point im making is we have alwasy said useign numa feature without the schduler filter is not supprot upstream the compute agenet protections are just a saftey net so that you can evenutally fix your scdhluer config without needing ot move your vms | |
| 12:08:32 | opendevreview | Merged openstack/nova master: Add service version for Gazpacho https://review.opendev.org/c/openstack/nova/+/979016 | |
| 12:27:57 | opendevreview | Merged openstack/python-novaclient master: Add version foot protector https://review.opendev.org/c/openstack/python-novaclient/+/978254 | |
| 13:21:44 | bauzas | sean-k-mooney: I'd appreciate if you could look at my func reproducer patch https://review.opendev.org/c/openstack/nova/+/979999 | |
| 13:22:02 | bauzas | I agree with you on the fact the compute claim should do the same than the filter and doublecheck the network metadata | |
| 13:22:38 | bauzas | but I'm still wondering why those limits are not passed, because functests say "yes this is passed unless reschedules without alternativesé" | |
| 13:33:27 | sean-k-mooney | bauzas: the are not passed because we can race with tow diffent request | |
| 13:33:57 | sean-k-mooney | i.e. we cant safely use what the schduler has calulated without moving the instnace calim to the schduler | |
| 13:41:52 | sean-k-mooney | bauzas: the instnace claim buidl the numa constraits form teh falvor/image locally on teh comptue agent | |
| 13:42:16 | sean-k-mooney | my guess based on what you discsoced is we are not incuding the network metadtaa in that | |
| 14:09:26 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2026.1 Gazpacho https://review.opendev.org/c/openstack/nova/+/961896 | |
| 14:15:33 | opendevreview | Merged openstack/nova stable/2025.2: [nova-tox-py312-threading]Ignore failing tests https://review.opendev.org/c/openstack/nova/+/980013 | |
| 14:18:56 | nicolairuckel | Is the CI having a bad day today? There are random jobs failing for my changes. | |
| 14:22:04 | opendevreview | ribaudr proposed openstack/nova master: Add Gazpacho prelude section https://review.opendev.org/c/openstack/nova/+/979023 | |
| 14:22:15 | nicolairuckel | I had to rerun them because one of the gates failed (which was correct at that point) but one job was failing that ran before. So I rerun them again and now that one passed but another one fails. | |
| 14:22:26 | nicolairuckel | https://review.opendev.org/c/openstack/nova/+/978908 | |
| 14:24:12 | dansmith | sean-k-mooney: IIRC, the only way the whole admin/instance password thing works is either by injection in libvirt (disabled/discouraged) or via metadata.. cloud-init seems to not even support this, but cloudbase-init (for windows) does.. that line up with your understanding? | |
| 14:24:27 | dansmith | asking you because you're usually one of the more vocal anti-injection voices :) | |
| 14:24:43 | sean-k-mooney | dansmith: liek alwasy there is a third option | |
| 14:24:55 | sean-k-mooney | it can be set via the qemu guest agent without injection | |
| 14:25:15 | sean-k-mooney | but yes cloudbase init is the only thing that generates it on the compute startup and post teh value back via metadta | |
| 14:25:18 | dansmith | that's the set-password thing, I know that.. that's sort of different than what I'm asking | |
| 14:25:57 | sean-k-mooney | ya so your summery basicly matches my understanding | |
| 14:25:58 | dansmith | sean-k-mooney: libvirt will generate/store it in sysmeta I think, cloudbase-init can rewrite it IIRC | |
| 14:29:38 | dansmith | hmm, maybe it only does that during set.. so maybe I'm thinking of older rax/xen behavior.. you'd create an instance in horizon and immediately get back a "here's your admin password" dialog | |
| 14:29:39 | sean-k-mooney | dansmith: by the way i didnt think we saved the admin-password | |
| 14:30:01 | sean-k-mooney | i tought we only returned it on create/rebuild (encyprted via your keypair) | |
| 14:30:07 | sean-k-mooney | but that was only in memory | |
| 14:30:28 | sean-k-mooney | perhasp i imagined that | |
| 14:30:48 | dansmith | we store it in sysmeta in password_N keys | |
| 14:30:49 | sean-k-mooney | the reason i ask is i dont think its in server show | |
| 14:31:24 | dansmith | I thought it was in the create response but I'm not seeing where that would be now | |
| 14:31:28 | dansmith | it's not in server show | |
| 14:31:41 | sean-k-mooney | it is in the create respocne and in the rebuild responce | |
| 14:32:30 | sean-k-mooney | i assumed we did not have it in show because we did nto have it rather then not including it intentionally hence the question | |
| 14:32:59 | dansmith | I just don't see where it could be actually used during create I mean | |
| 14:33:00 | sean-k-mooney | i guess we must | |
| 14:33:02 | sean-k-mooney | https://docs.openstack.org/api-ref/compute/#show-server-password | |
| 14:33:16 | dansmith | I'll keep looking, not asking you for that, just if you knew of any other edges | |
| 14:33:22 | dansmith | yep, that's the way to get it on a running instance | |
| 14:33:25 | sean-k-mooney | we likely kept it in its onw api to just not send it iin general when not needed | |
| 14:34:13 | sean-k-mooney | dansmith: no none that come to mind at the moment | |
| 14:34:41 | dansmith | ack, I'll try to chase down what we're doing (or not) with the password on create (I wonder if that was only ever honored by xen), | |
| 14:34:49 | sean-k-mooney | honestly it woudl be better if we sotred this in barbiacna but this is hella old | |
| 14:35:11 | dansmith | yeah, that's an idea I guess if we were going to modernize it | |
| 14:35:17 | sean-k-mooney | same for keyparis in keyston or barbican | |
| 14:35:49 | dansmith | the cloudbase-init post back to metadata sort of can't be private, so I guess it still needs to encrypt it manually or something | |
| 14:36:07 | dansmith | it's all pretty crusty hacks for 2026 | |
| 14:36:15 | sean-k-mooney | ya so that realying on just ssl at the infra level | |
| 14:36:29 | sean-k-mooney | but also cloudbase dont really do openstack stuff anymore | |
| 14:36:35 | sean-k-mooney | so we coudl perhaps deprecate/remove that | |
| 14:37:10 | sean-k-mooney | i think that was partly put in because we dont have a way for such an agent to call barbican or similar | |
| 14:37:31 | dansmith | what I mean is, no verifiable SSL cert for the metadata service | |
| 14:37:32 | sean-k-mooney | i.e. we dont have the k8s thing of providing a bootstrap/service token | |
| 14:38:08 | dansmith | yeah, deprecate would be best here I think.. but I also want to know if windows users are still relying on the nova->cloudbase-init for the initial password setting part | |
| 14:39:47 | sean-k-mooney | dansmith: by the way we alwasy encyrpt the password with your keypair at least when storign it/returning it | |
| 14:40:00 | sean-k-mooney | i dont knwo if cloud-base was doing that when posting it | |
| 14:40:06 | sean-k-mooney | or if that was on the metadta api side | |
| 14:40:14 | sean-k-mooney | im hoping it was in cloudbase | |
| 14:40:31 | sean-k-mooney | so that nova was never handelign the plaintext password | |
| 14:40:42 | dansmith | yep | |
| 14:41:24 | sean-k-mooney | windows user do often use this but they can use winrm (an i think ssh in newer windwos) or userdata to set the inital password | |
| 14:41:36 | dansmith | also, it looks like the password on create is only used by libvirt if we do injection or if we're creating config_drive.. I haven't yet seen that we store it if not doing either of htose | |
| 14:41:37 | sean-k-mooney | they likely wont | |
| 14:41:47 | sean-k-mooney | but that does at least work | |
| 14:42:07 | dansmith | oh also, I'm not sure if the admin password on create is encrypted ... it seems like not.. once you do set-admin-password its only encrypted, you're right about that | |
| 14:42:32 | sean-k-mooney | ... i dont alwasy like to be right | |
| 14:42:38 | dansmith | yeah, ssh in newer windows for sure | |
| 15:31:24 | dansmith | sean-k-mooney: FWIW, the adminPass generated on boot is stored in plaintext in the config_drive in meta_data.json | |
| 15:32:09 | dansmith | not stored in the database, so not visible via usual http metadata, but.. that would be easy to mount world-readable somewhere and expose your root password _if_ you decided to use it | |
| 15:33:58 | sean-k-mooney | not ideal. i hope we seting it to 440 or 400 today but ya with out nova providsion storage encycrption storing it in the configdriver vs root disk does nto really chagne the postture | |
| 15:35:25 | dansmith | it's either fat or iso9660, so no permissions.. depends on how you mount it | |
| 15:35:47 | dansmith | or if you have an OS that automounts CDs, it probably mounts it 555 | |
| 15:35:49 | sean-k-mooney | oh you mean in the guest rather then on the host | |
| 15:35:52 | dansmith | yes | |
| 15:36:07 | sean-k-mooney | ack | |
| 15:36:22 | dansmith | I meant expose your root password to any random untrusted thing that is running that can read world-readable things on the filesystem | |
| 15:36:22 | sean-k-mooney | i tought you ment an operator or soemoen with acess to read the file on the host | |
| 15:36:42 | sean-k-mooney | but yes so it wont automoutn by default however cloud init will do that | |
| 15:37:00 | sean-k-mooney | so if you have a clodu iamge i think its bindmounted in o /etc/cloud/config.... | |
| 15:37:38 | sean-k-mooney | i have never looked at hwo that is doen in the guest really so im not sure what permissison it will use | |
| 15:37:53 | dansmith | Windows will "mount" it automatically and a general-purpose OS would automount it | |
| 15:38:06 | dansmith | like ubuntu-desktop would end up with it mounted | |
| 15:38:32 | sean-k-mooney | oh right the DE(gnome) ectra woudl | |
| 15:39:02 | sean-k-mooney | not really sure now to avoid that other then contineut to advise agians enabling the fature | |
| 15:39:03 | dansmith | since cloud-init won't set the root password from that (AFAICT) then it's not a big deal there, but on Windows where cloudbase-init would/did it'd be readable | |
| 15:39:27 | dansmith | advise against enabling what, configdrive? | |