| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-07 | |||
| 19:34:55 | fungi | mikal: yes, i think this is the same degree of risk as a rogue dhcp server, assuming the system is configured to also run arbitrary code at boot that the dhcp server points it to? (not sure, that could be a bit of a pathological situation) | |
| 19:35:41 | sean-k-mooney | but there are several other first boot systems like ignition and cloudbase init that use the same 169.254.169.254 or http://f8e0 well know adddreses | |
| 19:36:12 | fungi | right, i have no idea if cloud-init supports getting metadata from those | |
| 19:36:36 | mikal | Wouldn't it be better to spend any development effort here on something like TLS for the metadata service with certificate verification turned on, instead of changing a mechanism that no one has complained about in nearly 20 years? | |
| 19:37:16 | sean-k-mooney | i think they "fixed" it by just chaging thte default in the config to not enmebrate by defualt | |
| 19:37:26 | fungi | well, technically someone *did* complain about it, hence the bug report | |
| 19:37:27 | sean-k-mooney | but i woudl expect distor to override that | |
| 19:38:10 | fungi | yes, distros could override it, or they could just build separate ec2/openstack images and othercloud images | |
| 19:38:12 | sean-k-mooney | as it stands they will have broken metadata supprot for ironic by default. so if you dont enbale configdrive with yoru ironic instnace it wong be able to use metadata out of the box | |
| 19:38:47 | fungi | i have no idea if e.g. debian's or ubuntu's generic cloud images are used on ironic | |
| 19:39:01 | sean-k-mooney | i have used them before | |
| 19:39:04 | mikal | Building N cloud images is actually a fairly big hosting problem I imagine. That's a fair bit of repeated data just for a very minor problem. | |
| 19:39:07 | sean-k-mooney | but it depend on yoru hardware | |
| 19:39:09 | fungi | seems like you'd want a more full-featured image with broader hardware support? | |
| 19:39:16 | fungi | yeah | |
| 19:39:18 | sean-k-mooney | somethime they dont have the drivers requried soemtimes they do | |
| 19:39:48 | sean-k-mooney | its been a long time since i tried but i think i used diskimage builder with the cloud image as a base to build one | |
| 19:40:19 | sean-k-mooney | cloud image doen nto mean qemu/kvm or vm based | |
| 19:40:21 | fungi | debian not including ahci support in the generic cloud kernels because "most" cloud providers (amazon/azure/google) is a good example of why i'd be skeptical of using such images in ironic for bare metal | |
| 19:40:31 | mikal | DIB definitely uses the various cloud images as the base for builds. | |
| 19:40:48 | sean-k-mooney | mikal: it can/does btu it can replace the kernel ectra | |
| 19:41:11 | mikal | sean-k-mooney: yeah, package installs etc etc. | |
| 19:41:23 | fungi | mmm, these days the dib elements we're using start from docker images instead, though previously we used elements that bootstrapped from scratch with tools like debootstrap | |
| 19:41:28 | mikal | This is how the SF "official" images are built. DIB with the SF agent sprinkled in.' | |
| 19:42:07 | sean-k-mooney | fungi: when did we move the ubuntu ones | |
| 19:42:09 | fungi | but yes, i think dib does have some non-"minimal" elements that instead modify cloud images with virsh tools | |
| 19:42:17 | sean-k-mooney | because they used to use the cloud image until relitivly recently | |
| 19:42:39 | fungi | sean-k-mooney: move from using debootstrap to container images? not sure if we have done that for the ubuntu images yet, i have to check | |
| 19:43:14 | fungi | but at least for the dib-created images we boot in opendev we've never used cloud images as the base | |
| 19:43:16 | sean-k-mooney | fungi: last time i looked and when i was lookign to add alpine most still did not use contianers | |
| 19:43:21 | sean-k-mooney | fedora i think did | |
| 19:43:32 | sean-k-mooney | but most still default to clodu images as a base | |
| 19:44:15 | sean-k-mooney | https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/ubuntu/root.d/10-cache-ubuntu-tarball#L16-L21 | |
| 19:44:25 | sean-k-mooney | ok they use the sqsfs rather then the qcow | |
| 19:44:53 | sean-k-mooney | or rather the filesystem image | |
| 19:44:56 | fungi | sean-k-mooney: yeah, looks like we're still using ubuntu-minimal (debootstrap into a chroot) not ubuntu-container | |
| 19:45:28 | sean-k-mooney | so i know there is a fedora container element https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/fedora-container | |
| 19:45:33 | fungi | the rocky images we build are using the rocky-container element by contrast | |
| 19:45:42 | sean-k-mooney | right | |
| 19:45:54 | sean-k-mooney | but it think that was only done for 1 or two and most didnt move yet | |
| 19:46:17 | fungi | yeah, looking back at it i think you're right, i assumed we were further along on that already | |
| 19:46:45 | fungi | i guess it's been on hold while we switch from nodepool to zuul-built images | |
| 19:47:35 | sean-k-mooney | i was playing with packer and trying to buidl a alpine image at the weekend and i was debating if i wanted to revive my dib serise for that | |
| 19:48:03 | sean-k-mooney | building os images is alwasy a pain regardess of the tooling | |
| 19:48:27 | sean-k-mooney | alpine now provides a cloud iamge with cloudinit however | |
| 19:48:39 | sean-k-mooney | its just larger then i was hoping | |
| 19:49:34 | sean-k-mooney | fungi: i still think exporeing alpine to have a replacement for cirros or moving to using nested kvm might be the only way we elimiate the kernel panics we see in our jobs | |
| 19:50:06 | fungi | it seems worth exploring, i agree | |
| 19:50:13 | sean-k-mooney | it does not have to be alpine but there are very few other distos that are small enough and well supproted to consider | |
| 19:51:24 | sean-k-mooney | anyway back to the topic at hand if folks are building an ironic image they can still use https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/cloud-init-datasources#environment-variables | |
| 19:51:30 | sean-k-mooney | to list openstack | |
| 19:52:03 | fungi | i want to say someone (doug?) was playing with using distros that target embedded systems (openwrt? emdebian? building something with yocto?) | |
| 19:52:15 | fungi | but it's been years ago now | |
| 19:52:37 | sean-k-mooney | fungi: i got it partly work i was just missing growpart :) | |
| 19:53:08 | sean-k-mooney | but ya i brefily considerd tinycore since ironic used it for the ipa | |
| 19:54:22 | sean-k-mooney | the problem is the combination of small disk footpring <100MB and small ram foot pring <128MB + cloud-init or glean +busybox/tools needed byt tempest | |
| 19:55:38 | sean-k-mooney | fungi: now that they offically publish cloud images i shoudl proably jsut poing my testing patch at that and see if it works | |
| 19:56:01 | fungi | worth a shot, sure | |
| 19:57:30 | sean-k-mooney | fungi: how is the ci since the roolout of the zuul change | |
| 19:57:57 | sean-k-mooney | fungi: have cross provider builds been avoided as expected? | |
| 19:58:16 | fungi | sean-k-mooney: not 100% solved, we discovered that we also need https://review.opendev.org/954284 | |
| 19:59:19 | sean-k-mooney | hum the cross tenant sharing of python object is slightly scary but ok | |
| 19:59:27 | sean-k-mooney | i guess that will rool out next weekend | |
| 19:59:49 | fungi | we'll probably restart the launchers onto it as soon as the container images promote | |
| 20:00:23 | fungi | turns out we had done that when the previous (partial) fix merged but i didn't realize it when discussing later in the week | |
| 20:03:03 | sean-k-mooney | oh i see the sharing is not a generic thing just in that funtion | |
| 20:03:32 | sean-k-mooney | i was slighly concured that semapheroes or secretes ectra could be shared but this is not related to the job objects | |
| 20:04:45 | fungi | no, you could look at it as the problem arising from a generally cross-tenant resource not having been correctly shared since zuul normally separates objects in one tenant from the others | |
| 20:06:49 | opendevreview | sean mooney proposed openstack/nova master: [WIP] use alpine instead of cirros https://review.opendev.org/c/openstack/nova/+/881912 | |
| 20:08:46 | sean-k-mooney | i think i had previously customised the alpin image i built to allow password login with gocubsgo for tempest although that might works via ssh key loging | |
| 20:11:10 | fungi | amusing, i never realized that was the default password for cirros images. i guess someone's a baseball fan | |
| 20:11:15 | sean-k-mooney | its hard to tell becase i never wrote the nodepool change to build the image in ci i only built it locally so i dont recall if i use the dev-user element or not | |
| 20:11:44 | mikal | fungi: that would be Scott Moser. | |
| 20:11:49 | sean-k-mooney | it used to be "cubswin:)" until they one and now its "gocubsgo" | |
| 20:11:54 | fungi | hah | |
| 20:12:12 | clarkb | re dhcp vs metadata service I think a difference is that typically dhcp is a fairly locked down service within hosting providers. The hosting services may not all realize hey needt o prevent traffic on that special metadata address | |
| 20:12:26 | clarkb | basically lock down dhcp is pretty standard stuff. Lock down metadata service isn't | |
| 20:12:47 | mikal | I just think this is a pretty big breaking change for a relatively minor "security vulnerability". | |
| 20:12:48 | sean-k-mooney | im not sure. i woudl cosnider both to be simialr leveles of treat | |
| 20:13:02 | clarkb | sean-k-mooney: they are. But one is super standard that everyone knows about. The other is not | |
| 20:13:13 | clarkb | basically the differencesi s one has an rfc the other doesn't | |
| 20:13:19 | mikal | Especially when I can think of use cases which want to get rid of the customized DMI data entirely | |
| 20:13:48 | fungi | also i don't know how i would go about using a malicious dhcp server to get a machine to run arbitrary code at boot or install access for my personal ssh key | |
| 20:13:54 | sean-k-mooney | mikal: i mentione dthis else whre but the dmi info is technically configuable by pakcager/deployers | |
| 20:14:02 | sean-k-mooney | nova reads a vendor.conf file if it exits | |
| 20:14:07 | sean-k-mooney | which change what that info is | |
| 20:14:39 | fungi | but yeah, all of this is great feedback to the cloud-init maintainers | |
| 20:14:41 | mikal | So now deployers can accidentally break cloud-init with a DMI customization? | |
| 20:14:44 | sean-k-mooney | we do not use that downstream in redhat partly because we saw the ablity to tell it was openstack or worse what version of openstack as a potital security problem | |
| 20:14:45 | mikal | That's not great either. | |
| 20:14:55 | fungi | and now that the bug is public, i suppose comments could be added tehre | |
| 20:16:31 | fungi | sean-k-mooney: i don't see us relying on the dev-user element anywhere, we have a custom element that we inject ssh keys with | |
| 20:17:01 | sean-k-mooney | fungi: that was not upstream i ment whewn i built my test alpine image locally | |
| 20:17:08 | fungi | ah | |
| 20:17:34 | sean-k-mooney | wheni was workign on the dib support i was usign the dev-user image to debug but its been 4 years so i have no idea if the iamge i was testing with still had that or not | |
| 20:17:49 | fungi | okay, just re-read your earlier comment about dev-user and now i get what you were saying | |
| 20:18:21 | sean-k-mooney | fungi: i got it to boot in the ci and it apear to be workign but boots hung because i did not expand the disk | |
| 20:18:29 | sean-k-mooney | so it would start ot boot and then fail | |
| 20:19:44 | sean-k-mooney | i actully did 2 impelmetions the most recent one was 2 years ago based on teh continar file support https://review.opendev.org/c/openstack/diskimage-builder/+/755410/4 | |
| 20:20:42 | sean-k-mooney | mikal: ya you need to have a file like this https://github.com/openstack/nova/blob/master/etc/nova/release.sample | |