| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-07 | |||
| 19:42:07 | sean-k-mooney | fungi: when did we move the ubuntu ones | |
| 19:42:09 | fungi | but yes, i think dib does have some non-"minimal" elements that instead modify cloud images with virsh tools | |
| 19:42:17 | sean-k-mooney | because they used to use the cloud image until relitivly recently | |
| 19:42:39 | fungi | sean-k-mooney: move from using debootstrap to container images? not sure if we have done that for the ubuntu images yet, i have to check | |
| 19:43:14 | fungi | but at least for the dib-created images we boot in opendev we've never used cloud images as the base | |
| 19:43:16 | sean-k-mooney | fungi: last time i looked and when i was lookign to add alpine most still did not use contianers | |
| 19:43:21 | sean-k-mooney | fedora i think did | |
| 19:43:32 | sean-k-mooney | but most still default to clodu images as a base | |
| 19:44:15 | sean-k-mooney | https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/ubuntu/root.d/10-cache-ubuntu-tarball#L16-L21 | |
| 19:44:25 | sean-k-mooney | ok they use the sqsfs rather then the qcow | |
| 19:44:53 | sean-k-mooney | or rather the filesystem image | |
| 19:44:56 | fungi | sean-k-mooney: yeah, looks like we're still using ubuntu-minimal (debootstrap into a chroot) not ubuntu-container | |
| 19:45:28 | sean-k-mooney | so i know there is a fedora container element https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/fedora-container | |
| 19:45:33 | fungi | the rocky images we build are using the rocky-container element by contrast | |
| 19:45:42 | sean-k-mooney | right | |
| 19:45:54 | sean-k-mooney | but it think that was only done for 1 or two and most didnt move yet | |
| 19:46:17 | fungi | yeah, looking back at it i think you're right, i assumed we were further along on that already | |
| 19:46:45 | fungi | i guess it's been on hold while we switch from nodepool to zuul-built images | |
| 19:47:35 | sean-k-mooney | i was playing with packer and trying to buidl a alpine image at the weekend and i was debating if i wanted to revive my dib serise for that | |
| 19:48:03 | sean-k-mooney | building os images is alwasy a pain regardess of the tooling | |
| 19:48:27 | sean-k-mooney | alpine now provides a cloud iamge with cloudinit however | |
| 19:48:39 | sean-k-mooney | its just larger then i was hoping | |
| 19:49:34 | sean-k-mooney | fungi: i still think exporeing alpine to have a replacement for cirros or moving to using nested kvm might be the only way we elimiate the kernel panics we see in our jobs | |
| 19:50:06 | fungi | it seems worth exploring, i agree | |
| 19:50:13 | sean-k-mooney | it does not have to be alpine but there are very few other distos that are small enough and well supproted to consider | |
| 19:51:24 | sean-k-mooney | anyway back to the topic at hand if folks are building an ironic image they can still use https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/cloud-init-datasources#environment-variables | |
| 19:51:30 | sean-k-mooney | to list openstack | |
| 19:52:03 | fungi | i want to say someone (doug?) was playing with using distros that target embedded systems (openwrt? emdebian? building something with yocto?) | |
| 19:52:15 | fungi | but it's been years ago now | |
| 19:52:37 | sean-k-mooney | fungi: i got it partly work i was just missing growpart :) | |
| 19:53:08 | sean-k-mooney | but ya i brefily considerd tinycore since ironic used it for the ipa | |
| 19:54:22 | sean-k-mooney | the problem is the combination of small disk footpring <100MB and small ram foot pring <128MB + cloud-init or glean +busybox/tools needed byt tempest | |
| 19:55:38 | sean-k-mooney | fungi: now that they offically publish cloud images i shoudl proably jsut poing my testing patch at that and see if it works | |
| 19:56:01 | fungi | worth a shot, sure | |
| 19:57:30 | sean-k-mooney | fungi: how is the ci since the roolout of the zuul change | |
| 19:57:57 | sean-k-mooney | fungi: have cross provider builds been avoided as expected? | |
| 19:58:16 | fungi | sean-k-mooney: not 100% solved, we discovered that we also need https://review.opendev.org/954284 | |
| 19:59:19 | sean-k-mooney | hum the cross tenant sharing of python object is slightly scary but ok | |
| 19:59:27 | sean-k-mooney | i guess that will rool out next weekend | |
| 19:59:49 | fungi | we'll probably restart the launchers onto it as soon as the container images promote | |
| 20:00:23 | fungi | turns out we had done that when the previous (partial) fix merged but i didn't realize it when discussing later in the week | |
| 20:03:03 | sean-k-mooney | oh i see the sharing is not a generic thing just in that funtion | |
| 20:03:32 | sean-k-mooney | i was slighly concured that semapheroes or secretes ectra could be shared but this is not related to the job objects | |
| 20:04:45 | fungi | no, you could look at it as the problem arising from a generally cross-tenant resource not having been correctly shared since zuul normally separates objects in one tenant from the others | |
| 20:06:49 | opendevreview | sean mooney proposed openstack/nova master: [WIP] use alpine instead of cirros https://review.opendev.org/c/openstack/nova/+/881912 | |
| 20:08:46 | sean-k-mooney | i think i had previously customised the alpin image i built to allow password login with gocubsgo for tempest although that might works via ssh key loging | |
| 20:11:10 | fungi | amusing, i never realized that was the default password for cirros images. i guess someone's a baseball fan | |
| 20:11:15 | sean-k-mooney | its hard to tell becase i never wrote the nodepool change to build the image in ci i only built it locally so i dont recall if i use the dev-user element or not | |
| 20:11:44 | mikal | fungi: that would be Scott Moser. | |
| 20:11:49 | sean-k-mooney | it used to be "cubswin:)" until they one and now its "gocubsgo" | |
| 20:11:54 | fungi | hah | |
| 20:12:12 | clarkb | re dhcp vs metadata service I think a difference is that typically dhcp is a fairly locked down service within hosting providers. The hosting services may not all realize hey needt o prevent traffic on that special metadata address | |
| 20:12:26 | clarkb | basically lock down dhcp is pretty standard stuff. Lock down metadata service isn't | |
| 20:12:47 | mikal | I just think this is a pretty big breaking change for a relatively minor "security vulnerability". | |
| 20:12:48 | sean-k-mooney | im not sure. i woudl cosnider both to be simialr leveles of treat | |
| 20:13:02 | clarkb | sean-k-mooney: they are. But one is super standard that everyone knows about. The other is not | |
| 20:13:13 | clarkb | basically the differencesi s one has an rfc the other doesn't | |
| 20:13:19 | mikal | Especially when I can think of use cases which want to get rid of the customized DMI data entirely | |
| 20:13:48 | fungi | also i don't know how i would go about using a malicious dhcp server to get a machine to run arbitrary code at boot or install access for my personal ssh key | |
| 20:13:54 | sean-k-mooney | mikal: i mentione dthis else whre but the dmi info is technically configuable by pakcager/deployers | |
| 20:14:02 | sean-k-mooney | nova reads a vendor.conf file if it exits | |
| 20:14:07 | sean-k-mooney | which change what that info is | |
| 20:14:39 | fungi | but yeah, all of this is great feedback to the cloud-init maintainers | |
| 20:14:41 | mikal | So now deployers can accidentally break cloud-init with a DMI customization? | |
| 20:14:44 | sean-k-mooney | we do not use that downstream in redhat partly because we saw the ablity to tell it was openstack or worse what version of openstack as a potital security problem | |
| 20:14:45 | mikal | That's not great either. | |
| 20:14:55 | fungi | and now that the bug is public, i suppose comments could be added tehre | |
| 20:16:31 | fungi | sean-k-mooney: i don't see us relying on the dev-user element anywhere, we have a custom element that we inject ssh keys with | |
| 20:17:01 | sean-k-mooney | fungi: that was not upstream i ment whewn i built my test alpine image locally | |
| 20:17:08 | fungi | ah | |
| 20:17:34 | sean-k-mooney | wheni was workign on the dib support i was usign the dev-user image to debug but its been 4 years so i have no idea if the iamge i was testing with still had that or not | |
| 20:17:49 | fungi | okay, just re-read your earlier comment about dev-user and now i get what you were saying | |
| 20:18:21 | sean-k-mooney | fungi: i got it to boot in the ci and it apear to be workign but boots hung because i did not expand the disk | |
| 20:18:29 | sean-k-mooney | so it would start ot boot and then fail | |
| 20:19:44 | sean-k-mooney | i actully did 2 impelmetions the most recent one was 2 years ago based on teh continar file support https://review.opendev.org/c/openstack/diskimage-builder/+/755410/4 | |
| 20:20:42 | sean-k-mooney | mikal: ya you need to have a file like this https://github.com/openstack/nova/blob/master/etc/nova/release.sample | |
| 20:20:53 | sean-k-mooney | mikal: https://github.com/openstack/nova/commit/4aa45d2900a3063305a7b2c727abfc2371909b21 | |
| 20:21:27 | sean-k-mooney | the support for that still technially exists https://github.com/openstack/nova/blob/master/nova/version.py | |
| 20:21:37 | sean-k-mooney | but nothing i know of actully uses that | |
| 20:22:10 | mikal | But that's the key point. OpenStack tries hard not to break existing behaviour. cloud-init apparently does not. | |
| 20:22:39 | mikal | I think this is the strongest argument I've seen as to why people should move to glean, especially because glean implies that config drive will not in fact one day be removed. | |
| 20:23:14 | sean-k-mooney | there are other light wight alternitives too. the problem with glean is partly its streght | |
| 20:23:22 | sean-k-mooney | it does not supprot metadtaa only config drive | |
| 20:23:30 | sean-k-mooney | which si why glean is so reliable | |
| 20:24:15 | sean-k-mooney | but since we cant update config dirve it only really good for first boot but honestly if your doing day two thinbgs use ansible | |
| 20:25:04 | fungi | i don't think it's entirely fair to say that cloud-init doesn't try to preserve existing behaviors, just in this particular case they judged the existing behavior to be a greater risk than the impact from breaking it (whether they judged that accurately is another matter) | |
| 20:25:51 | sean-k-mooney | i guess we will see if we get a spike in reports of "my ssh key was not injected" or similar | |
| 20:26:05 | fungi | they're typically pretty good about not breaking functionality, and in this particular case the bug was open in private for over a year while they debated the risks of doing so | |
| 20:27:15 | sean-k-mooney | on a related not the thing i whish we had time to do was add supprot for jwt or applciation creditials via config dirve. i.e. provide a way simialr to k8s for you to provide and authentication token to the workloads | |
| 20:27:37 | fungi | but yes, the complexity and expansive scope of cloud-init, as well as their choice to rely on lots of different non-stdlib python libs, is why we made and use glean for our ci nodes | |
| 20:28:02 | sean-k-mooney | its something that aws and all the other cloud already do but providing a way to do athestation or the server and auth is a gap we have today. | |
| 20:28:30 | fungi | zuul has jwt support and can do oidc claims too | |
| 20:29:08 | sean-k-mooney | fungi: we are only using glean for the nodepool images right? cirros has its own mini implemation in bash that it uses instead | |
| 20:29:11 | fungi | though i'm not sure if we could leverage that to authenticate metadata server responses | |
| 20:29:24 | fungi | sean-k-mooney: correct | |
| 20:29:44 | clarkb | fungi: the zuul implementation requires the third party set up zuul as a trusted oidc provider | |
| 20:29:46 | fungi | we don't (currently) build cirros images ourselves afaik | |
| 20:30:32 | fungi | clarkb: yeah, i was trying to think if there was a way client-side to leverage that, but the booting server instance would be the client in that case and needs to be reachable before zuul can interact with it | |
| 20:30:35 | sean-k-mooney | this is there code https://github.com/cirros-dev/cirros/blob/main/src/usr/bin/ec2metadata and not we dont we use the upstream images and back them into the ci images to avoid downloadeing it in the job | |
| 20:31:38 | fungi | yeah, we download the upstream cirros images and ship convenience copies in the cache in our test nodes | |