| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-11 | |||
| 18:45:54 | mordred | so at this point the main difference is size - glean assumes the only thing you want it to do is get networking set up and inject ssh keys - and that you NEVER want to talk to non-config-drive metadata service | |
| 18:46:12 | mordred | cloud-init is much more flexible and lets you inject arbitrary scripts | |
| 18:46:20 | mordred | which some people think is cool | |
| 18:47:05 | mordred | and I support them in that - I, for myself, _never_ want to run scripts via cloud-init because I can run scripts over ssh much more easily and more debuggably :) | |
| 18:47:07 | dansmith | mordred: orly, glean is configdrive-only? | |
| 18:47:15 | mordred | dansmith: yup | |
| 18:47:18 | dansmith | interesting | |
| 18:47:34 | mordred | we have had nothing but terrible experiences with the ec2-style metadata service | |
| 18:47:42 | mordred | crashing clouds and instances not booting and instances timing out and whatnot | |
| 18:47:50 | mordred | so we avoid it like the plague | |
| 18:48:12 | mordred | config-drive is teh ossum | |
| 18:48:30 | dansmith | mordred: yeah, especially when you have puppet running every 15 minutes on every instance against central metadata services | |
| 18:48:44 | mordred | dansmith: yup | |
| 18:49:07 | dansmith | and when I say every fifteen minutes I mean "for 14 minutes total until smoke" | |
| 18:49:47 | mordred | dansmith: since cloud-init does support config-drive, i'd LOVE it if nova just defaulted config-drive to on, removed the option to turn it off, and then removed the ec2 metadata service | |
| 18:50:07 | dansmith | yeah, the only problem is the staticness of it | |
| 18:50:12 | mordred | dansmith: but I suppose that's at odds with my stance about not changing things, so maybe I'm a hypocrit :) | |
| 18:50:16 | dansmith | i.e. volume attach and nic attach | |
| 18:50:40 | mordred | right- so ... | |
| 18:50:55 | jaypipes | lmao | |
| 18:51:21 | mordred | network attach should just work, since clouds should just be offering dhcp, which means there is no need for config-drive to be updated (unless the cloud is broken) | |
| 18:51:36 | dansmith | mordred: unless you want/need metadata about that nic | |
| 18:52:06 | mordred | but if you're adding a NIC after boot, you're adding the nic, so you as the orchestrator likely already know you're doing it and should be able to direct the instance to do something different,no? | |
| 18:52:37 | dansmith | unless you have an agent that looks for nics with tags of "multipath-to-db" and enslaves them into a bond, | |
| 18:52:40 | mordred | I mean, the only reason we need config-drive for network info anyway is for clouds without dhcp - for the rest I recommend using your orchestration tool to talk to the nova api | |
| 18:52:50 | dansmith | not necessarily | |
| 18:52:53 | jaypipes | mordred: right, unless you're a VNFM or MANO system that is dynamically swapping metadata about your workload's vNICs at runtime and having the workload reconfigure service function chains. | |
| 18:53:07 | dansmith | that ^ | |
| 18:53:50 | mordred | I'm still not sure why that isn't just an API call on the service API - which would of course be hard because we haven't bothered with per-instance credentials :) | |
| 18:53:50 | dansmith | jaypipes: it does kinda sound like a chemical | |
| 18:54:24 | jaypipes | mordred: bingo. | |
| 18:54:29 | mordred | but ec2 metadata services is a poor workaround for having an agent in an instance just make API calls to the control plane in a sane fashion | |
| 18:54:46 | jaypipes | mordred: also correct. | |
| 18:55:30 | dansmith | so you want the instance to hit the api, find out the ports that are on the instance, and corellate them all by mac address to know what to do with which actual device in the OS? | |
| 18:55:56 | mordred | dansmith: well, the list ports call also lists the mac addresses of the ports | |
| 18:56:02 | mordred | but yah - I do that for every instance we spin up in nodepool | |
| 18:56:16 | mordred | because I have to to figure out if the instance needs a floating ip or not | |
| 18:56:42 | mordred | so sure! | |
| 18:57:13 | dansmith | that works if you've got creds in the instance | |
| 18:57:29 | mordred | we could also just have an API call that gives a rollup report similar to what's in the metadata service - I'm just saying it doesn't need to be a special network port that's only usable to talk to the special service that doesn't scale | |
| 18:57:37 | mordred | and instead we should focus on getting creds into the instance | |
| 18:57:47 | mordred | which would make many people happy | |
| 18:58:29 | dansmith | well, I dunno, given where we are, tags on the devices we add seems like a reasonable thing to me | |
| 18:58:42 | dansmith | especially given that there are some devices for which there is no service to talk to | |
| 18:58:46 | dansmith | like a raw pci device | |
| 18:59:15 | mordred | wouldn't that raw pci device be there at boot? or be a thing you added to the vm via nova? | |
| 18:59:16 | dansmith | if cinder gives us the serial number of the block device that we're going to add or something, then I guess we can do the same for cinder, | |
| 18:59:22 | dansmith | but otherwise that's a problem | |
| 18:59:43 | mordred | I mean - I hear you | |
| 18:59:45 | dansmith | mordred: what does it matter if it was there at boot? | |
| 19:00:04 | mordred | if it's there at boot, it can have data in config-drive without any issues | |
| 19:00:23 | mordred | the static nature of config-drive and dynamic devices was the issue | |
| 19:00:34 | dansmith | sure, I thought we were talking about not needing metadata at al | |
| 19:00:38 | mordred | oh golly no | |
| 19:00:41 | mordred | metadata is great | |
| 19:00:43 | dansmith | cinder doesn't show me a device serial number or aything, | |
| 19:00:51 | dansmith | so not sure that I can do the same correlation inside the guest for it | |
| 19:00:57 | mordred | well, mounting volumes is hard no matter what | |
| 19:01:09 | mordred | because how they're presented to the guest varies by hypervisor | |
| 19:01:22 | mordred | and the api can lie to you about the device name | |
| 19:01:24 | dansmith | well, same for nic, except we have an address in the case of ethernet | |
| 19:01:39 | mordred | dansmith: WHAT ABOUT TOKEN RING??? | |
| 19:01:47 | dansmith | mordred: indeed, what about token ring indeed, | |
| 19:01:55 | dansmith | but while you say that, | |
| 19:02:04 | dansmith | there are definitely requests for weird custom L2 things | |
| 19:02:15 | mordred | oh, I'm sure there are | |
| 19:02:15 | dansmith | at which I laugh, but.. | |
| 19:03:13 | mordred | so ... my world view has given up on magical agents sitting inside of guests looking for metadata updates to automagically reconfigure themselves | |
| 19:03:20 | mordred | that might color my opinion on how these things should work | |
| 19:03:41 | dansmith | well, in that case you don't really need instance creds, right? | |
| 19:03:56 | mordred | having systems that sit outside of the guests, that konw what's going on and that can reach in and execute commands in the guests works much better in my world so far | |
| 19:04:03 | mordred | dansmith: that is correct, I do not | |
| 19:04:23 | dansmith | anyway, I just think that being consistent about how you plumb things at boot or after boot requires metadata, and I like configdrive for that for all cases other than that it can't be updated after boot | |
| 19:04:25 | mordred | however, if instance creds are there, then in-instance things have access to the same power I have while sitting outside of the guest | |
| 19:04:36 | mordred | dansmith: ++ | |
| 19:04:57 | mordred | and I'd like more metadata - especially in cases where our current metadata is missing or lies | |
| 19:05:07 | mordred | "this is the volume you named bob" | |
| 19:05:07 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Idempotent PUT /resource_classes/{name} https://review.openstack.org/448791 | |
| 19:05:19 | dansmith | mordred: right and we have that now, but only at boot for configdrive | |
| 19:05:23 | dansmith | the volume thing I mean | |
| 19:05:32 | dansmith | because we can generate that data only in the hypervisor driver | |
| 19:05:39 | mordred | dansmith: we ALSO have the volume thing in libvirt with device names | |
| 19:05:48 | dansmith | and because it's virt-specific, we only show that to the guest not the external API | |
| 19:05:59 | dansmith | mordred: not with volume tags which we now have | |
| 19:06:12 | dansmith | mordred: now you get a metadata blob that says "your vdb IS the volume you tagged as foo" | |
| 19:06:37 | mordred | and that tag does't show up in /dev/disk? | |
| 19:06:53 | dansmith | meaning /dev/disk/by-tag/? | |
| 19:07:05 | mordred | yah | |
| 19:07:07 | dansmith | something in the guest would have to do that for us, of course | |
| 19:08:10 | mordred | we get labels for normal cinder volumes now though - but I guess a disk can only have one lable and could have many tags | |
| 19:08:27 | dansmith | we get labels if they lay a filesystem down, right? | |
| 19:08:57 | mordred | we get a by-$something thathas the volume name in somewhere - one sec, lemme find the thing I'm talking about | |
| 19:09:35 | dansmith | I don't think so | |
| 19:09:46 | dansmith | that would be by-label, and if they put a filesystem in and labeled it | |
| 19:10:13 | dansmith | or are you talking about the virtio-$neutered_uuid ? | |
| 19:12:04 | mordred | dansmith: ok. I think I may just be smoking crack | |
| 19:12:11 | dansmith | I would totally believe that | |
| 19:12:13 | dansmith | :) | |
| 19:12:30 | mordred | dansmith: it's 19:12 UTC, which everyone knows is crack-o'clock | |
| 19:12:38 | dansmith | obvi | |