| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-09 | |||
| 19:30:47 | mriedem | i think i'll probably be back at ibm working on php / xenapi by the time that happens.. | |
| 19:31:03 | mriedem | you know, the up and coming tech | |
| 19:31:04 | dansmith | it's a thing people want though | |
| 19:31:27 | mriedem | yeah, huawei's public cloud has a beta for cross-cell live migration, | |
| 19:31:28 | dansmith | especially if it were to make it easier to migrate from an older cloud to a newer one by making one a cell of the other until it's emptied | |
| 19:31:30 | mriedem | i'm not sure how they do it but... | |
| 19:31:34 | dansmith | that's been requested since icehouse or so | |
| 19:32:52 | sean-k-mooney | dansmith: we all know livemigration never works :P also if i get sirov live migration working this cycle i have no plan to test cross cell, sriov, nuam aware live migration between different releases during upgrade :) | |
| 19:33:14 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: WIP: Test report_ironic_standard_resource_class_inventory=False https://review.openstack.org/609107 | |
| 19:33:40 | mriedem | there is no cross cell live migration... | |
| 19:34:51 | sean-k-mooney | mriedem: didnt you jsut say huawei's public cloud has beta supprot. i would assume they will ask you to upstream it a some point or is that complete different devision form yours? | |
| 19:35:40 | mriedem | completely different | |
| 19:35:54 | mriedem | their public cloud is still using cascading, which is their proprietary cells v1 | |
| 19:36:17 | mriedem | they are working on migrating off that to cells v2 | |
| 19:36:22 | dansmith | that's more like cross-deployment live migration | |
| 19:36:50 | sean-k-mooney | dansmith: is that still an ask from the edge working group? | |
| 19:37:06 | dansmith | is what? cross-deployment live migration? | |
| 19:37:12 | dansmith | I'm sure it'll be on their list at some point | |
| 19:37:27 | dansmith | I can see the cross-cell thing being fine, but I dunno about cross-deploy | |
| 19:38:17 | sean-k-mooney | ya when i was in the edge room in dublin i spent 15 minute explaining why cross cloud inter hyperviror live migration was never going to be a thing and should not be in the phase 1 basic feature support for edge | |
| 19:39:15 | sean-k-mooney | they litrally wanted to live migrate form libvirt + ceph in one edge site to vmware on another | |
| 19:39:56 | jaypipes | dansmith, mriedem, melwitt: is there a reason we don't pass instance metadata to the RequestSpec? | |
| 19:40:18 | dansmith | jaypipes: why would we need to? | |
| 19:40:34 | jaypipes | dansmith: so that scheduler filters can look at the instance metadata? :) | |
| 19:40:40 | dansmith | they should never do that | |
| 19:40:46 | dansmith | instance metadata is owned by the user not nova | |
| 19:40:52 | jaypipes | dansmith: oh, but Oath disagrees strongly. ;) | |
| 19:40:54 | dansmith | looking at it, especially for placement would violate | |
| 19:40:58 | dansmith | jaypipes: -2 | |
| 19:41:02 | jaypipes | hehe | |
| 19:41:03 | melwitt | but what about... custom filters YALL | |
| 19:41:13 | jaypipes | dansmith: what melwitt said :) | |
| 19:41:17 | dansmith | yeah | |
| 19:41:19 | jaypipes | dansmith: example... | |
| 19:41:23 | sean-k-mooney | jaypipes: jay just stuff the info in a schduler hint and use the json fileter | |
| 19:41:57 | mriedem | the request spec has an instance_uuid on it, you can get the instance from that and pull the metadata off the instance; that won't work for multi-create, but you probably don't care at oath | |
| 19:42:11 | dansmith | yup | |
| 19:42:15 | jaypipes | dansmith: nova boot --property ytag=SOME_CUSTOM_YAHOO_GOOP; nova scheduler has a filter that does an external lookup to our inventory management system of the ytag to grab the availability zone (really, just a power domain) to send the instance to | |
| 19:42:22 | dansmith | or hint yourself to an external artifact, fetch it and go nuts | |
| 19:42:44 | dansmith | jaypipes: yep, I got it, but metadata is off limits | |
| 19:43:17 | dansmith | --hint ytag-fml -> look up fml externally, do a thing | |
| 19:43:19 | jaypipes | mriedem: you can't do that. the instance_uuid is for the mapping, but the metadata doesn't exist yet, so if you try to call Instance.get_by_uuid(instance_uuid) from the filter, that's a dead end. | |
| 19:43:34 | mriedem | jaypipes: get the build request then | |
| 19:43:43 | jaypipes | mriedem: build request doesn't store metadata. | |
| 19:43:44 | melwitt | last I heard, oath does use multi-create. there's auto-scale stuff that boots several instances at once | |
| 19:43:47 | mriedem | jaypipes: sure it does, | |
| 19:43:49 | mriedem | it stores the instance | |
| 19:43:52 | mriedem | which stores the metadata | |
| 19:44:23 | dansmith | right, that's how we know what to create when we've picked a host :) | |
| 19:44:27 | jaypipes | mriedem: https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L1004-L1007 | |
| 19:44:39 | jaypipes | mriedem: where exactly does the build request store the instance metadata? | |
| 19:44:40 | mriedem | instance=instance | |
| 19:44:50 | dansmith | it's in instance | |
| 19:44:51 | dansmith | heh yeah | |
| 19:45:00 | mriedem | instance.update(base_options) | |
| 19:45:08 | mriedem | build request stores a serialized instance object | |
| 19:45:13 | jaypipes | ah.. | |
| 19:45:19 | jaypipes | I missed that. sorry. | |
| 19:45:30 | jaypipes | ok, so I'll change the filter to pull the build request by instance_uuid. | |
| 19:45:34 | mriedem | https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L936 | |
| 19:45:35 | jaypipes | thanks y'all | |
| 19:45:45 | mriedem | performance will suck | |
| 19:45:47 | mriedem | but... | |
| 19:45:52 | dansmith | but you're going to hell anyway? | |
| 19:45:53 | dansmith | yah. | |
| 19:45:57 | jaypipes | mriedem: yes, understood. | |
| 19:46:01 | sean-k-mooney | hum in that case the json fileter and compute capablity filters can already read it and do stuff... | |
| 19:46:25 | jaypipes | sean-k-mooney: we already have a custom IronicCapabilitiesFilter. don't get me started :) | |
| 19:46:48 | melwitt | jaypipes: yeah, you might probably run into perf problems during a db lookup in a filter (but I guess you said earlier you're doing an external system lookup in a filter already and that wasn't hurting perf?) | |
| 19:46:58 | sean-k-mooney | jaypipes: are you thinking of pulling this suff out in a pre placement filter or post out of interest | |
| 19:47:27 | jaypipes | this one's actually a network availability filter that looks for num_additional_ipv4 and num_ipv6 custom metadata key/values, calls out to our IPAM system from within the filter itself, and determines if the target system has enough IP addresses available. | |
| 19:47:31 | jaypipes | dansmith: you're welcome ^ | |
| 19:47:35 | melwitt | I remember when I worked at yahoo, we ran into perf issues with an in-tree filter that was doing db lookups and had to patch it out (and upstream fixed it soon after) | |
| 19:48:04 | jaypipes | melwitt: this is even worse. :) it's doing out of band calls to a REST API from within the in-tree filter :) | |
| 19:48:11 | sean-k-mooney | jaypipes: for l3 routeded network we are storing that kindo of info in placement | |
| 19:48:16 | melwitt | yeah. interesting that it's not causing perf issues | |
| 19:48:34 | jaypipes | melwitt: well, it's not like the traffic to the scheduler is huge... | |
| 19:49:00 | sean-k-mooney | jaypipes: any way you could jsut write some kind of bridge between neutron and the ipam to model it in placement and not use a scheduler filter | |
| 19:49:00 | jaypipes | melwitt: I mean, it's not like there's thousands of concurrent callers of nova boot for ironic hosts. | |
| 19:49:06 | melwitt | it used to be, is what I'm saying. but that was back before we had placement filtering the set of compute nodes down | |
| 19:49:14 | jaypipes | sean-k-mooney: baby steps :) | |
| 19:49:43 | jaypipes | melwitt: we're getting there.. slowly :) | |
| 19:50:07 | sean-k-mooney | jaypipes: hehe ok it just seam like some of that code might already exists for the l3 routed network and with gibis resouce stuff you might be able to have neutron pass say a host aggreage and ip reouce request back | |
| 19:50:29 | sean-k-mooney | * placement aggreate | |
| 19:51:20 | melwitt | oh, ironic. I don't know much about that. I think what I was referring to was VMs. it was since there were 1000 compute nodes, have a db lookup once per the 1000 and having concurrent requests caused the problems | |
| 19:51:27 | jaypipes | sean-k-mooney: ALL of that code already exists in the L3 routed net segments stuff :) we just need to get there first.. | |
| 19:51:46 | melwitt | *db lookup for each of the 1000 | |
| 19:53:51 | sean-k-mooney | jaypipes: so playing devils advocate here if you wrote a neutron ipam plugin driver and and we had a way to get the info from neutron to nova via the port you would not need the filter right. | |
| 19:54:35 | jaypipes | sean-k-mooney: correct. | |
| 19:54:46 | sean-k-mooney | jaypipes: im just thinking this would be generically useful out side oath too but for mass reuse we would not want to go the schduler filter route | |
| 19:54:48 | jaypipes | sean-k-mooney: I welcome your imminent pull request to our repo. | |
| 19:55:35 | sean-k-mooney | :) well have you talked to any neutron folk about how much effort that would be? | |
| 19:55:56 | sean-k-mooney | sound like you can do the schduler filter out of tree without nova changes in anycase | |
| 19:56:18 | mriedem | dansmith: on the detach/attach root volume spec https://review.openstack.org/#/c/600628/ did you see that it was updated to also allow detaching the root volume from stopped instances in addition to shelved offloaded? | |
| 19:56:30 | jaypipes | sean-k-mooney: though the "interesting" part about this particular filter is that it's quantitative -- "make sure this baremetal host is in rack that has X number of available IPv4s in its subnet" -- but the request isn't actually *for* that amount. It's basically "ok, just make sure I *could* get this many IPs, but don't actually grab those IPs for me. right now. maybe later, ok thx bai" | |
| 19:57:00 | dansmith | mriedem: no I saw some activity on that this morning and have it queued | |
| 19:57:12 | mriedem | jaypipes: sounds like what you want is quotas and resource claims brother! | |
| 19:57:31 | melwitt | jaypipes: that sounds kind of like the key/value discussion we've been having. need disk_typeA=4 but don't want to actually consume them | |
| 19:58:01 | mriedem | your fingers say no but your mouth....also says no | |
| 19:58:15 | melwitt | so I wonder if you could solve your complex partitioning/layout issues similarly | |