| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-09 | |||
| 19:45:57 | jaypipes | mriedem: yes, understood. | |
| 19:46:01 | sean-k-mooney | hum in that case the json fileter and compute capablity filters can already read it and do stuff... | |
| 19:46:25 | jaypipes | sean-k-mooney: we already have a custom IronicCapabilitiesFilter. don't get me started :) | |
| 19:46:48 | melwitt | jaypipes: yeah, you might probably run into perf problems during a db lookup in a filter (but I guess you said earlier you're doing an external system lookup in a filter already and that wasn't hurting perf?) | |
| 19:46:58 | sean-k-mooney | jaypipes: are you thinking of pulling this suff out in a pre placement filter or post out of interest | |
| 19:47:27 | jaypipes | this one's actually a network availability filter that looks for num_additional_ipv4 and num_ipv6 custom metadata key/values, calls out to our IPAM system from within the filter itself, and determines if the target system has enough IP addresses available. | |
| 19:47:31 | jaypipes | dansmith: you're welcome ^ | |
| 19:47:35 | melwitt | I remember when I worked at yahoo, we ran into perf issues with an in-tree filter that was doing db lookups and had to patch it out (and upstream fixed it soon after) | |
| 19:48:04 | jaypipes | melwitt: this is even worse. :) it's doing out of band calls to a REST API from within the in-tree filter :) | |
| 19:48:11 | sean-k-mooney | jaypipes: for l3 routeded network we are storing that kindo of info in placement | |
| 19:48:16 | melwitt | yeah. interesting that it's not causing perf issues | |
| 19:48:34 | jaypipes | melwitt: well, it's not like the traffic to the scheduler is huge... | |
| 19:49:00 | sean-k-mooney | jaypipes: any way you could jsut write some kind of bridge between neutron and the ipam to model it in placement and not use a scheduler filter | |
| 19:49:00 | jaypipes | melwitt: I mean, it's not like there's thousands of concurrent callers of nova boot for ironic hosts. | |
| 19:49:06 | melwitt | it used to be, is what I'm saying. but that was back before we had placement filtering the set of compute nodes down | |
| 19:49:14 | jaypipes | sean-k-mooney: baby steps :) | |
| 19:49:43 | jaypipes | melwitt: we're getting there.. slowly :) | |
| 19:50:07 | sean-k-mooney | jaypipes: hehe ok it just seam like some of that code might already exists for the l3 routed network and with gibis resouce stuff you might be able to have neutron pass say a host aggreage and ip reouce request back | |
| 19:50:29 | sean-k-mooney | * placement aggreate | |
| 19:51:20 | melwitt | oh, ironic. I don't know much about that. I think what I was referring to was VMs. it was since there were 1000 compute nodes, have a db lookup once per the 1000 and having concurrent requests caused the problems | |
| 19:51:27 | jaypipes | sean-k-mooney: ALL of that code already exists in the L3 routed net segments stuff :) we just need to get there first.. | |
| 19:51:46 | melwitt | *db lookup for each of the 1000 | |
| 19:53:51 | sean-k-mooney | jaypipes: so playing devils advocate here if you wrote a neutron ipam plugin driver and and we had a way to get the info from neutron to nova via the port you would not need the filter right. | |
| 19:54:35 | jaypipes | sean-k-mooney: correct. | |
| 19:54:46 | sean-k-mooney | jaypipes: im just thinking this would be generically useful out side oath too but for mass reuse we would not want to go the schduler filter route | |
| 19:54:48 | jaypipes | sean-k-mooney: I welcome your imminent pull request to our repo. | |
| 19:55:35 | sean-k-mooney | :) well have you talked to any neutron folk about how much effort that would be? | |
| 19:55:56 | sean-k-mooney | sound like you can do the schduler filter out of tree without nova changes in anycase | |
| 19:56:18 | mriedem | dansmith: on the detach/attach root volume spec https://review.openstack.org/#/c/600628/ did you see that it was updated to also allow detaching the root volume from stopped instances in addition to shelved offloaded? | |
| 19:56:30 | jaypipes | sean-k-mooney: though the "interesting" part about this particular filter is that it's quantitative -- "make sure this baremetal host is in rack that has X number of available IPv4s in its subnet" -- but the request isn't actually *for* that amount. It's basically "ok, just make sure I *could* get this many IPs, but don't actually grab those IPs for me. right now. maybe later, ok thx bai" | |
| 19:57:00 | dansmith | mriedem: no I saw some activity on that this morning and have it queued | |
| 19:57:12 | mriedem | jaypipes: sounds like what you want is quotas and resource claims brother! | |
| 19:57:31 | melwitt | jaypipes: that sounds kind of like the key/value discussion we've been having. need disk_typeA=4 but don't want to actually consume them | |
| 19:58:01 | mriedem | your fingers say no but your mouth....also says no | |
| 19:58:15 | melwitt | so I wonder if you could solve your complex partitioning/layout issues similarly | |
| 19:58:39 | mriedem | dansmith: yeah i'm not sure how i feel about it....but i'm also not sure i have a good excuse against allowing it | |
| 19:58:41 | sean-k-mooney | jaypipes: right ok so your not reseving the ips so your hoping that if you need them in the future you could resrve them | |
| 19:58:52 | sean-k-mooney | jaypipes: that sound more like a weigher | |
| 19:59:04 | dansmith | mriedem: well, I thought the shelve bit was because we didn't have to worry about disconnecting on the compute node | |
| 19:59:25 | mriedem | it's definitely more straight forward if the instance is shelved | |
| 19:59:30 | dansmith | yeah | |
| 19:59:31 | dansmith | anywa | |
| 19:59:35 | dansmith | I'll try to get around to it | |
| 20:05:58 | sean-k-mooney | melwitt: jaypipes not sure how the disk_typeA=4 is intended to work but asumming this was all in placement you "could" ask placement for 4 ips in this case but instead of claiming all the resouces in the allocation candiate only calim 1 ip. | |
| 20:06:48 | sean-k-mooney | melwitt: jaypipes that said for that code to be in the nova tree it would have to be generic and not for jsut this usecase. not sure how you would model that in flvor extraspecs however | |
| 20:06:49 | mriedem | jaypipes: efried: i'm trying to figure out what's going on with https://review.openstack.org/#/c/552105/ and https://review.openstack.org/#/c/544683/ from reading the ptg etherpad and it's not really clear to me if those are supposed to be combined or initial allocation ratios is a dependency for the other spec? | |
| 20:07:23 | mriedem | i spoke with yikun last night and he's confused as to what should be changed based on the etherpad, and i kind of am too since the etherpad is just mostly discussion | |
| 20:07:39 | melwitt | I think the initial ratios spec is a dependency for the other spec. two specs needed | |
| 20:07:58 | mriedem | https://review.openstack.org/#/c/544683/ says "#agreed in Stein PTG to squash this into [1]" | |
| 20:08:08 | mriedem | and the etherpad says "Sounds like the two specs need to be combined a bit." | |
| 20:08:23 | melwitt | ok, then I must not have understood | |
| 20:08:55 | melwitt | I thought it was two specs, one to define the initial allocation ratios and another to define how to handle the initial allocation ratios | |
| 20:09:20 | mriedem | why wouldn't that just be one spec? | |
| 20:09:55 | sean-k-mooney | mriedem: we said to combine them at the ptg yes but im trying to rember the details | |
| 20:10:02 | melwitt | I don't know. but that's what was being talked about in the room at the time, or so I thought | |
| 20:10:07 | jaypipes | sean-k-mooney, mriedem, dansmith, melwitt: to be clear, I'm not asking for anything at all :) I'm really just gonna do this custom filter thing as a stop-gap measure until we get on a more up to date version of nova. | |
| 20:10:29 | mriedem | agreed to add new initial_allocation ratio options with the default values from the ComputeNode object today, | |
| 20:10:39 | mriedem | change the existing *_allocation_ratio values from 0.0 defaults to None | |
| 20:10:51 | sean-k-mooney | jaypipes: oh i know. it just sounded like a useful thing. maybe for T | |
| 20:10:53 | mriedem | delete the code in the ComputeNode object so it's all config driven | |
| 20:11:00 | jaypipes | mriedem: yes, change them back to None from 0.0. | |
| 20:11:02 | mriedem | and then something something if config is set, that trumps the API | |
| 20:11:27 | jaypipes | mriedem: but I distinctly remember saying I was at the end of my proverbial rope with both of those specs and someone else would need to pick it up. | |
| 20:11:29 | mriedem | so mgagne's use case can use the API exclusively and CERN can use the config exclusively | |
| 20:11:29 | jaypipes | :) | |
| 20:11:39 | mriedem | jaypipes: yes yikun is happy to pick it up, | |
| 20:11:43 | jaypipes | cool, thx | |
| 20:11:46 | mriedem | but he doesn't understand what the direction is... | |
| 20:11:48 | sean-k-mooney | mriedem: yes i think we said if you want to be api drive in the config you set the value to none or remvoe it | |
| 20:11:54 | mriedem | which is why i'm trying to be a middleman here | |
| 20:12:12 | sean-k-mooney | mriedem: if you want to be config driven you set the config value and dont touch it from the api | |
| 20:12:43 | jaypipes | mriedem: you are wonderful middleware. | |
| 20:13:19 | mriedem | and yikun had a question, "How to address the upgrade case? If we already have a 0.0 cpu ratio in db, should we change it to 16.0 first? online migration?" | |
| 20:13:55 | sean-k-mooney | mriedem: spefically you set cpu_allocation_ratio=None initall_cpu_allocation_ratio=16.0 if you want to set a default for new node but manage the actuall value form api | |
| 20:14:04 | mriedem | would we change the ComputeNode.*_allocation_ratio to the config value on read if the value in the db is 0.0? | |
| 20:14:13 | sean-k-mooney | and set cpu_allocation_ratio=x if you want to manage via config | |
| 20:15:48 | sean-k-mooney | mriedem: does that make sense? | |
| 20:16:42 | mriedem | yes i get that, | |
| 20:16:52 | mriedem | the question is upgrades https://review.openstack.org/#/c/552105/5/specs/stein/approved/initial-allocation-ratios.rst@114 | |
| 20:18:29 | sean-k-mooney | mriedem: i guess on upgrade if 0.0 is set in the db that would also imply that the resouce provider allocation_ratio or what ever is 0. | |
| 20:18:37 | sean-k-mooney | 0.0 also correct | |
| 20:20:09 | sean-k-mooney | mriedem: if the resouce provide exists and we get 0.0 from the db but placemetn has another vaule i would assuem we shoudl keep the placement value but not sure if that case can happen today | |
| 20:21:10 | sean-k-mooney | the compute node will just override placement with the value it gets from the resouce tracker today in update provider tree right? | |
| 20:25:04 | mriedem | the allocation ratio in placement can't be 0.0 | |
| 20:25:09 | mriedem | it will literally shit itself | |
| 20:26:16 | cfriesen | I assume it | |
| 20:26:21 | cfriesen | it's a divide by zero thing? | |
| 20:26:30 | sean-k-mooney | mriedem: ok in that case the logic is simple. on upgrade if placement provider exits and db value is 0 set db value to placement value. if not placement provider exeits set db to intiall_* value and create provider as normal | |
| 20:26:31 | cfriesen | should placement check for that? | |
| 20:27:20 | openstackgerrit | iain MacDonnell proposed openstack/nova master: Handle online_data_migrations exceptions https://review.openstack.org/608091 | |
| 20:27:36 | sean-k-mooney | cfriesen: its not actully a device by 0 but we multiply the available capasity by 0 and see if its larger then what we requested | |
| 20:28:05 | sean-k-mooney | cfriesen: so placement will not have a math error but you wont be able to get allocation against that resouce provider ever | |
| 20:28:12 | cfriesen | ah, thanks | |
| 20:50:31 | efried | mriedem: We talked about this in the sched meeting yesterday. jaypipes said he was about ready to abandon those two specs. We also discussed the possibility of generic inventory yaml leading to a solution. | |
| 20:51:21 | mriedem | efried: i know i was there and said i'd reach out to yikun to pick up the specs, | |
| 20:51:26 | mriedem | but he's confused about the direction, as am i | |
| 20:51:28 | mriedem | hence questions | |
| 20:51:43 | mriedem | i'm going through https://review.openstack.org/#/c/552105/ again now, | |
| 20:51:50 | mriedem | some of that is outdated given https://github.com/openstack/nova/commit/2588af87c862cfd02d860f6b860381e907b279ff | |
| 21:07:52 | mriedem | alright i've dumped comments in https://review.openstack.org/#/c/552105/ - i think i could probably update the spec at this point to cover the upgrade impact | |