| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-06 | |||
| 10:50:58 | lyarwood | mdbooth: I was, but I've dropped the patch above now | |
| 10:51:06 | lyarwood | mdbooth: so I either rebase this on your stuff or kill it | |
| 10:51:15 | lyarwood | mdbooth: and it works, I think. | |
| 11:28:54 | openstackgerrit | Merged openstack/nova master: Clean up cpu_shared_set config docs https://review.openstack.org/614864 | |
| 11:29:00 | openstackgerrit | Merged openstack/nova master: Delete NeutronLinuxBridgeInterfaceDriver https://review.openstack.org/616995 | |
| 11:53:02 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810 | |
| 11:53:03 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells https://review.openstack.org/591657 | |
| 12:04:41 | openstackgerrit | Chris Dent proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972 | |
| 12:25:24 | openstackgerrit | Guo Jingyu proposed openstack/nova master: Add rfb.VNC support for novncproxy https://review.openstack.org/622336 | |
| 12:33:32 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Handle tags in _bury_in_cell0 https://review.openstack.org/621856 | |
| 12:58:49 | openstack | Launchpad bug 1805989 in OpenStack Compute (nova) "Weight policy to stack/spread instances and "max_placement_results"" [Undecided,New] | |
| 12:58:49 | mriedem | tssurya: belmoreira: i'm not sure what i'm missing here https://bugs.launchpad.net/nova/+bug/1805989 | |
| 13:03:17 | tssurya | mriedem: I guess from our prespective since we don't use any scheduler fitlers (except the Compute Filter), as opposed to a scenario where placement would return all results as in all possible available resources on which the weights would be applied versus what happens now when we do this only the max_placement_results number of available resources (hosts) | |
| 13:03:46 | tssurya | would effect the way packing/spreading is done | |
| 13:03:52 | mriedem | that's working as designed.... | |
| 13:04:04 | mriedem | max_placement_results is just a front-end filter | |
| 13:04:23 | sean-k-mooney | tssurya: you dont use any filetre so you dont use sriov/pci passthrough/cpu pinning/hugepages or numa in your cloud | |
| 13:04:26 | mriedem | like i said in the bug, even before placement, if you have 1k hosts, if the filters narrow that down to 10, then only 10 get weighed | |
| 13:04:33 | mriedem | sean-k-mooney: no they don't, | |
| 13:04:37 | mriedem | oh well, | |
| 13:04:39 | mriedem | nvm | |
| 13:04:55 | mriedem | b/c of cells v1 there were several things they couldn't support before, like affinity | |
| 13:05:04 | mriedem | idk about pci/numa | |
| 13:05:15 | openstack | Launchpad bug 1805984 in OpenStack Compute (nova) "Placement is not aware of disable compute nodes" [Wishlist,Triaged] | |
| 13:05:15 | sean-k-mooney | this seams kindof like a duplicate of https://bugs.launchpad.net/nova/+bug/1805984 | |
| 13:05:18 | tssurya | sean-k-mooney: we use pci passthrough, but not that much (just for one cell) | |
| 13:05:48 | mriedem | sean-k-mooney: that is definitely a different problem | |
| 13:06:02 | mriedem | that bug is that placement is returning only disabled computes | |
| 13:06:11 | mriedem | because cern sets max_placement_results very low | |
| 13:06:14 | sean-k-mooney | yes | |
| 13:06:21 | mriedem | this other one is different | |
| 13:06:35 | mriedem | placement has no relationship to weighers in nova-scheduler | |
| 13:06:47 | sean-k-mooney | it is a different problem but they are both because the limiting is done in placmeent not after teh filters | |
| 13:07:06 | mriedem | for the weigher bug it doesn't matter | |
| 13:07:13 | mriedem | filters are before weighers, | |
| 13:07:21 | mriedem | so if placement filtered out the hosts or the scheduler filters did, it doesn't matter | |
| 13:07:33 | mriedem | the weighers are only going to weigh what they get | |
| 13:07:54 | mriedem | so they will spread/pack across whatever comes through the filters | |
| 13:07:55 | sean-k-mooney | it maters if you have lost of disabled host in the limited set of result you got form plamcnet | |
| 13:08:40 | sean-k-mooney | it will skew the weighing by reducing the set of results | |
| 13:08:45 | tssurya | mriedem: exactly for us when not using "max_placement_results" filters would give us almost all the hosts from a cell | |
| 13:09:15 | openstack | bug 1805989 in OpenStack Compute (nova) "Weight policy to stack/spread instances and "max_placement_results"" [Undecided,New] https://launchpad.net/bugs/1805989 | |
| 13:09:15 | mriedem | tssurya: yeah, so IMO bug 1805989 is not really a bug, it's working as designed | |
| 13:09:20 | tssurya | but yea probably this iss not a bug its design thing | |
| 13:09:29 | openstack | bug 1805984 in OpenStack Compute (nova) "Placement is not aware of disable compute nodes" [Wishlist,Triaged] https://launchpad.net/bugs/1805984 | |
| 13:09:29 | mriedem | bug 1805984 is definitely a problem for which we have solutions | |
| 13:09:38 | sean-k-mooney | that said it is working as desinged as mriedem said. that desing my not be desireable and we might want to consider how to change it | |
| 13:09:58 | gibi | mriedem: I saw your comment on the notification deprecation patch. I have to organize my thoughts formulate an opinion | |
| 13:10:15 | mriedem | what we need to do is get CERN to the point that they don't have to workaround perf issues by setting max_placement_results to 10 | |
| 13:10:18 | mriedem | when they have 14K hosts | |
| 13:10:27 | mriedem | gibi: heh ok :) | |
| 13:10:29 | mriedem | no rush | |
| 13:10:35 | gibi | mriedem: yeah, I will take my time | |
| 13:10:38 | gibi | :) | |
| 13:10:46 | mriedem | gibi: it's a bit depressing huh? | |
| 13:10:48 | tssurya | mriedem: sure its a design thing, we just laid it out in case there were ideas | |
| 13:10:57 | tssurya | mriedem: yea perf stuff is still going on | |
| 13:11:20 | tssurya | hopefully we won't need to set "max_placement_results" to a low number then | |
| 13:11:22 | gibi | mriedem: I understand that the original goals of that work might not be applicable in the today situaion on OpenStack | |
| 13:11:37 | sean-k-mooney | tssurya: the placment randomisation was how we wanted peole to enable spreading behavior | |
| 13:12:55 | sean-k-mooney | so if you had a 1000 hosts that could fit the request and you requested 50 then with the randomisation enbled you would get a random 50 out of that 1000 for the filters and weigher to select form | |
| 13:12:55 | mriedem | tssurya: ok i marked the bug as invalid (working as designed) | |
| 13:14:00 | sean-k-mooney | tssurya: do you rember what you set currently and what is that value as a propotion of your average cell size | |
| 13:14:26 | mriedem | average cell size is 200 at cern | |
| 13:14:32 | mriedem | i think they set max_placement_results to 10 | |
| 13:14:32 | tssurya | sean-k-mooney: 10 versus 800 in normal scenario | |
| 13:14:53 | mriedem | 800? i thought it was 200 on average | |
| 13:15:10 | mriedem | with about 72-74 cells? | |
| 13:15:27 | tssurya | mriedem: yea 200 for special project to cell mappings, on an average for normal users we set aside 5 to 7 default cells | |
| 13:15:43 | tssurya | each cell having 200 | |
| 13:15:59 | mriedem | ok | |
| 13:16:29 | openstack | Launchpad bug 1737465 in OpenStack Compute (nova) "[cellv2] the performance issue of cellv2 when creating 500 instances concurrently" [Medium,Confirmed] | |
| 13:16:29 | mriedem | as far as i know, https://bugs.launchpad.net/nova/+bug/1737465 is still the biggest perf issue | |
| 13:16:35 | sean-k-mooney | ok so could you increase the limit to say 10% of the host that tenat can be expect to selct form | |
| 13:16:57 | sean-k-mooney | e.g. 80 in this case? | |
| 13:17:15 | mriedem | also, as far as i know, cern is not yet grouping cells via host aggregate | |
| 13:17:24 | mriedem | or are you? | |
| 13:17:34 | tssurya | we model cells as aggregates | |
| 13:17:48 | tssurya | as in every cell is an aggregate | |
| 13:17:48 | mriedem | because of https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#tenant-isolation-with-placement right> | |
| 13:17:49 | mriedem | ? | |
| 13:17:52 | mriedem | ok | |
| 13:18:04 | tssurya | yea because of the pre-filter | |
| 13:18:07 | mriedem | so, really, | |
| 13:18:22 | jaypipes | mriedem: wait, there's a bug in shelve code?! | |
| 13:18:28 | sean-k-mooney | tssurya: so you then use the teanay affinity filter to map tenants to thos aggreates and therefor to cells? | |
| 13:18:30 | mriedem | jaypipes: working as designed | |
| 13:18:47 | tssurya | sean-k-mooney: yea | |
| 13:18:48 | mriedem | tssurya: if the pre-filter is working, then the max_placement_results probably doesn't need to be so low... | |
| 13:19:01 | jaypipes | mriedem: I only have time to argue with cfriesen today, so I'll wait til he's online. :P | |
| 13:19:12 | sean-k-mooney | mriedem: that is what i was wondering too | |
| 13:19:16 | tssurya | mriedem: surely it need not be as low as 10, but even with it being 10 we have 5 to 8 seconds scheduling time | |
| 13:19:19 | mriedem | because if my project is mapped to a cell with ~200 hosts, then filtering on 200 hosts shouldn't be so bad | |
| 13:19:54 | tssurya | the time it would tkae to gather all those host states , well we are still trying to check iut why its 5 to 8 seconds | |
| 13:20:06 | tssurya | out* | |
| 13:20:17 | sean-k-mooney | tssurya: when you say schduiling are you refering to just the time it take for the filter schduler or the time the vm is in that sate | |
| 13:20:34 | mriedem | tssurya: do you know if this is set on the computes and the scheduler nodes? https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.track_instance_changes | |
| 13:20:44 | tssurya | time for the whole boot until select destinations is done fully | |
| 13:20:48 | jaypipes | mriedem: I think it's pretty obvious I wouldn't support adding disabled flags to provider records, yes? | |
| 13:21:01 | mriedem | jaypipes: yes, i could find you my reply to that | |