Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-13
22:21:54 dansmith is that enough? it might do that by itself by chance
22:22:53 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in aggregates.py https://review.openstack.org/445230
22:23:18 openstackgerrit Anusha Unnam proposed openstack/nova master: POC: add descriptions to server tags https://review.openstack.org/434842
22:23:26 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in agents.py https://review.openstack.org/445208
22:23:47 mriedem dansmith: how?
22:23:53 mriedem w/o digging into details i guess i don't know
22:24:11 mriedem i'd also like to be able to force the scheduler to pick a host in the test that we've setup to fail
22:24:28 mriedem i'm not entirely sure how to do that in functional tests w/o stubbing something out
22:24:29 dansmith if you asked the scheduler to pick what would have been its second choice and it retries, it'd choose the other one without this fix
22:24:35 dansmith right exactly
22:24:45 mriedem yeah we have to make the test pick the hosts in the order we need for the recreate
22:24:49 mriedem that's the tricky part,
22:24:54 mriedem but doesn't seem too hard
22:25:15 dansmith don't we shuffle the results?
22:25:16 mriedem i'd think a weigher could be used
22:25:32 dansmith yeah maybe
22:25:44 mriedem that's scheduler black magic to me though
22:26:08 dansmith https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L124-L124
22:26:24 mriedem i don't want to make a big deal out of it, but this is a complicated piece of code and we clearly don't have it covered sufficiently in tests
22:26:32 dansmith you'd have to set the subset size to 1
22:26:48 melwitt it defaults to 1, so no shuffle by default
22:26:57 dansmith okay
22:27:12 mriedem are either of you interested in writing that functional test? :)
22:27:13 dansmith that seems like an unfortunate default for regular people
22:27:30 mriedem it's only random if the weights are equal isn't it?
22:28:06 dansmith wait,
22:28:14 dansmith oh, I see
22:28:42 dansmith huh, I never knew this picked a single host by default
22:28:44 melwitt mriedem: I can give it a go. I've written a weird retry test before https://github.com/openstack/nova/blob/master/nova/tests/functional/test_compute_mgr.py#L40
22:28:57 melwitt I agree with you that we need coverage on this stuff
22:30:44 melwitt I think maybe the subset thing is only a problem if you get parallel requests often (tm) that all pick the same host and retry. we did so we increased it from 1
22:31:19 dansmith sure, but that's a common complaint
22:31:32 dansmith that and running multiple schedulers choosing the same host
22:32:00 melwitt yeah. I guess no one thought of changing the default
22:32:13 dansmith I just can't think of a good reason to have the default be 1
22:32:34 dansmith if it's larger it doesn't hurt low-traffic clouds, but being =1 does hurt high traffic ones by default
22:32:40 mriedem isn't this exactly what johnthetubaguy was saying this morning, why people don't use the filter scheduler with packing?
22:32:46 dansmith mriedem: yes
22:32:57 mriedem because we don't randomize anything
22:33:00 dansmith mriedem: but remember I said "but we randomize the result"
22:33:04 melwitt mriedem today mentioned a good point in a review somewhere I think (?) that it might be since people want to stack instances rather than spread them usually
22:33:08 dansmith mriedem: and he said "right, but at a certain level it doesn't matter"
22:33:15 melwitt well, I typed that way too slowly
22:34:19 mriedem melwitt: yes because then you can't place large instances
22:34:29 dansmith mriedem: for RAX-level, I can imagine it's not enough, but for medium sizes I would expect this works fine
22:34:49 dansmith mriedem: but we're not doing anyone favors by defaulting that to 1, IMHO
22:34:49 melwitt yeah. well with subset = 1, that means no randomization. are you saying there's a different randomize somewhere?
22:35:20 dansmith melwitt: no, I'm saying even with that on, it's not enough to satisfy rax's need for parallelism
22:35:33 melwitt oh
22:36:48 dansmith if you set it to infinity it defeats all your weighting right? so there's probably some inflection point where it starts hurting you
22:37:52 melwitt yeah, I'm not sure how that randomize plays with the weighing part
22:38:25 dansmith I think we get back an ordered list from the weighers,
22:38:36 melwitt unless it just shuffles the weighed order of N subset
22:38:38 dansmith and that subset size randomizes the N best-weighed hosts
22:38:52 melwitt yeah
22:38:59 dansmith so if you set it to N=$n_computes, you get no weighing
22:39:03 melwitt right
22:39:11 dansmith which I hear is bad and stuff
22:39:15 melwitt hah
22:44:11 melwitt so I guess a good-ish reason to default to 1 is because the actual good default will vary so much from deployment to deployment
22:44:59 dansmith I dunno, if people have less than say five candidate machines they're probably running a pretty damned small cloud
22:45:17 dansmith shooting for the middle and tweaking up or down depending on your size seems reasonable,
22:45:32 dansmith but yeah, someone will probably be confused with a two-node test cloud if it's !=1
22:45:33 melwitt yeah, so I was thinking if you have 10 compute nodes and your subset defaulted to 2, then you could run into problems trying to schedule large instances?
22:45:34 dansmith just unfortunate
22:45:50 dansmith no the weighed hosts are all candidates
22:46:17 dansmith we could make that default to -1 and choose a subset that is like 5% of the node count or something
22:46:20 dansmith but anyway..
22:46:26 dansmith I just wonder how many people don't realize
22:46:28 dansmith might just be me
22:47:44 jroll that bit our ironic install, fwiw, because the candidate order is always the same as well
22:47:49 melwitt yeah, I was thinking if subset is 2 and you get a random choice of the two highest weights, then that will spread the instances slightly, no?
22:48:06 jroll we had a QE bot building periodically, chewed through the disks on the first few boxes in that list pretty quick
22:49:12 melwitt jroll: ah, that's an interesting point too
22:50:17 mriedem dansmith: melwitt: so here is the quota usage fix backport for ocata, which is a mix of both fixes from pike; i've tried explaining that in the commit message: https://review.openstack.org/#/c/445236/
22:50:32 mriedem let me know if that's terrible
22:54:02 melwitt mriedem: looks okay to me. commit message explanation is good
22:57:38 mriedem ok. backporting the other change from pike wouldn't help either b/c the functional test on the 1st change still fails without the tweaks i had to make here
22:57:50 mriedem so the only thing you could really do if backporting both changes is to squash them
23:00:41 dansmith yeah, I guess if you want pack first, then anything but =1 spreads them
23:00:49 dansmith like pack first above anything else
23:01:35 dansmith which might've been john's point since I think they do care about pack first, and might be why the randomization doesn't help him
23:04:15 melwitt ah, yeah
23:05:31 openstackgerrit Sujitha proposed openstack/nova master: Adding tags field to InstancePayload https://review.openstack.org/407228
23:06:24 openstackgerrit Sujitha proposed openstack/nova master: Change tags to default field in Instance object. https://review.openstack.org/415298
23:06:36 openstackgerrit Sujitha proposed openstack/nova master: Reduce calls to load_tags() to 0 https://review.openstack.org/435146
23:06:49 openstackgerrit Sujitha proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185
23:16:17 openstackgerrit Sujitha proposed openstack/nova master: Add description to policies in assisted_volume_snapshots.py https://review.openstack.org/445238
23:27:49 gmann mriedem: johnthetubaguy sdague alex_xu oomichi api-ref for server sort/filter keys - https://review.openstack.org/#/c/421760/
#openstack-nova - 2017-03-14
00:01:52 openstackgerrit STEW TY proposed openstack/nova master: Transform instance.resize_confirm notification https://review.openstack.org/390654
00:05:20 openstackgerrit Takashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration https://review.openstack.org/334732
00:11:07 openstackgerrit Anusha Unnam proposed openstack/nova master: Add description to policies to availability_zone https://review.openstack.org/440836
00:15:35 openstackgerrit Takashi NATSUME proposed openstack/nova-specs master: List/show all server migration types https://review.openstack.org/407237
00:15:40 openstackgerrit Takashi NATSUME proposed openstack/nova-specs master: List/show all server migration types https://review.openstack.org/407237
00:16:35 openstackgerrit Sivasathurappan Radhakrishnan proposed openstack/nova master: Allow live migrate paused instance when post copy is enabled https://review.openstack.org/444517
00:26:55 openstackgerrit Anusha Unnam proposed openstack/nova master: Add description to volumes_attachments.py https://review.openstack.org/445252
00:29:16 openstackgerrit STEW TY proposed openstack/nova master: Transform instance.resize_revert notification https://review.openstack.org/396811
00:54:55 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/443357
01:01:29 openstackgerrit STEW TY proposed openstack/nova master: Transform instance.rescue notifications https://review.openstack.org/385644
04:26:48 openstackgerrit Takashi NATSUME proposed openstack/nova master: Enable cold migration with target host(1/2) https://review.openstack.org/408955

Earlier   Later