| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-09 | |||
| 17:31:20 | openstackgerrit | Dan Smith proposed openstack/nova master: Resource tracker compatibility with Ocata and Pike https://review.openstack.org/491012 | |
| 17:31:30 | dansmith | mriedem: jaypipes ^ | |
| 17:33:00 | mriedem | cfriesen: commented in your change | |
| 17:33:16 | mriedem | cfriesen: if you're using the caching scheduler, you still rely on those filters since the caching scheduler doesn't use placement | |
| 17:33:37 | dansmith | mriedem: did we ever push up that deprecation for those schedulers? | |
| 17:33:38 | mriedem | but the caching scheduler isn't the default filter driver, the filter_scheduler is, so i think it's fine for the default enabled filters to match the default scheduler driver | |
| 17:33:43 | mriedem | dansmith: i didn't see one | |
| 17:33:45 | dansmith | sounded like sdague was going to but I don't know that he did | |
| 17:34:04 | dansmith | mriedem: is it enough to put something in the config help text or do you want to log something? | |
| 17:34:48 | mriedem | i think we'd need both | |
| 17:35:09 | mnaser | cfriesen i'm onto something, with a totally empty hypervisor -- i see this -- AVAILABLE_SIBLINGS: [CoercedSet([1, 17]), CoercedSet([31, 15]), CoercedSet([23, 7]), CoercedSet([13, 29]), CoercedSet([27, 11]), CoercedSet([19, 3]), CoercedSet([9, 25]), CoercedSet([5, 21])] | |
| 17:35:26 | dansmith | mriedem: okay | |
| 17:35:27 | cdent | dansmith: was the resolution on the question of what to do when we get to the end of that set of conditionals where jay had “literally no idea what to do” to do nothing? | |
| 17:35:32 | mnaser | [1, 17] shouldn't be there, it should just be 17, because vcpu_pin_set is 2-31 | |
| 17:35:32 | mriedem | dansmith: plus reno of course | |
| 17:35:43 | mnaser | i dont think that codebase takes vcpu_pin_set into consideration | |
| 17:35:49 | dansmith | cdent: yes | |
| 17:35:55 | cdent | ✔ | |
| 17:36:57 | mriedem | gibi: ok so i guess it is a regression in pike then so i'll mark it pike-rc-potential | |
| 17:37:20 | cfriesen | mnaser: you had changed vcpu_pin_set to 0,16, no? | |
| 17:38:00 | cfriesen | mriedem: okay, I'll respin | |
| 17:38:42 | mnaser | cfriesen i switched back. by setting it to 0,16 => i end up being unable to spin the last instance because all cores + memory are taken on numanode #2 | |
| 17:38:43 | mnaser | s/#2/#1/ | |
| 17:38:49 | mnaser | and numanode 0 has 4gb memory left unused | |
| 17:39:00 | mnaser | by using 0,16 -- i have a single thread in both cores that can use the leftover 4gb | |
| 17:39:28 | mnaser | using 0,16 effectively means 14 threads for 1 numanode and 16 threads for the other. vs 15/15 | |
| 17:40:11 | cfriesen | mnaser: please double-check that you restarted nova-compute or rebooted after changing vcpu_pin_set. I agree that you shouldn't see 1 in the AVAILABLE_SIBLINGS list. | |
| 17:40:13 | mriedem | gdi, people, tox -e fast8 before git review | |
| 17:40:14 | mriedem | https://review.openstack.org/#/c/488510/ | |
| 17:41:03 | mnaser | DEBUG oslo_service.service [req-39a92c29-83e3-4cb0-9f78-29b97d61147d - - - - -] vcpu_pin_set = 2-31 log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2622 in the logs, /proc/cmdline contains isolcpus=2-31 | |
| 17:44:01 | mnaser | http://paste.openstack.org/raw/617965/ this is what is generated from `resources` in claims.py | |
| 17:44:42 | mnaser | i dont even see it in the siblings there, i wonder if it is in the process of the transformation to the DB object | |
| 17:47:15 | cfriesen | mriedem: what do you think of this wording? http://paste.openstack.org/show/617966/ | |
| 17:48:40 | mriedem | "If you have ``scheduler_driver`` set to ``caching_scheduler``, you should probably add these entries back in along with CoreFilter." | |
| 17:48:51 | mriedem | issue the first: scheduler_driver is the old option name | |
| 17:48:57 | mriedem | it's now [scheduler]/driver | |
| 17:49:14 | cfriesen | whoops, was looking at wrong window. :) | |
| 17:49:42 | mriedem | issue the 2nd: let's not tell them what to do if they're not using the filter scheduler, but just point out that non-FilterScheduler drivers, like the CachingScheduler, may want to have these filters enabled | |
| 17:49:49 | cfriesen | okay | |
| 17:50:09 | mriedem | if they already have that option defined in their nova.conf, then they don't have to 'add' anything back in, as it's already there, and we're just changing the defaults | |
| 17:50:26 | cfriesen | mriedem: the issue I'm worried about is if they were relying on the defaults | |
| 17:50:48 | mriedem | sure, and that's why you point out that non-FilterScheduler drivers, like the CachingScheduler, may want to have these enabled | |
| 17:50:58 | cfriesen | fair enough | |
| 17:52:16 | mriedem | isn't there also a case that you don't want the DiskFilter enabled - if you're using shared storage? | |
| 17:52:29 | mriedem | looks like we never documented that in the description for the DiskFilter | |
| 17:53:07 | mriedem | i'm not saying you need to for this change, but just thinking of the operator that then starts looking at if they need to enable these or not - will they have enough information to make that decision | |
| 17:53:49 | cdent | thanks mriedem, by being assigned the fix for that bug, I should receive a gloriously huge bug bounty from … someone | |
| 17:54:02 | mriedem | tjones | |
| 17:54:11 | mriedem | has the bug bounty kitty | |
| 17:54:27 | openstackgerrit | Dan Smith proposed openstack/nova master: Mark Chance and Caching schedulers as deprecated https://review.openstack.org/492210 | |
| 17:54:31 | dansmith | mriedem: ^ | |
| 17:57:07 | openstackgerrit | Chris Friesen proposed openstack/nova master: Remove ram/disk sched filters from default list https://review.openstack.org/491854 | |
| 17:57:28 | mriedem | dansmith: lgtm | |
| 17:57:48 | mnaser | cfriesen my apologies, looks like wrong config.. but i think i've narrowed it down to this: AVAILABLE_SIBLINGS: [CoercedSet([8, 24]), CoercedSet([2, 18]), CoercedSet([10, 26]), CoercedSet([28, 12]), CoercedSet([22, 6]), CoercedSet([30, 14]), CoercedSet([20, 4])] .. technically there should be more set in there with CoercedSet([16]) -- because the 0 is reserved | |
| 17:58:01 | mnaser | it's removed the entire set rather than the single one | |
| 17:58:09 | cfriesen | dansmith: is the performance of FilterScheduler with Placement comparable to CachingScheduler? | |
| 17:58:12 | mnaser | (this might be obvious to you but i'm just figuring it out, heh) | |
| 17:58:58 | dansmith | cfriesen: it should end up being much better | |
| 17:59:24 | cfriesen | mnaser: if you've got 0/1 reserved then I think arguably there should be a set with just 16 and another set with just 17 | |
| 18:00:03 | cfriesen | it should be noted though that the performance of 16/17 is going to be variable, depending on what load is on 0/1 | |
| 18:00:40 | cfriesen | dansmith: better? isn't CachingScheduler in memory for the most part? | |
| 18:00:42 | mnaser | of course, that's a given | |
| 18:01:38 | dansmith | cfriesen: we've been discussing this in this channel for like I year, so I hope I don't have to summarize all of it, but if you count the ability to multi-thread the scheduler with placement, and zero reschedules, yes, it should be much better | |
| 18:01:54 | dansmith | s/I/a | |
| 18:02:02 | mriedem | roman numeral I | |
| 18:02:05 | mriedem | you meant | |
| 18:02:11 | dansmith | heh, yeah | |
| 18:02:11 | sdague | dansmith: I did not push the deprecation of the other schedulers | |
| 18:02:17 | sdague | dansmith: if that's a patch that is needed, I could do it this afternoon | |
| 18:02:18 | dansmith | sdague: I just did | |
| 18:02:23 | mriedem | https://review.openstack.org/#/c/492210/1 | |
| 18:02:25 | sdague | dansmith: ok, coolio | |
| 18:02:35 | cfriesen | dansmith: ah, you mean overall scheduler load. got it. (I was thinking time to perform a single schedule operation.) | |
| 18:02:42 | mnaser | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5632-L5633 -- bingo.. now to see why the decision of filtering out singles is there | |
| 18:02:51 | mriedem | cfriesen: it's also that | |
| 18:03:00 | mriedem | time to schedule should be faster when you don't have reschedules | |
| 18:03:02 | mriedem | b/c of bad decisions | |
| 18:03:41 | mriedem | cfriesen: your team seems to do some performance testing, it would be cool if your guy(s) could benchmark and compare the two now in pike | |
| 18:04:00 | mriedem | like single caching scheduler vs multiple filter schedulers | |
| 18:04:00 | dansmith | if you really care about the milliseconds requried to place something, you should be able to do better with a bunch of filterschedulers than one in-memory caching scheduler I would think | |
| 18:04:08 | cdent | cfriesen: it may be the case, but I’m not sure that anyone has proven it yet, that single schedule operations will speed up because placement will help limit the initial set of available candidates | |
| 18:04:51 | dansmith | cdent: cachingscheduler only refreshes the list when it runs out of spots I think, so it varies, but it also is using very outdated information which means fast but poor decisions | |
| 18:05:09 | cfriesen | I'd expect the FilterScheduler to speed up due to Placement. Wasn't sure that it'd be faster than CachingScheduler. Anyways, we've been using FilterScheduler since we care about accurate resource tracking. | |
| 18:05:16 | mriedem | caching scheduler is also on a periodic | |
| 18:05:45 | cfriesen | (We modified stuff like resize and migration to update the resource tracking immediately rather than wait for the resource audit.) | |
| 18:05:49 | dansmith | caching scheduler is filter scheduler, with cached host list data | |
| 18:06:18 | mriedem | dansmith: so this resize/rt series has a pep8 failure in the bottom change, | |
| 18:06:31 | mriedem | dansmith: but i guess hold off on fixing that so i can go through them now | |
| 18:06:33 | mriedem | the last 2 i mean | |
| 18:06:41 | dansmith | okay | |
| 18:06:46 | mriedem | dansmith: it's T-~3 hours to 0 dan time yeah? | |
| 18:06:53 | dansmith | check queue is 7h long | |
| 18:07:26 | dansmith | mriedem: I think you mean T-3 hours until free-of-dan time | |
| 18:07:27 | dansmith | but yeah | |
| 18:07:55 | dansmith | I assume everyone brought liquor and cake to work today for the post-2pm PDT party | |
| 18:08:14 | mriedem | brought? i think it's just generally available | |
| 18:08:47 | dansmith | I was trying to act like it was the 90s | |
| 18:13:40 | mnaser | cfriesen thanks for all your feedback, i think the best solution for now is going to be use 0,16 to have predictable performance for the host and assign hugepages with a slight bias towards node1 because it has 2 more cores so things can get properly packed | |
| 18:23:37 | sdague | dansmith: yeh, we're down to 800 nodes in ci | |
| 18:23:54 | sdague | which is unfortunately not sufficient | |