| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-08 | |||
| 20:20:20 | dansmith | it's not compatible that way | |
| 20:20:46 | dansmith | so passing None there would break that older code if we were to interact with ti | |
| 20:20:53 | dansmith | mriedem: all the 4.x stuff goes away in rocky anyway | |
| 20:21:04 | dansmith | so yes, the same cleanup that removes all this would remove that anyway | |
| 20:21:13 | openstackgerrit | melanie witt proposed openstack/nova master: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/540258 | |
| 20:21:19 | mriedem | +2 | |
| 20:21:22 | mriedem | onto the cleanup patch | |
| 20:22:16 | dansmith | I havent' written the "drop 4.x" patch yet, but can get on that once we start to slow down | |
| 20:22:52 | mriedem | +2s all the way | |
| 20:22:56 | mriedem | time to find another core | |
| 20:23:00 | mriedem | they tend to hide | |
| 20:26:08 | openstackgerrit | Merged openstack/nova master: Workaround glanceclient bug when CONF.glance.api_servers not set https://review.openstack.org/541008 | |
| 20:26:19 | openstackgerrit | Merged openstack/nova master: Add a prelude release note for the 17.0.0 Queens GA https://review.openstack.org/541495 | |
| 20:38:57 | dansmith | gah pep8 | |
| 20:40:53 | openstackgerrit | Dan Smith proposed openstack/nova master: Clean up reservations in migrate_task call path https://review.openstack.org/542409 | |
| 20:42:46 | cdent | mriedem, dansmith: wrote up some of the scale stuff I was playing with last night (and today too): https://anticdent.org/placement-scale-fun.html (efried, jaypipes might be of interest to you too). Some conclusions/observations towards the end. | |
| 20:45:14 | mriedem | ok | |
| 20:45:16 | mriedem | thanks | |
| 20:46:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Encode libvirt domain XML in UTF-8 https://review.openstack.org/522161 | |
| 20:47:12 | openstackgerrit | melanie witt proposed openstack/nova master: Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442 | |
| 20:47:15 | dansmith | cdent: conductors do more than boot things, so conductors=schedulers isn't right | |
| 20:47:25 | dansmith | you'll always need more conductors than schedulers | |
| 20:47:53 | melwitt | gdi I forgot another thing I wanted to update | |
| 20:47:54 | cdent | dansmith: the point I was trying to make there was that _if_ you have eight conductors and you slam the api, then the conductor is going to slam the scheduler | |
| 20:47:59 | dansmith | you're measuring mostly boot load there which is why it seems like they're the same | |
| 20:48:05 | mriedem | dansmith: melwitt: need another +2 on https://review.openstack.org/#/c/522161/ - tempest regression test is passing with that, and it's clearly a regression in queens because the tempest test before i added the depends-on failed in master and passed on all the stable branches | |
| 20:48:30 | cdent | so at the starting point you're always going to overwhelm the scheduler processes | |
| 20:48:44 | cdent | with an otherwise initially idle conductor | |
| 20:48:50 | dansmith | cdent: also s/rcp timeout/rpc timeout/ | |
| 20:48:54 | cdent | thanks | |
| 20:49:26 | dansmith | cdent: also, the reasoning for the lack of workers= on the scheduler is that it used to be that >1 would just kill you because they'd stomp all over each other | |
| 20:49:34 | dansmith | with claims in the scheduling process, that stops being a problem | |
| 20:50:11 | cdent | yeah, bauzas and I were talking about that earlier today | |
| 20:50:26 | dansmith | ack | |
| 20:50:42 | dansmith | obviously the next thing would be to figure out what scheduler is doing the most of in there | |
| 20:51:13 | cdent | when I had 8 workers, the scheduler seemed to work just fine, it didn't show unreasonable suffering | |
| 20:52:28 | melwitt | mriedem: that's interesting, I had thought in the past we have encoded only the instance name in the XML to let non-ascii names work, not the entire XML. but I never knew if there's any harm in encoding the whole thing. must not be if that's what py3 does by default, right? | |
| 20:52:28 | dansmith | with one worker, all the load for scheduler work must be serialized, on a single core and in a single process | |
| 20:52:49 | dansmith | it will be penalized for that against anything else wanting to run | |
| 20:52:50 | dansmith | and on a system with everything running on it, it will get a smallish slice of the system, being CPU-bound | |
| 20:53:13 | openstackgerrit | melanie witt proposed openstack/nova master: Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442 | |
| 20:53:14 | dansmith | everything else having multiple workers gives those things way more bandwidth and time slice, relative to scheduler, yet everything funnels through scheduler | |
| 20:53:30 | dansmith | which isn't necessarily how it would be on a real system with things separated, depending on the layout | |
| 20:54:01 | dansmith | but | |
| 20:54:16 | dansmith | on a multi-core system(i.e. anything from this decade) a single scheduler worker on the box makes no sense either | |
| 20:54:47 | dansmith | allowing workers= to be configured on the scheduler process is probably something we can/should do now | |
| 20:55:00 | dansmith | maybe default=1 for the moment, but get some feedback on that | |
| 20:55:03 | openstackgerrit | melanie witt proposed openstack/nova master: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/540258 | |
| 20:55:32 | cdent | yeah, I tried to make it clear that I wasn't modeling a real world with what I was doing | |
| 20:55:52 | dansmith | no, I know, I'm just saying words | |
| 20:56:12 | cdent | it was pretty cool the way stuff mostly just worked | |
| 20:57:47 | mriedem | cdent: you could brush up https://review.openstack.org/#/c/159382/ and throw it into the env and see what difference it makes | |
| 20:58:34 | mriedem | heh i suppose i'd have to get jogo's -2 removed | |
| 20:58:35 | cdent | mriedem: I already did (my own version of) that. I had to go to eight workers to get things to not timeout | |
| 21:03:25 | cdent | mriedem: what I'm not clear on (yet) is what needs to be done to exercise any of the hoststate management concerns | |
| 21:03:46 | mriedem | in the scheduler? | |
| 21:03:54 | mriedem | using the default enabled filters is probably a good start | |
| 21:04:10 | dansmith | what does that mean? | |
| 21:04:15 | dansmith | hoststate management concerns | |
| 21:04:17 | mriedem | he disabled all of the filters | |
| 21:04:20 | mriedem | oh | |
| 21:04:31 | dansmith | oh, I missed that.. what is the scheduler doing then? | |
| 21:04:39 | mriedem | querying placement | |
| 21:04:43 | mriedem | fin | |
| 21:04:55 | dansmith | oh, yeah, I totally missed that part | |
| 21:05:16 | mriedem | cdent: another thing to consider is that all 75 fake computes are going to be rpc broadcasting the shit out of the scheduler(s) for all instance resource tracker type actions, and every update_available_resource periodic | |
| 21:05:16 | dansmith | that's not awesome, but maybe it's just the "8 people asking 1 person to do all the work" then | |
| 21:05:32 | dansmith | ...with no IO delays between them yeah | |
| 21:05:53 | mriedem | CONF.filter_scheduler.track_instance_changes is that guy | |
| 21:05:57 | mriedem | defaults to True | |
| 21:06:10 | mriedem | although, | |
| 21:06:14 | cdent | dansmith: the comments on https://review.openstack.org/#/c/159382/ and the email in http://thread.gmane.org/gmane.comp.cloud.openstack.devel/47882 has more about the host state stuff from the original stuff | |
| 21:06:29 | mriedem | in your setup it should be using superconductor mode, | |
| 21:06:36 | mriedem | where we disable track_instance_changes b/c compute can't reach scheduler | |
| 21:07:03 | mriedem | http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n859 | |
| 21:07:04 | dansmith | cdent: okay but that is super old and unrelated anymore I think | |
| 21:07:22 | dansmith | cdent: all irrelevant after claims being in the scheduler | |
| 21:07:51 | dansmith | we're also pumping all the old RT stats into the database now, which we shouldn't need to do anymore because placement | |
| 21:07:52 | cdent | dansmith: I'm only reporting it to provide context to the earlier conversation with bauzas, who thought there might be still be issues, but I had been assuming it ought to irrelevant with claims in the scheduler | |
| 21:07:54 | dansmith | which just adds to the mq load | |
| 21:08:07 | dansmith | cdent: irrelevant afaik | |
| 21:08:31 | mriedem | you probably got lucky here | |
| 21:08:33 | mriedem | nova-manage cell_v2 discover_hosts openstack hypervisor list | |
| 21:08:36 | mriedem | swap those around | |
| 21:08:50 | mriedem | need i link to our favorite dokken video? | |
| 21:08:54 | cdent | mriedem: ? | |
| 21:09:01 | mriedem | https://www.youtube.com/watch?v=mBluR6cLxJ8 | |
| 21:09:14 | cdent | not the video, why the order | |
| 21:09:22 | cdent | if I don't discover hosts, I've got no hypervisors | |
| 21:09:36 | mriedem | you should run openstack service list first, before discover_hosts | |
| 21:09:44 | mriedem | to make sure the compute node records are created | |
| 21:10:11 | mriedem | grep service-list in here https://docs.openstack.org/nova/latest/user/cells.html | |
| 21:10:27 | mriedem | since you're using the fake driver it's probably just fast enough | |
| 21:10:36 | cdent | wait_for_compute won't pass | |
| 21:10:43 | cdent | unless the service records are started | |
| 21:10:43 | mriedem | oh right devstack does that... | |
| 21:11:05 | cdent | I had to fix how wait_for_compute works in the devstack fix | |
| 21:13:02 | cdent | Perhaps tomorrow I'll try it with the filters enabled to see what happens | |
| 21:14:21 | mriedem | in your nova boot calls, | |
| 21:14:27 | mriedem | i see you used --min-count once | |