Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-08
20:54:16 dansmith on a multi-core system(i.e. anything from this decade) a single scheduler worker on the box makes no sense either
20:54:47 dansmith allowing workers= to be configured on the scheduler process is probably something we can/should do now
20:55:00 dansmith maybe default=1 for the moment, but get some feedback on that
20:55:03 openstackgerrit melanie witt proposed openstack/nova master: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/540258
20:55:32 cdent yeah, I tried to make it clear that I wasn't modeling a real world with what I was doing
20:55:52 dansmith no, I know, I'm just saying words
20:56:12 cdent it was pretty cool the way stuff mostly just worked
20:57:47 mriedem cdent: you could brush up https://review.openstack.org/#/c/159382/ and throw it into the env and see what difference it makes
20:58:34 mriedem heh i suppose i'd have to get jogo's -2 removed
20:58:35 cdent mriedem: I already did (my own version of) that. I had to go to eight workers to get things to not timeout
21:03:25 cdent mriedem: what I'm not clear on (yet) is what needs to be done to exercise any of the hoststate management concerns
21:03:46 mriedem in the scheduler?
21:03:54 mriedem using the default enabled filters is probably a good start
21:04:10 dansmith what does that mean?
21:04:15 dansmith hoststate management concerns
21:04:17 mriedem he disabled all of the filters
21:04:20 mriedem oh
21:04:31 dansmith oh, I missed that.. what is the scheduler doing then?
21:04:39 mriedem querying placement
21:04:43 mriedem fin
21:04:55 dansmith oh, yeah, I totally missed that part
21:05:16 dansmith that's not awesome, but maybe it's just the "8 people asking 1 person to do all the work" then
21:05:16 mriedem cdent: another thing to consider is that all 75 fake computes are going to be rpc broadcasting the shit out of the scheduler(s) for all instance resource tracker type actions, and every update_available_resource periodic
21:05:32 dansmith ...with no IO delays between them yeah
21:05:53 mriedem CONF.filter_scheduler.track_instance_changes is that guy
21:05:57 mriedem defaults to True
21:06:10 mriedem although,
21:06:14 cdent dansmith: the comments on https://review.openstack.org/#/c/159382/ and the email in http://thread.gmane.org/gmane.comp.cloud.openstack.devel/47882 has more about the host state stuff from the original stuff
21:06:29 mriedem in your setup it should be using superconductor mode,
21:06:36 mriedem where we disable track_instance_changes b/c compute can't reach scheduler
21:07:03 mriedem http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n859
21:07:04 dansmith cdent: okay but that is super old and unrelated anymore I think
21:07:22 dansmith cdent: all irrelevant after claims being in the scheduler
21:07:51 dansmith we're also pumping all the old RT stats into the database now, which we shouldn't need to do anymore because placement
21:07:52 cdent dansmith: I'm only reporting it to provide context to the earlier conversation with bauzas, who thought there might be still be issues, but I had been assuming it ought to irrelevant with claims in the scheduler
21:07:54 dansmith which just adds to the mq load
21:08:07 dansmith cdent: irrelevant afaik
21:08:31 mriedem you probably got lucky here
21:08:33 mriedem nova-manage cell_v2 discover_hosts openstack hypervisor list
21:08:36 mriedem swap those around
21:08:50 mriedem need i link to our favorite dokken video?
21:08:54 cdent mriedem: ?
21:09:01 mriedem https://www.youtube.com/watch?v=mBluR6cLxJ8
21:09:14 cdent not the video, why the order
21:09:22 cdent if I don't discover hosts, I've got no hypervisors
21:09:36 mriedem you should run openstack service list first, before discover_hosts
21:09:44 mriedem to make sure the compute node records are created
21:10:11 mriedem grep service-list in here https://docs.openstack.org/nova/latest/user/cells.html
21:10:27 mriedem since you're using the fake driver it's probably just fast enough
21:10:36 cdent wait_for_compute won't pass
21:10:43 mriedem oh right devstack does that...
21:10:43 cdent unless the service records are started
21:11:05 cdent I had to fix how wait_for_compute works in the devstack fix
21:13:02 cdent Perhaps tomorrow I'll try it with the filters enabled to see what happens
21:14:21 mriedem in your nova boot calls,
21:14:27 mriedem i see you used --min-count once
21:14:33 mriedem but not later in the loops, is there a reason for that?
21:14:46 mriedem like, boot 100 per request, in a loop of 10
21:15:57 cdent yeah, it's only when making lots of requests to /allocation_candidates that placement shows any load at all (and loading placement is the goal of all this)
21:16:10 bauzas dansmith: like I said to cdent, now there are no reasons to not support workers
21:16:16 dansmith this is all unsurprising though right?
21:16:25 dansmith placement should be mostly an http-to-sql adapter
21:16:31 dansmith bauzas: ack
21:16:31 cdent so many smal requests hits things way harder than anything
21:16:46 cdent dansmith: yes, and I've been saying that all along, but it's good to confirm it
21:16:51 dansmith cdent: ack, okay
21:16:53 cdent as an initial starting point of the exploration
21:17:31 cdent it supports the assertion I've been making for a while: Placement is not going to be a a bottleneck, at least not until we clean up some of the other stuff.
21:18:03 cdent mriedem: the first paragraphs after "playing with it" header talk about some of
21:18:05 cdent that
21:18:06 dansmith well, that was the assertion jaypipes made when he proofed all this early on, but yeah :)
21:18:48 mriedem i think this is still useful / good to see someone kicking things around,
21:18:55 ianw apropos nothing -- does configdrive work with aarch64? afaict it's just a libvirt level thing, so shouldn't have particular problems on a different architecture?
21:19:00 mriedem i've literally had some of this on my paper todo list on my desk for months
21:19:17 dansmith mriedem: oh yah, I'm not saying it's bad at all
21:19:35 dansmith mriedem: we learned a lot doing this for my instance list patch you were trying to blame :P
21:19:41 cdent dansmith: of course, but his tests were at just sql level, and the sql's changed a lot since then
21:19:50 dansmith cdent: totes
21:19:54 cdent so more playing the merrier
21:20:18 mriedem dansmith: heh and see how productive that was?
21:20:59 dansmith mriedem: again, not saying we shouldn't do this, I just thought this was going somewhere and thought the implication was "huh, well, placement isn't the bottleneck"
21:22:27 cdent it is going somewhere: more (concrete) info on how to arrange services and confirmation of stuff we mostly know (like scheduer needs workers and "apache is meh, use nginx") etc
21:22:34 efried cdent: s/duel/dual/
21:22:39 cdent thanks efried
21:22:46 efried unless it is duel, idk
21:25:15 cdent dansmith: what were you hoping for? Maybe I can try it now that I've leveled my brain up a bit.
21:25:19 efried cdent: s/message bug/message bus/ ?
21:25:26 cdent efried: heh, yup
21:25:31 efried I mean....
21:25:33 efried could be...
21:25:52 cdent plenty of message bugs
21:26:02 mriedem cdent: this might be of interest to you https://cloudblog.switch.ch/2017/08/28/starting-1000-instances-on-switchengines/
21:26:25 cdent mriedem: yeah saw that when it came out
21:29:20 efried cdent: "the placement api process are lost" => "...processes..."?
21:29:37 efried "of the much great effort"?
21:29:44 cdent you're hired
21:29:53 mriedem heh, better do this blog post via gerrit review first next time
21:30:03 efried At some point I'll contribute a substantive comment.
21:30:07 mriedem and then pick up the brain and skull bits from the wall
21:30:38 mriedem melwitt: on https://review.openstack.org/#/c/522161/ idk

Earlier   Later