| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-08 | |||
| 20:52:28 | dansmith | with one worker, all the load for scheduler work must be serialized, on a single core and in a single process | |
| 20:52:28 | melwitt | mriedem: that's interesting, I had thought in the past we have encoded only the instance name in the XML to let non-ascii names work, not the entire XML. but I never knew if there's any harm in encoding the whole thing. must not be if that's what py3 does by default, right? | |
| 20:52:49 | dansmith | it will be penalized for that against anything else wanting to run | |
| 20:52:50 | dansmith | and on a system with everything running on it, it will get a smallish slice of the system, being CPU-bound | |
| 20:53:13 | openstackgerrit | melanie witt proposed openstack/nova master: Refine waiting for vif plug events during _hard_reboot https://review.openstack.org/541442 | |
| 20:53:14 | dansmith | everything else having multiple workers gives those things way more bandwidth and time slice, relative to scheduler, yet everything funnels through scheduler | |
| 20:53:30 | dansmith | which isn't necessarily how it would be on a real system with things separated, depending on the layout | |
| 20:54:01 | dansmith | but | |
| 20:54:16 | dansmith | on a multi-core system(i.e. anything from this decade) a single scheduler worker on the box makes no sense either | |
| 20:54:47 | dansmith | allowing workers= to be configured on the scheduler process is probably something we can/should do now | |
| 20:55:00 | dansmith | maybe default=1 for the moment, but get some feedback on that | |
| 20:55:03 | openstackgerrit | melanie witt proposed openstack/nova master: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/540258 | |
| 20:55:32 | cdent | yeah, I tried to make it clear that I wasn't modeling a real world with what I was doing | |
| 20:55:52 | dansmith | no, I know, I'm just saying words | |
| 20:56:12 | cdent | it was pretty cool the way stuff mostly just worked | |
| 20:57:47 | mriedem | cdent: you could brush up https://review.openstack.org/#/c/159382/ and throw it into the env and see what difference it makes | |
| 20:58:34 | mriedem | heh i suppose i'd have to get jogo's -2 removed | |
| 20:58:35 | cdent | mriedem: I already did (my own version of) that. I had to go to eight workers to get things to not timeout | |
| 21:03:25 | cdent | mriedem: what I'm not clear on (yet) is what needs to be done to exercise any of the hoststate management concerns | |
| 21:03:46 | mriedem | in the scheduler? | |
| 21:03:54 | mriedem | using the default enabled filters is probably a good start | |
| 21:04:10 | dansmith | what does that mean? | |
| 21:04:15 | dansmith | hoststate management concerns | |
| 21:04:17 | mriedem | he disabled all of the filters | |
| 21:04:20 | mriedem | oh | |
| 21:04:31 | dansmith | oh, I missed that.. what is the scheduler doing then? | |
| 21:04:39 | mriedem | querying placement | |
| 21:04:43 | mriedem | fin | |
| 21:04:55 | dansmith | oh, yeah, I totally missed that part | |
| 21:05:16 | dansmith | that's not awesome, but maybe it's just the "8 people asking 1 person to do all the work" then | |
| 21:05:16 | mriedem | cdent: another thing to consider is that all 75 fake computes are going to be rpc broadcasting the shit out of the scheduler(s) for all instance resource tracker type actions, and every update_available_resource periodic | |
| 21:05:32 | dansmith | ...with no IO delays between them yeah | |
| 21:05:53 | mriedem | CONF.filter_scheduler.track_instance_changes is that guy | |
| 21:05:57 | mriedem | defaults to True | |
| 21:06:10 | mriedem | although, | |
| 21:06:14 | cdent | dansmith: the comments on https://review.openstack.org/#/c/159382/ and the email in http://thread.gmane.org/gmane.comp.cloud.openstack.devel/47882 has more about the host state stuff from the original stuff | |
| 21:06:29 | mriedem | in your setup it should be using superconductor mode, | |
| 21:06:36 | mriedem | where we disable track_instance_changes b/c compute can't reach scheduler | |
| 21:07:03 | mriedem | http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n859 | |
| 21:07:04 | dansmith | cdent: okay but that is super old and unrelated anymore I think | |
| 21:07:22 | dansmith | cdent: all irrelevant after claims being in the scheduler | |
| 21:07:51 | dansmith | we're also pumping all the old RT stats into the database now, which we shouldn't need to do anymore because placement | |
| 21:07:52 | cdent | dansmith: I'm only reporting it to provide context to the earlier conversation with bauzas, who thought there might be still be issues, but I had been assuming it ought to irrelevant with claims in the scheduler | |
| 21:07:54 | dansmith | which just adds to the mq load | |
| 21:08:07 | dansmith | cdent: irrelevant afaik | |
| 21:08:31 | mriedem | you probably got lucky here | |
| 21:08:33 | mriedem | nova-manage cell_v2 discover_hosts openstack hypervisor list | |
| 21:08:36 | mriedem | swap those around | |
| 21:08:50 | mriedem | need i link to our favorite dokken video? | |
| 21:08:54 | cdent | mriedem: ? | |
| 21:09:01 | mriedem | https://www.youtube.com/watch?v=mBluR6cLxJ8 | |
| 21:09:14 | cdent | not the video, why the order | |
| 21:09:22 | cdent | if I don't discover hosts, I've got no hypervisors | |
| 21:09:36 | mriedem | you should run openstack service list first, before discover_hosts | |
| 21:09:44 | mriedem | to make sure the compute node records are created | |
| 21:10:11 | mriedem | grep service-list in here https://docs.openstack.org/nova/latest/user/cells.html | |
| 21:10:27 | mriedem | since you're using the fake driver it's probably just fast enough | |
| 21:10:36 | cdent | wait_for_compute won't pass | |
| 21:10:43 | mriedem | oh right devstack does that... | |
| 21:10:43 | cdent | unless the service records are started | |
| 21:11:05 | cdent | I had to fix how wait_for_compute works in the devstack fix | |
| 21:13:02 | cdent | Perhaps tomorrow I'll try it with the filters enabled to see what happens | |
| 21:14:21 | mriedem | in your nova boot calls, | |
| 21:14:27 | mriedem | i see you used --min-count once | |
| 21:14:33 | mriedem | but not later in the loops, is there a reason for that? | |
| 21:14:46 | mriedem | like, boot 100 per request, in a loop of 10 | |
| 21:15:57 | cdent | yeah, it's only when making lots of requests to /allocation_candidates that placement shows any load at all (and loading placement is the goal of all this) | |
| 21:16:10 | bauzas | dansmith: like I said to cdent, now there are no reasons to not support workers | |
| 21:16:16 | dansmith | this is all unsurprising though right? | |
| 21:16:25 | dansmith | placement should be mostly an http-to-sql adapter | |
| 21:16:31 | dansmith | bauzas: ack | |
| 21:16:31 | cdent | so many smal requests hits things way harder than anything | |
| 21:16:46 | cdent | dansmith: yes, and I've been saying that all along, but it's good to confirm it | |
| 21:16:51 | dansmith | cdent: ack, okay | |
| 21:16:53 | cdent | as an initial starting point of the exploration | |
| 21:17:31 | cdent | it supports the assertion I've been making for a while: Placement is not going to be a a bottleneck, at least not until we clean up some of the other stuff. | |
| 21:18:03 | cdent | mriedem: the first paragraphs after "playing with it" header talk about some of | |
| 21:18:05 | cdent | that | |
| 21:18:06 | dansmith | well, that was the assertion jaypipes made when he proofed all this early on, but yeah :) | |
| 21:18:48 | mriedem | i think this is still useful / good to see someone kicking things around, | |
| 21:18:55 | ianw | apropos nothing -- does configdrive work with aarch64? afaict it's just a libvirt level thing, so shouldn't have particular problems on a different architecture? | |
| 21:19:00 | mriedem | i've literally had some of this on my paper todo list on my desk for months | |
| 21:19:17 | dansmith | mriedem: oh yah, I'm not saying it's bad at all | |
| 21:19:35 | dansmith | mriedem: we learned a lot doing this for my instance list patch you were trying to blame :P | |
| 21:19:41 | cdent | dansmith: of course, but his tests were at just sql level, and the sql's changed a lot since then | |
| 21:19:50 | dansmith | cdent: totes | |
| 21:19:54 | cdent | so more playing the merrier | |
| 21:20:18 | mriedem | dansmith: heh and see how productive that was? | |
| 21:20:59 | dansmith | mriedem: again, not saying we shouldn't do this, I just thought this was going somewhere and thought the implication was "huh, well, placement isn't the bottleneck" | |
| 21:22:27 | cdent | it is going somewhere: more (concrete) info on how to arrange services and confirmation of stuff we mostly know (like scheduer needs workers and "apache is meh, use nginx") etc | |
| 21:22:34 | efried | cdent: s/duel/dual/ | |
| 21:22:39 | cdent | thanks efried | |
| 21:22:46 | efried | unless it is duel, idk | |
| 21:25:15 | cdent | dansmith: what were you hoping for? Maybe I can try it now that I've leveled my brain up a bit. | |
| 21:25:19 | efried | cdent: s/message bug/message bus/ ? | |
| 21:25:26 | cdent | efried: heh, yup | |
| 21:25:31 | efried | I mean.... | |
| 21:25:33 | efried | could be... | |
| 21:25:52 | cdent | plenty of message bugs | |
| 21:26:02 | mriedem | cdent: this might be of interest to you https://cloudblog.switch.ch/2017/08/28/starting-1000-instances-on-switchengines/ | |