| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-08 | |||
| 21:04:17 | mriedem | he disabled all of the filters | |
| 21:04:20 | mriedem | oh | |
| 21:04:31 | dansmith | oh, I missed that.. what is the scheduler doing then? | |
| 21:04:39 | mriedem | querying placement | |
| 21:04:43 | mriedem | fin | |
| 21:04:55 | dansmith | oh, yeah, I totally missed that part | |
| 21:05:16 | dansmith | that's not awesome, but maybe it's just the "8 people asking 1 person to do all the work" then | |
| 21:05:16 | mriedem | cdent: another thing to consider is that all 75 fake computes are going to be rpc broadcasting the shit out of the scheduler(s) for all instance resource tracker type actions, and every update_available_resource periodic | |
| 21:05:32 | dansmith | ...with no IO delays between them yeah | |
| 21:05:53 | mriedem | CONF.filter_scheduler.track_instance_changes is that guy | |
| 21:05:57 | mriedem | defaults to True | |
| 21:06:10 | mriedem | although, | |
| 21:06:14 | cdent | dansmith: the comments on https://review.openstack.org/#/c/159382/ and the email in http://thread.gmane.org/gmane.comp.cloud.openstack.devel/47882 has more about the host state stuff from the original stuff | |
| 21:06:29 | mriedem | in your setup it should be using superconductor mode, | |
| 21:06:36 | mriedem | where we disable track_instance_changes b/c compute can't reach scheduler | |
| 21:07:03 | mriedem | http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/nova#n859 | |
| 21:07:04 | dansmith | cdent: okay but that is super old and unrelated anymore I think | |
| 21:07:22 | dansmith | cdent: all irrelevant after claims being in the scheduler | |
| 21:07:51 | dansmith | we're also pumping all the old RT stats into the database now, which we shouldn't need to do anymore because placement | |
| 21:07:52 | cdent | dansmith: I'm only reporting it to provide context to the earlier conversation with bauzas, who thought there might be still be issues, but I had been assuming it ought to irrelevant with claims in the scheduler | |
| 21:07:54 | dansmith | which just adds to the mq load | |
| 21:08:07 | dansmith | cdent: irrelevant afaik | |
| 21:08:31 | mriedem | you probably got lucky here | |
| 21:08:33 | mriedem | nova-manage cell_v2 discover_hosts openstack hypervisor list | |
| 21:08:36 | mriedem | swap those around | |
| 21:08:50 | mriedem | need i link to our favorite dokken video? | |
| 21:08:54 | cdent | mriedem: ? | |
| 21:09:01 | mriedem | https://www.youtube.com/watch?v=mBluR6cLxJ8 | |
| 21:09:14 | cdent | not the video, why the order | |
| 21:09:22 | cdent | if I don't discover hosts, I've got no hypervisors | |
| 21:09:36 | mriedem | you should run openstack service list first, before discover_hosts | |
| 21:09:44 | mriedem | to make sure the compute node records are created | |
| 21:10:11 | mriedem | grep service-list in here https://docs.openstack.org/nova/latest/user/cells.html | |
| 21:10:27 | mriedem | since you're using the fake driver it's probably just fast enough | |
| 21:10:36 | cdent | wait_for_compute won't pass | |
| 21:10:43 | mriedem | oh right devstack does that... | |
| 21:10:43 | cdent | unless the service records are started | |
| 21:11:05 | cdent | I had to fix how wait_for_compute works in the devstack fix | |
| 21:13:02 | cdent | Perhaps tomorrow I'll try it with the filters enabled to see what happens | |
| 21:14:21 | mriedem | in your nova boot calls, | |
| 21:14:27 | mriedem | i see you used --min-count once | |
| 21:14:33 | mriedem | but not later in the loops, is there a reason for that? | |
| 21:14:46 | mriedem | like, boot 100 per request, in a loop of 10 | |
| 21:15:57 | cdent | yeah, it's only when making lots of requests to /allocation_candidates that placement shows any load at all (and loading placement is the goal of all this) | |
| 21:16:10 | bauzas | dansmith: like I said to cdent, now there are no reasons to not support workers | |
| 21:16:16 | dansmith | this is all unsurprising though right? | |
| 21:16:25 | dansmith | placement should be mostly an http-to-sql adapter | |
| 21:16:31 | dansmith | bauzas: ack | |
| 21:16:31 | cdent | so many smal requests hits things way harder than anything | |
| 21:16:46 | cdent | dansmith: yes, and I've been saying that all along, but it's good to confirm it | |
| 21:16:51 | dansmith | cdent: ack, okay | |
| 21:16:53 | cdent | as an initial starting point of the exploration | |
| 21:17:31 | cdent | it supports the assertion I've been making for a while: Placement is not going to be a a bottleneck, at least not until we clean up some of the other stuff. | |
| 21:18:03 | cdent | mriedem: the first paragraphs after "playing with it" header talk about some of | |
| 21:18:05 | cdent | that | |
| 21:18:06 | dansmith | well, that was the assertion jaypipes made when he proofed all this early on, but yeah :) | |
| 21:18:48 | mriedem | i think this is still useful / good to see someone kicking things around, | |
| 21:18:55 | ianw | apropos nothing -- does configdrive work with aarch64? afaict it's just a libvirt level thing, so shouldn't have particular problems on a different architecture? | |
| 21:19:00 | mriedem | i've literally had some of this on my paper todo list on my desk for months | |
| 21:19:17 | dansmith | mriedem: oh yah, I'm not saying it's bad at all | |
| 21:19:35 | dansmith | mriedem: we learned a lot doing this for my instance list patch you were trying to blame :P | |
| 21:19:41 | cdent | dansmith: of course, but his tests were at just sql level, and the sql's changed a lot since then | |
| 21:19:50 | dansmith | cdent: totes | |
| 21:19:54 | cdent | so more playing the merrier | |
| 21:20:18 | mriedem | dansmith: heh and see how productive that was? | |
| 21:20:59 | dansmith | mriedem: again, not saying we shouldn't do this, I just thought this was going somewhere and thought the implication was "huh, well, placement isn't the bottleneck" | |
| 21:22:27 | cdent | it is going somewhere: more (concrete) info on how to arrange services and confirmation of stuff we mostly know (like scheduer needs workers and "apache is meh, use nginx") etc | |
| 21:22:34 | efried | cdent: s/duel/dual/ | |
| 21:22:39 | cdent | thanks efried | |
| 21:22:46 | efried | unless it is duel, idk | |
| 21:25:15 | cdent | dansmith: what were you hoping for? Maybe I can try it now that I've leveled my brain up a bit. | |
| 21:25:19 | efried | cdent: s/message bug/message bus/ ? | |
| 21:25:26 | cdent | efried: heh, yup | |
| 21:25:31 | efried | I mean.... | |
| 21:25:33 | efried | could be... | |
| 21:25:52 | cdent | plenty of message bugs | |
| 21:26:02 | mriedem | cdent: this might be of interest to you https://cloudblog.switch.ch/2017/08/28/starting-1000-instances-on-switchengines/ | |
| 21:26:25 | cdent | mriedem: yeah saw that when it came out | |
| 21:29:20 | efried | cdent: "the placement api process are lost" => "...processes..."? | |
| 21:29:37 | efried | "of the much great effort"? | |
| 21:29:44 | cdent | you're hired | |
| 21:29:53 | mriedem | heh, better do this blog post via gerrit review first next time | |
| 21:30:03 | efried | At some point I'll contribute a substantive comment. | |
| 21:30:07 | mriedem | and then pick up the brain and skull bits from the wall | |
| 21:30:38 | mriedem | melwitt: on https://review.openstack.org/#/c/522161/ idk | |
| 21:30:41 | mriedem | don't know what regressed it | |
| 21:30:46 | mriedem | but i know it's busted w/o this | |
| 21:31:07 | cdent | I suspect if it had gone through gerrit it would fall on the floor and I'd give up before it got through the nit factory. post hoc nit fixing ++ | |
| 21:31:14 | mriedem | and for anyone in china/japan/etc they'd probably like to be able to create a vm in queens | |
| 21:33:46 | mriedem | melwitt: ha https://review.openstack.org/#/c/541442/3/nova/virt/libvirt/driver.py@5404 | |
| 21:33:52 | mriedem | no one wins when it's block_device_info | |
| 21:34:12 | melwitt | you can say that again | |
| 21:34:19 | mriedem | not all entries are a list of dicts | |
| 21:34:24 | mriedem | 'root_device_name': instance.root_device_name, | |
| 21:34:25 | mriedem | AND | |
| 21:34:30 | melwitt | :| | |
| 21:34:32 | mriedem | the 'block_device_mapping' entry is not a list of dicts | |
| 21:34:35 | mriedem | no ma'am | |
| 21:34:44 | melwitt | FINE | |
| 21:34:47 | melwitt | :) | |