| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-02 | |||
| 16:51:18 | mriedem | question inline | |
| 16:53:24 | jroll | ah thanks | |
| 16:56:42 | mriedem | dansmith: so for this CLI to fix allocations in placement for instances scheduled using the CachingScheduler, i'm trying to think where that should live, since none of the nova-manage subcommands really fit that; we could do something like, | |
| 16:56:47 | mriedem | a) nova-manage placement ... | |
| 16:56:54 | mriedem | b) placement-manage ... | |
| 16:56:57 | mriedem | c) throw it in osc-placement | |
| 16:57:09 | mriedem | although (c) isn't really meant for something like this | |
| 16:57:20 | dansmith | yeah, it's a pretty oddball thing | |
| 16:57:30 | mriedem | we're fixing allocations for nova instances, | |
| 16:57:33 | dansmith | it's also a placement client sort of thing, so it's weird to have it in nova-manage, | |
| 16:57:36 | mriedem | so thinking: nova-manage placement fix_allocations | |
| 16:57:40 | dansmith | but placement-manage would be the opposite | |
| 16:58:06 | mriedem | right, we need the instance uuid to find it's host/node to get the node uuid to find the resource provider it should have allocations against | |
| 16:58:13 | dansmith | mriedem: maybe we need a generic nova-manage group, like "nova-manage fix-our-mess" or "nova-manage maintenance ..." | |
| 16:58:17 | mriedem | so doing instance lookups from a placement-manage is weird | |
| 16:58:56 | dansmith | could use that generic group for syncing aggregate membership, fixing cinder attachments (historically) | |
| 16:59:25 | mriedem | jaypipes: where were you going to throw the nova-manage sync CLI for aggregates? | |
| 16:59:37 | dansmith | mriedem: wherever you tell him to | |
| 16:59:38 | dansmith | :P | |
| 17:00:29 | mriedem | was just wondering if his gears were grinding on that yet | |
| 17:03:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove [scheduler]/host_manager config option https://review.openstack.org/565806 | |
| 17:03:50 | mriedem | i'm partial to 'nova-manage placement ...' | |
| 17:03:59 | mriedem | but don't care too much | |
| 17:04:02 | mriedem | gonna get lunch | |
| 17:04:20 | dansmith | melwitt: mriedem: tssurya: I've been completely covered up in other stuff for the last week.. I have nothing cellsy to talk about, unless the cern peeps have things to report? | |
| 17:04:35 | dansmith | mriedem: well, I just don't know that we'll have enough in there to justify the group, but whatever you want | |
| 17:05:13 | mriedem | dansmith: we'll have at least 2 things, this and jay's sync aggs thing | |
| 17:05:20 | mriedem | i don't have any cells stuff to talk about | |
| 17:07:07 | tssurya | dansmith: not really except that we moved to cellsv2, and basic operations take more time than for cellsv1 , we are trying to optimize whatever possible, but since belmiro won't be around today as its a Euro unfriendly meeting timing, we can give the overall experience in the next week's meeting | |
| 17:07:56 | dansmith | tssurya: okay, yeah would be good to hear an analysis of what takes longer so we can think about what to do | |
| 17:08:27 | tssurya | dansmith: yep | |
| 17:09:00 | dansmith | cellsv1 cheated with cache coherency, so not surprising that some things will take longer to be correct ;) | |
| 17:09:43 | tssurya | dansmith: haha, yea I get it but querying to all 70 DBs versus top cell having all info is also the toll maybe | |
| 17:09:59 | dansmith | tssurya: you guys are on pike yeah? | |
| 17:10:10 | tssurya | dansmith: skipped pike, moved straight to queens :) | |
| 17:10:17 | tssurya | from ocata | |
| 17:10:38 | dansmith | oh | |
| 17:11:02 | dansmith | yeah, so you should have the parallel instance listing, | |
| 17:11:13 | dansmith | although we do iterate the cells in other places where we could optimize | |
| 17:11:37 | dansmith | tssurya: https://review.openstack.org/#/c/509003/ | |
| 17:11:41 | dansmith | that would help you more than most | |
| 17:11:56 | jaypipes | mriedem: nova-manage placement ... works for me. | |
| 17:11:57 | tssurya | dansmith: exactly, we are trying to optimize such things, however we also had some DB side issues, some being slower than the others, | |
| 17:12:36 | tssurya | dansmith: oh I have never come across that patch | |
| 17:12:40 | tssurya | thanks | |
| 17:13:04 | dansmith | tssurya: that would make us only query the cells a particular tenant has instances in, which I think in your case would be a big win.. like going from 70 to 2 or something | |
| 17:14:49 | tssurya | dansmith: wow, too bad we did not know about this patch being there all along | |
| 17:15:01 | dansmith | tssurya: well, my fault for letting it languish | |
| 17:15:03 | tssurya | I didn't even know CellMappingList.get_by_project_id existed | |
| 17:15:11 | dansmith | tssurya: it doesn't, look below :) | |
| 17:15:24 | tssurya | what we did was to go through InstanceMappings | |
| 17:15:30 | dansmith | tssurya: could be an easy backport | |
| 17:15:32 | tssurya | and nova list became fast, | |
| 17:15:34 | dansmith | tssurya: yeah that's what it does | |
| 17:17:08 | tssurya | dansmith: yes this is exactly what we did, we patched it the same way | |
| 17:17:20 | tssurya | except if I had known you had this we would have backported :D | |
| 17:17:20 | dansmith | cool | |
| 17:17:23 | dansmith | heh | |
| 17:17:27 | dansmith | I will get this rebased today | |
| 17:17:44 | tssurya | dansmith: thanks ! would be really nice to have this upstream | |
| 17:20:04 | melwitt | tssurya: so even with the all cell query executed in parallel, you saw it being slow to list instances? | |
| 17:20:29 | dansmith | melwitt: they have some very slow databases, which will hold up the query unnecessarily | |
| 17:20:40 | melwitt | I see | |
| 17:21:01 | tssurya | melwitt: once we had the list using InstanceMapping it was fast, but yes even with parallel querying under the current master code it is slow than cellsv1 | |
| 17:21:45 | tssurya | due to DB issues like dan said | |
| 17:21:57 | melwitt | gotcha | |
| 17:22:26 | dansmith | it makes it as slow as the slowest db, of course | |
| 17:22:56 | dansmith | which isn't really avoidable in general, but since they have lots of cells, but few per tenant, they will get more gain from this than people that don't do that | |
| 17:23:08 | dansmith | tssurya: some of your cell dbs are far away from the controller too right? | |
| 17:23:20 | tssurya | dansmith: eys | |
| 17:23:23 | tssurya | yes* | |
| 17:23:29 | dansmith | yeah, so that's related | |
| 17:23:32 | dansmith | punishes everyone | |
| 17:23:41 | dansmith | tssurya: since the beginning we've said we will probably have to do some caching at the top for situations like that, | |
| 17:23:51 | dansmith | but we also always said we'd wait until we had something to measure | |
| 17:23:57 | dansmith | so that bill may be coming due soon | |
| 17:24:10 | tssurya | dansmith: yea we are measuring stuff, belmiro will give a upgrade talk at the summit | |
| 17:24:24 | dansmith | cool | |
| 17:25:14 | melwitt | to the earlier question about the cells meeting, I don't have anything other than a request for review on this patch to make affinity work with multiple cells https://review.openstack.org/540258 | |
| 17:26:20 | openstack | Launchpad bug 1767303 in OpenStack Compute (nova) "Scheduler connects to all cells DBs to gather compute nodes info" [Undecided,New] - Assigned to Surya Seetharaman (tssurya) | |
| 17:26:20 | tssurya | for instance another point I wanted to know was this : https://bugs.launchpad.net/nova/+bug/1767303, we filed a new bug to only go through cells that are needed now that we have disabled the host_state cache. I will put a POC once we test the performance in our envo. | |
| 17:27:33 | tssurya | but yea, we can also talk in the next week's meeting after we see if scheduling time improves | |
| 17:27:36 | mgagne | tssurya: is there a blog post or something I can read about your migration to CellsV2? Or should I wait until summit to hear it first-hand at one of the talk? | |
| 17:28:10 | tssurya | mgagne: we will try to put up one soon :) we are still putting out fires | |
| 17:28:33 | mgagne | tssurya: all fine, I'm currently sitting in a fire too =) | |
| 17:28:54 | dansmith | tssurya: cool, makes sense | |
| 17:28:58 | tssurya | mgagne: your also on cellsv2 ? | |
| 17:28:59 | dansmith | mgagne: how many cells do you have? | |
| 17:29:09 | tssurya | you are* | |
| 17:29:10 | mgagne | tssurya: running CellsV1 Mitaka | |
| 17:29:24 | tssurya | mgagne: ack | |
| 17:29:28 | mgagne | dansmith: more than 1 =) | |
| 17:29:47 | mgagne | dansmith: I'm not (yet) concerned with performance, only the migration process with multiple cells | |
| 17:38:58 | mriedem | tssurya: do you know if cern was/is using the CachingScheduler before upgrading to ocata+? | |
| 17:39:14 | tssurya | mriedem: no we weren't and aren't | |
| 17:39:22 | mriedem | ok | |
| 17:41:01 | tssurya | dansmith: not urgent, but whenever you get the time could you re-instate the +2 on https://review.openstack.org/#/c/559299/ , its the clean up on the bp which has been held up for some time. | |
| 17:42:07 | dansmith | tssurya: done, sorry | |
| 17:42:16 | tssurya | dansmith: thank you! :) | |
| 18:10:39 | mriedem | jroll: is oath using the CachingScheduler for baremetal scheduling? | |