| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-07 | |||
| 16:47:18 | bauzas | and I think it's quite important for a lot of people | |
| 16:47:30 | stephenfin | bauzas: Jinx :) Scroll up | |
| 16:47:46 | bauzas | damn | |
| 16:47:56 | bauzas | we're harassing mriedem | |
| 16:47:59 | bauzas | come on | |
| 16:48:49 | mriedem | once we're past RC1 i plan on organizing the ptg etherpad a bit | |
| 16:49:06 | mriedem | until then, just put your topic ideas in there and i'll deal with them later | |
| 16:49:10 | cdent | mriedem: I've been playing around locally with the ideas you've done in your 1000 instance create thing: https://review.openstack.org/#/c/507918/ Can be a bit slow, but seems to behave. Now I'd like to change the parameters a bit and I'm not sure if I can: | |
| 16:49:14 | stephenfin | mriedem: Sweet. Thanks for the info. It's noted in the doc | |
| 16:49:56 | cdent | What I want to do is create a large number of nova-cpu processes (with the fake driver), but something needs to constrain for the fact that the resource providers needs to have different names (which usually comes from the hostname). Is there an already established way to do that? | |
| 16:50:43 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make the InstanceMapping marker UUID-like https://review.openstack.org/539323 | |
| 16:51:07 | cfriesen | cdent: can't you specify the host name in the config file? | |
| 16:51:19 | cdent | cfriesen: I was hoping to avoid creating multiple config files | |
| 16:51:42 | mriedem | cdent: i've wanted to play around with doing something like that too using NUMBER_FAKE_NOVA_COMPUTE | |
| 16:51:50 | mriedem | not sure if that actually works properly anymore, which is what i wanted to test | |
| 16:51:53 | mriedem | so tinker with NUMBER_FAKE_NOVA_COMPUTE i think | |
| 16:51:57 | cdent | could I override the host name from the ciommand line somhow? | |
| 16:52:03 | cdent | mriedem: cool, looking | |
| 16:52:35 | mriedem | i'm not sure if/how it creates a unique hostname for the n-cpu process | |
| 16:52:48 | cdent | found it, it does | |
| 16:52:58 | mriedem | oh | |
| 16:52:59 | mriedem | iniset $fake_conf DEFAULT nhost "${HOSTNAME}${i}" | |
| 16:53:03 | mriedem | heh | |
| 16:53:05 | mriedem | \o/ | |
| 16:53:06 | cdent | just appends a number in a separate fake conf file | |
| 16:53:09 | cdent | woot | |
| 16:53:16 | cdent | I will get this started then | |
| 16:53:35 | mriedem | i've been meaning to kick this for months now to compare some different setups against placement | |
| 16:53:43 | cdent | ditto | |
| 16:53:44 | mriedem | like caching vs filter scheduler | |
| 16:53:59 | cdent | my goal is basically to see if I can get placement to break under pressure | |
| 16:54:11 | bauzas | caching vs. filter scheduler is a big deal | |
| 16:54:13 | cfriesen | has anyone ever heard of an instance getting information for another instance when querying the metadata server? Seems unlikely, but we've apparently hit it occasionally. (Assuming the test isn't mucked up.) | |
| 16:54:36 | sean-k-mooney | mriedem: i think jay did some benchmeraks comparing cacheing vs filtering a while back | |
| 16:55:31 | melwitt | mriedem: I was thinking this might be something for rc1? https://review.openstack.org/#/c/540258/ | |
| 16:55:38 | cdent | bbs | |
| 16:56:09 | melwitt | I think affinity scheduling doesn't work for multi-cell | |
| 16:56:22 | bauzas | sean-k-mooney: well, it requires a real nova deployment tbh | |
| 16:56:36 | mriedem | melwitt: that would be broken since pike right? | |
| 16:56:44 | openstackgerrit | Merged openstack/nova master: Test case: new standard resource class unusable https://review.openstack.org/539732 | |
| 16:56:49 | melwitt | mriedem: yes | |
| 16:56:54 | melwitt | for multi-cell only | |
| 16:57:00 | sean-k-mooney | bauzas: to test schduling. not really it really just needs the right db entries | |
| 16:57:01 | bauzas | because retries mechanism w/ caching has to be compared vs. no retries w/ placement | |
| 16:57:19 | bauzas | sean-k-mooney: I disagree | |
| 16:57:46 | sean-k-mooney | bauzas: if you include retries you are not testing just schduling you are also mesuing boot time and other factors | |
| 16:57:55 | mriedem | melwitt: can an instance group have members in multiple cells? | |
| 16:57:59 | bauzas | sean-k-mooney: if you wanna compare apples and apples, fire 1000 instances concurrently and see how many are in ERROR state and how long it took for booting the successful ones | |
| 16:58:10 | bauzas | that's for me the success criteria for comparison | |
| 16:58:32 | bauzas | mriedem: melwitt: I think we discussed that but we never implemented a solution | |
| 16:58:46 | sean-k-mooney | bauzas: sound like a rally test to me :) which is also valid just if you want to compare just the schduler part that is not need | |
| 16:58:46 | melwitt | mriedem: hmm. not sure | |
| 16:59:06 | sean-k-mooney | bauzas: rally is proably more meaning full however | |
| 16:59:10 | bauzas | melwitt: mriedem: I'm pretty sure the affinity filters don't take the cells awareness in mind | |
| 16:59:31 | bauzas | sean-k-mooney: I don't care about the tooling, I care more about the criterias | |
| 16:59:49 | bauzas | you can achieve that with a single curl script | |
| 17:00:32 | melwitt | bauzas: it's not that, it's that the InstanceGroup.get_hosts method doesn't query all cells, so it won't pick up any group members in a multi-cell environment | |
| 17:00:43 | sean-k-mooney | bauzas: yes but you cant just run curl with filter schduler and then run it agin after chaning the config | |
| 17:00:48 | openstackgerrit | Jacek Tomasiak proposed openstack/python-novaclient master: Fix listing of instances above API max_limit https://review.openstack.org/534222 | |
| 17:00:59 | sean-k-mooney | the db needs to be reset to the same state e.g. redploy | |
| 17:01:11 | bauzas | melwitt: oh, because get_hosts() is looking up at the cell DB ? | |
| 17:01:27 | bauzas | I was thinking we migrated instance_groups table to the API DB | |
| 17:02:14 | melwitt | bauzas: we did, but it's looking for members (instances). we aren't targeting to any cell, so it's looking at "local DB" which the API doesn't have | |
| 17:02:56 | bauzas | ah sad | |
| 17:03:09 | bauzas | couldn't we just look at the instance mappings ? | |
| 17:03:19 | melwitt | mriedem: so I think in multi-cell since we're not targeting any cells for the get_hosts query (for instances) we will always get an empty list because the API doesn't have a local cell DB, yeah? | |
| 17:03:19 | bauzas | and then the host mappings ? | |
| 17:03:53 | bauzas | I voted on jaypipes's twitter question with "instance groups" | |
| 17:03:58 | bauzas | don't blame me | |
| 17:04:19 | melwitt | bauzas: maybe. I didn't think of that. I added a scatter-gather to query all cells as a first proposal | |
| 17:04:54 | bauzas | melwitt: well, my proposition was stupid | |
| 17:05:08 | bauzas | melwitt: because the instance/host relationship is stored in the child DB | |
| 17:05:28 | bauzas | since RequestSpec doesn't tell where your instance is located (fortunately) | |
| 17:05:39 | melwitt | oh, right | |
| 17:06:23 | mriedem | yeah the instance mapping record has no association to the host mapping record | |
| 17:06:43 | bauzas | melwitt: what we could optimize tho is the scatter-gather method you wrote to only check the related cells by looking at the instance mappings table | |
| 17:06:47 | mriedem | melwitt: so this looks ok, but we can skip cell0 in your scatter/gather call | |
| 17:07:00 | bauzas | like, get all the cells from that member list | |
| 17:07:15 | bauzas | and then, query each cell to get each host from that list of member | |
| 17:07:19 | bauzas | members* even | |
| 17:07:19 | mriedem | bauzas: i'm not sure how much that's going to save perf wise | |
| 17:07:26 | melwitt | mriedem: yeah ... I had been thinking we could have group members in ERROR state in cell0 *but* they would have no host so indeed I think could skip | |
| 17:07:35 | bauzas | mriedem: well, you're right | |
| 17:07:43 | bauzas | mriedem: I don't expect more than 80 cells per deployment | |
| 17:07:43 | mriedem | melwitt: gonna want dansmith to also look at this | |
| 17:07:52 | bauzas | if we consider the big guys | |
| 17:07:56 | mriedem | bauzas: where do you come up with that number? | |
| 17:08:01 | bauzas | CERN, nope ? | |
| 17:08:04 | mriedem | dude, | |
| 17:08:10 | bauzas | isn't that 80 cells v1 ? | |
| 17:08:15 | mriedem | you've forgotten about M'FING EDGE?! | |
| 17:08:17 | dansmith | bauzas: yeah I think that's right | |
| 17:08:23 | mriedem | 10K cells | |
| 17:08:26 | mriedem | AT THE EDGE | |
| 17:08:41 | bauzas | anyway, 10K is a processable number | |
| 17:09:07 | bauzas | ask Jay to refrain Verizon then | |
| 17:09:11 | mriedem | melwitt: ideally we'd have a functional regression test for this, but likely need to do that on top separately | |
| 17:09:23 | openstackgerrit | Dan Smith proposed openstack/nova master: Make host_manager use scatter-gather and ignore down cells https://review.openstack.org/539617 | |
| 17:09:23 | openstackgerrit | Dan Smith proposed openstack/nova master: Make service all-cells min version helper use scatter-gather https://review.openstack.org/539590 | |
| 17:10:12 | bauzas | oh man, I'd love to have a tox -ereleasenotes -- -HEAD | |