Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-26
16:33:17 mriedem no
16:33:30 mriedem it takes the allocations for the instance on the source node, and makes those same allocations for the instance on the dest node
16:33:40 dansmith which will erase the source allocation
16:33:41 mriedem it's basically what the scheduler would do,
16:33:44 dansmith because... only one consumer
16:33:55 mriedem oh it calls claim_resources,
16:33:57 mriedem so yeah it doubles
16:34:02 mriedem this is the thing where force=True
16:34:07 mriedem so we don't call the scheduler to double the allocs
16:34:20 mriedem and i said i wanted to move back into the scheduler, but we'd need a skip_filters flag in select_destinations
16:34:27 dansmith okay I didn't think claim_resources was the doubling one, but maybe so, I'll dig a bit
16:34:39 mriedem claim_resources calls the double stuff method
16:34:59 dansmith okay
16:35:48 dansmith cdent: can you look at my comment on the DRY thing and see if you buy what I'm sellin' ?
16:38:24 cdent dansmith: I will buy that with an entire whole dollar, if you comment the plan
16:38:50 dansmith ack
16:40:23 dansmith cdent: you saw the "when we have an atomic operation we should remove this" right?
16:42:13 cdent yes, but (unless I missed it) there’s no “this dupe with that other thing but we don’t care cuz”
16:42:27 dansmith I will add more words
16:42:31 cdent I’ll still buy it for a dollar even if you don’t
16:45:16 openstackgerrit Dan Smith proposed openstack/nova master: Make allocation cleanup honor new by-migration rules https://review.openstack.org/498948
16:45:16 openstackgerrit Dan Smith proposed openstack/nova master: Pre-create migration object https://review.openstack.org/498950
16:45:17 openstackgerrit Dan Smith proposed openstack/nova master: Revert allocations by migration uuid https://review.openstack.org/498949
16:45:17 openstackgerrit Dan Smith proposed openstack/nova master: Refactor resource tracker to account for migration allocations https://review.openstack.org/506419
16:45:18 openstackgerrit Dan Smith proposed openstack/nova master: Make migration uuid hold allocations for migrating instances https://review.openstack.org/506420
16:51:53 openstackgerrit Sean Dague proposed openstack/nova master: Move ploop commands to privsep. https://review.openstack.org/492325
16:54:07 openstackgerrit Sean Dague proposed openstack/nova master: Move ploop commands to privsep. https://review.openstack.org/492325
16:58:06 mriedem notifications meeting in openstack-meeting-4 in 2 minutes
17:00:18 gibi ... and now it is started
17:27:00 openstackgerrit Moshe Levi proposed openstack/nova master: Don't overwrite binding-profile https://review.openstack.org/505613
17:32:29 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: config drive https://review.openstack.org/409404
17:38:24 mriedem dansmith: aha, i think i'm hitting issues in devstack where placement isn't getting cleaned up for instances that get 'local' deleted in the api
17:38:45 mriedem not totally sure yet, but failing to burst 500 new instances, hitting NoValidHost
17:38:53 mriedem and i assume it's placement b/c it's not the scheduler filters
17:39:17 dansmith mriedem: and why do you have locally-deleted instances for this test?
17:39:17 melwitt for local deletes, allocations aren't cleaned up till the compute host heals it
17:39:24 dansmith right, what melwitt said
17:39:27 mriedem mysql> select count(id) from consumers;
17:39:27 mriedem | count(id) |
17:39:27 mriedem | 2002 |
17:39:27 mriedem +-----------+
17:39:28 mriedem 1 row in set (0.01 sec)
17:40:05 mriedem melwitt: there is no compute for these
17:40:07 mriedem they failed during scheduling
17:40:22 mriedem although yeah why would placement have allocations for these...
17:40:23 mriedem wtf
17:40:37 mriedem stack@devstack:~$ nova list | grep -c ERROR
17:40:37 mriedem 1000
17:40:39 melwitt oh, hm
17:40:48 mriedem so i've got 1000 instances in ERROR state, and 2002 consumers in the api db
17:41:20 melwitt allocations are written at claim time?
17:41:27 mriedem from the scheduler yeah
17:42:10 melwitt so that would explain the ones you do have. but I guess your point is why are there more allocation consumers than non error instances
17:42:49 mriedem that's because i've deleted 1000 over time
17:43:17 mriedem i was hitting messaging timeouts between conductor and the scheduler earlier today, so had 500 in error which i needed to be active, so deleted all of those, restarted conductor and scheduler, and was able to create a single instance
17:43:22 mriedem so tried with 500 more again
17:43:26 mriedem and hit novalidhost on all of those
17:44:18 openstackgerrit Merged openstack/nova master: cleanup test-requirements https://review.openstack.org/507063
17:55:32 openstackgerrit Dan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation https://review.openstack.org/507638
17:55:45 dansmith jaypipes: cdent: ^ quick stab at the live migrate version of this
17:56:02 dansmith it's probably rough at this point, but worth a look I think
18:32:22 cdent dansmith: haven’t had a chance to give it a proper look, but saw a weird when skimming the live migrate thing
18:32:48 dansmith lol
18:33:04 dansmith it's returning True-ish which is what I wanted for the functional tests
18:33:09 dansmith so.. working as designed? :)
18:33:26 dansmith s/returning/being/
18:34:36 openstackgerrit Merged openstack/nova master: Set the Pike release version for scheduler RPC https://review.openstack.org/507245
18:34:51 cdent go python!
18:38:00 mriedem wtf, so i can't create multiple instances, i get novalidhost, but i can create one at a time
18:38:21 melwitt are you using multi-create?
18:38:27 mriedem yeah
18:38:30 mriedem wasn't a problem yesterday
18:38:37 melwitt oh
18:38:41 mriedem but i had a bit of a cleaner env yesterday
18:39:46 melwitt multi-create will reject you if any one of min_count can't be accommodated. so one at a time would work if you're in that situation, if some/most of them fit
18:40:22 mriedem yesterday i created 100, then like 200, then 500 more or something
18:40:24 mriedem eventually got to 1000
18:41:49 mriedem i can just restack this env, but it makes me worry that we aren't properly cleaning up allocations somewhere
18:42:37 melwitt yeah
18:43:15 melwitt did you say yesterday you don't have computes, or something like that? I just wonder what happens with FakeDriver, if it somehow doesn't call the healing allocations code
18:43:38 mriedem we don't heal since pike
18:43:51 mriedem if you don't have computes < pike, we don't heal
18:44:04 mriedem this is just a single compute, single node devstack
18:44:11 mriedem with the fake driver and noop quota
18:44:22 melwitt oh wait, sorry I was thinking of local delete
18:44:53 melwitt I was trying to think if with FakeDriver, does the code that deletes allocations when an instance is deleted, run
18:45:02 melwitt or if that even matters
18:45:17 mriedem the compute manager cleans up allocations when an instance is deleted
18:45:34 dansmith mriedem: we heal for deletes
18:45:44 dansmith mriedem: did you archive them after local delete before you started up?
18:46:01 mriedem i've been archiving yeah
18:46:05 melwitt I'm not sure whether he had local deletes
18:46:06 dansmith so that's why
18:46:14 dansmith I thought he did
18:46:34 melwitt I thought he did too but I'm getting a little confused
18:46:53 mriedem yesterday i didn't have any instances in ERROR state, so they were all in the cell
18:47:04 mriedem i deleted all of those and then archived cell0 and cell1
18:47:21 dansmith deleted them locally?
18:47:21 mriedem today i've been trying to get 500 ERROR during scheduling, and 500 ACTIVE
18:47:24 mriedem based on the flavor i use

Earlier   Later