| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-19 | |||
| 18:13:55 | dansmith | the point there is to overlap some long-running things like image download with network provisioning, | |
| 18:13:57 | melwitt | and they're using nova-net via hack | |
| 18:14:03 | melwitt | so it's not related to neutron | |
| 18:14:07 | dansmith | which wouldn't be as much of a thing on delete | |
| 18:14:17 | mriedem | ocata would mean cleaning up allocations in placement | |
| 18:14:24 | mriedem | but not sure why that would take much longer | |
| 18:14:44 | openstackgerrit | sahid proposed openstack/nova-specs master: update: introducing isolate emulthreads on host https://review.openstack.org/511188 | |
| 18:15:04 | melwitt | but huawei said mitaka is slow and that's pre-placement | |
| 18:15:05 | jaypipes | dansmith: I don't see how cur_gen can be None in that patch. | |
| 18:15:19 | dansmith | jaypipes: you're doing a .get() on it, that's how | |
| 18:16:12 | jaypipes | dansmith: the comment I removed is this: | |
| 18:16:13 | jaypipes | # Update our generation immediately, if possible. Even if there | |
| 18:16:13 | jaypipes | # are no inventories we should always have a generation but let's | |
| 18:16:13 | jaypipes | # be careful. | |
| 18:16:33 | jaypipes | dansmith: I should have just replaced the get() with a dict access. | |
| 18:16:40 | sahid | jaypipes: about your tweet, i understand that it's a bit confusing, i just try to address an use-case | |
| 18:16:40 | dansmith | jaypipes: that's what I'm saying | |
| 18:16:45 | jaypipes | dansmith: because there simply isn't a way to get a None value there. | |
| 18:16:56 | sahid | in any case i update the spec to fix the issue with ci I hope everytinhg is ok now :) | |
| 18:17:08 | dansmith | jaypipes: either do that so we break, or check the result of the get() and raise something specific | |
| 18:17:09 | dansmith | jaypipes: but don't just ignore it and pass None to placement | |
| 18:17:22 | jaypipes | dansmith: will do the former. | |
| 18:17:42 | sahid | oh... I just noticed your comments, let see | |
| 18:17:58 | dansmith | jaypipes: thanks | |
| 18:18:17 | penick_ | melwitt: we "fixed" the delete timing this morning by setting reclaim_instance_interval to a non-zero number. Deletes still take a minute or two on the backend, but at least it'll seem fast for users. | |
| 18:18:38 | mriedem | melwitt: the new internal perf thing i got yesterday was they were saying claims were taking too long in the compute because of the lock | |
| 18:18:49 | mriedem | they are testing creating 70 instances on the same host at the same time | |
| 18:19:47 | melwitt | penick_: you are now leveraging soft-delete | |
| 18:19:56 | penick_ | yep | |
| 18:20:33 | melwitt | mriedem: claims in mitaka, right? the old style claim | |
| 18:20:43 | mriedem | yes | |
| 18:20:58 | mriedem | but, we still have all that code if you're using the ram/core/disk filters | |
| 18:21:03 | mriedem | and/or caching scheduler | |
| 18:21:26 | melwitt | so the COMPUTE_RESOURCE_SEMAPHORE | |
| 18:21:28 | mriedem | anyway, there was no low level profiling done | |
| 18:21:29 | mriedem | yeah | |
| 18:21:35 | dansmith | and that's all been there for a long time | |
| 18:21:41 | mriedem | i was going to see if there are things we hit in the db while we have the lock held that could be moved out | |
| 18:21:41 | openstackgerrit | sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | |
| 18:21:49 | mriedem | like getting instance pci requests | |
| 18:21:57 | melwitt | I think that's what caused some stuff in ironic to be slow too right ... the slow initial query for ironic nodes | |
| 18:22:05 | dansmith | maybe penick_ wasn't running real conductor before but now is? | |
| 18:22:36 | mriedem | i did notice this yesterday https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1473 | |
| 18:22:42 | mriedem | that will call update_usage in the RT which has the lock | |
| 18:22:49 | mriedem | and we do that for no apparent reason | |
| 18:22:58 | mriedem | the RT doesn't care about an instance setting up networking | |
| 18:23:06 | melwitt | hm | |
| 18:23:19 | mriedem | i told them to comment that line out and see what happens | |
| 18:23:27 | mriedem | and adjust their update_available_resource interval | |
| 18:24:15 | penick_ | dansmith: that's correct.. Our Ocata clusters use conductor, and our SE team is deploying the conductor service to our juno clusters today. | |
| 18:24:30 | penick_ | use the conductor service, i mean. | |
| 18:25:21 | openstackgerrit | Merged openstack/nova stable/pike: Fix sending legacy instance.update notification https://review.openstack.org/510957 | |
| 18:25:26 | openstackgerrit | Merged openstack/nova stable/pike: Keep updating allocations for Ironic https://review.openstack.org/513198 | |
| 18:25:31 | dansmith | penick_: so I'd look to see if maybe you're pegging those whereas before you were doing a bunch of direct access with lower latency? | |
| 18:26:06 | dansmith | mriedem's thing was also a single compute, so lots of contention for things in the claim process | |
| 18:26:17 | dansmith | whereas I assume penick_ has a plethora of computes | |
| 18:26:33 | dansmith | a veritable cornucopia of bountiful resources | |
| 18:26:42 | mriedem | oh yeah, i guess on _complete_deletion we are updating the rt, | |
| 18:26:45 | mriedem | which grabs the lock | |
| 18:26:48 | penick_ | The ocata clusters are all dev and qe with only a 1-5 computes each, they're sitting pretty much idle. But i'll doublecheck to make sure we don't have something else cropping up gobbling up the resources on them | |
| 18:26:52 | mriedem | and makes all of the deletes on a single node serialized | |
| 18:28:26 | ildikov | jgriffith: added some answers | |
| 18:29:04 | mriedem | so looking way back https://github.com/openstack/nova/blob/0ca98979da842f9accf0f8d8c4eb7d4f1f9eac7a/nova/compute/manager.py#L612 | |
| 18:29:07 | mriedem | before versioned objects | |
| 18:29:22 | mriedem | we always updated an instance state using that method, which also updated the RT | |
| 18:29:24 | mriedem | which grabs the lock | |
| 18:29:38 | mriedem | so we have some vestigial use of that now, updating RT unnecessarily | |
| 18:29:52 | mriedem | RT should only care if you're creating, moving or deleting an instance | |
| 18:30:00 | mriedem | not that you're setting up networking | |
| 18:31:41 | penick_ | The ocata api hosts running the conductor service are all idle | |
| 18:31:54 | dansmith | penick_: during a delete storm? | |
| 18:32:26 | penick_ | That I don't know yet, i'm going to set one up. I'll let you know what I come up with. | |
| 18:32:38 | dansmith | at idle I'd expect them to be idle ;) | |
| 18:33:15 | penick_ | But first our -juno- conductors are all exploding as they're being deployed. Gotta help the team tune those first | |
| 18:34:14 | penick_ | I set myself up for that :| | |
| 18:35:06 | melwitt | exploding meaning they won't run? or they are overloaded and you need to adjust workers? | |
| 18:38:16 | penick_ | Overloaded, we're only running 26 workers per node, on 3 nodes. Going to try adding more | |
| 18:40:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't update RT in _allocate_network https://review.openstack.org/513473 | |
| 18:51:26 | openstackgerrit | sean mooney proposed openstack/nova-specs master: Use neutron's new port binding API for live migration https://review.openstack.org/375580 | |
| 18:53:13 | melwitt | penick_: that seems like a lot to me but would have to consult dansmith. I don't remember what the sane starting point for number of workers per node is | |
| 18:54:25 | dansmith | workers per node is not a useful metric, it's workers per compute | |
| 18:54:57 | dansmith | 26*3 is a fair number of conductor workers, but if you have a thousand computes, then ... probably not insane | |
| 18:55:04 | dansmith | it also depends on what is happening | |
| 18:55:17 | dansmith | if everything is idle, that's probably more than enough | |
| 18:55:20 | melwitt | okay, sorry I think I was thinking of earlier when penick_ said 1-5 computes but if he's talking about juno then that would be 100s of computes | |
| 18:55:33 | melwitt | the 1-5 computes is the ocata testing | |
| 18:55:53 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: integrate ProviderTree to report client https://review.openstack.org/415921 | |
| 18:55:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: add nested resource providers https://review.openstack.org/377138 | |
| 18:55:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: allow filter providers in tree https://review.openstack.org/377215 | |
| 18:55:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807 | |
| 18:55:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693 | |
| 18:55:59 | jaypipes | dansmith: ^ | |
| 18:57:36 | dansmith | melwitt: right | |
| 19:13:52 | sahid | jaypipes: ci is green https://review.openstack.org/#/c/485522/ | |
| 19:14:31 | sahid | btw mriedem, did you take descison for this: https://review.openstack.org/#/c/511188/ ? | |
| 19:21:42 | mriedem | sahid: no | |
| 19:21:47 | mriedem | it's not on my list | |
| 19:24:02 | sahid | mriedem: thanks | |
| 19:31:15 | penick_ | melwitt: this particular cluster is about 1500 nodes. But I think I found the issue. They had the hypervisors set to use the conductor service, but the API was still configured to use local conductor. Which i'm pretty sure will never work. I had them fix it and so far it appears things are working. They'll do some more tests and get back to me | |
| 19:32:11 | melwitt | aha, cool | |