| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-19 | |||
| 16:24:40 | bauzas | mriedem: honestly a specless BP seems good if we don't change the CLI | |
| 16:25:12 | bauzas | then we could defer the CLI change in a Rocky spec I guess | |
| 16:37:23 | stephenfin | bauzas: Just to reiterate, the CLI won't change from a usage perspective (minus some help messages, return codes). How it's implemented under the hood would | |
| 16:50:27 | openstackgerrit | Eric Fried proposed openstack/nova master: [Trivial] Fix up a docstring https://review.openstack.org/513447 | |
| 16:50:39 | efried | sdague ^ that's been on my backlog for a while :) | |
| 16:53:13 | sdague | +A | |
| 17:17:37 | dansmith | jaypipes: are you working on a rev to 415921? | |
| 17:32:52 | mriedem | ildikov: ok notes are in https://review.openstack.org/#/c/499777/ | |
| 17:32:59 | mriedem | groot: johnthetubaguy: ^ | |
| 17:35:42 | ildikov | mriedem: thanks! | |
| 17:47:03 | ildikov | mriedem: johnthetubaguy: how much time do I have to upload the new version? | |
| 17:59:06 | openstackgerrit | sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | |
| 17:59:28 | openstackgerrit | Ildiko Vancsa proposed openstack/nova-specs master: Add multiattach support to Nova https://review.openstack.org/499777 | |
| 18:00:38 | melwitt | mriedem: I was talking to penick_ the other day and they've been testing ocata internally, and noticed delete is really slow (same as you encountered in your devstack fakedriver testing) | |
| 18:00:51 | ildikov | mriedem: johnthetubaguy: jgriffith: new old version of the spec ^^ | |
| 18:01:05 | melwitt | as in, the REST API call returns fast but it takes like a minute for the instance to disappear | |
| 18:01:58 | melwitt | hopefully they can investigate but I've made myself a TODO to look into it when I get a chance | |
| 18:07:56 | jaypipes | dansmith: yes | |
| 18:09:37 | dansmith | alright. | |
| 18:11:14 | mriedem | ildikov: 4 hours? | |
| 18:11:46 | ildikov | mriedem: uploaded a new version plz check :) | |
| 18:11:49 | mriedem | melwitt: cool, ok - i think what huawei did internally was run the volume and network cleanup in parallel like when you're creating the instance | |
| 18:12:07 | mriedem | melwitt: understanding if there is a regression would be nice too though | |
| 18:12:24 | jgriffith | ildikov couple questions/nits in review | |
| 18:12:33 | melwitt | mriedem: and that solved it? will they upstream it? | |
| 18:12:51 | mriedem | melwitt: not sure, i think they were still saying it's slow, but they are on mitaka | |
| 18:13:04 | melwitt | mriedem: yeah, it was a regression for sure compared with juno, so I'll dig into what happened there | |
| 18:13:20 | dansmith | mriedem: you didn't have volumes in your test, right? | |
| 18:13:25 | mriedem | nope | |
| 18:13:29 | mriedem | nor networking | |
| 18:13:31 | dansmith | yeah | |
| 18:13:41 | dansmith | so the networking parallelism isn't going to buy you as much on delete I think | |
| 18:13:46 | melwitt | yahoo/oath doesn't have volumes either | |
| 18:13:55 | dansmith | the point there is to overlap some long-running things like image download with network provisioning, | |
| 18:13:57 | melwitt | and they're using nova-net via hack | |
| 18:14:03 | melwitt | so it's not related to neutron | |
| 18:14:07 | dansmith | which wouldn't be as much of a thing on delete | |
| 18:14:17 | mriedem | ocata would mean cleaning up allocations in placement | |
| 18:14:24 | mriedem | but not sure why that would take much longer | |
| 18:14:44 | openstackgerrit | sahid proposed openstack/nova-specs master: update: introducing isolate emulthreads on host https://review.openstack.org/511188 | |
| 18:15:04 | melwitt | but huawei said mitaka is slow and that's pre-placement | |
| 18:15:05 | jaypipes | dansmith: I don't see how cur_gen can be None in that patch. | |
| 18:15:19 | dansmith | jaypipes: you're doing a .get() on it, that's how | |
| 18:16:12 | jaypipes | dansmith: the comment I removed is this: | |
| 18:16:13 | jaypipes | # Update our generation immediately, if possible. Even if there | |
| 18:16:13 | jaypipes | # are no inventories we should always have a generation but let's | |
| 18:16:13 | jaypipes | # be careful. | |
| 18:16:33 | jaypipes | dansmith: I should have just replaced the get() with a dict access. | |
| 18:16:40 | sahid | jaypipes: about your tweet, i understand that it's a bit confusing, i just try to address an use-case | |
| 18:16:40 | dansmith | jaypipes: that's what I'm saying | |
| 18:16:45 | jaypipes | dansmith: because there simply isn't a way to get a None value there. | |
| 18:16:56 | sahid | in any case i update the spec to fix the issue with ci I hope everytinhg is ok now :) | |
| 18:17:08 | dansmith | jaypipes: either do that so we break, or check the result of the get() and raise something specific | |
| 18:17:09 | dansmith | jaypipes: but don't just ignore it and pass None to placement | |
| 18:17:22 | jaypipes | dansmith: will do the former. | |
| 18:17:42 | sahid | oh... I just noticed your comments, let see | |
| 18:17:58 | dansmith | jaypipes: thanks | |
| 18:18:17 | penick_ | melwitt: we "fixed" the delete timing this morning by setting reclaim_instance_interval to a non-zero number. Deletes still take a minute or two on the backend, but at least it'll seem fast for users. | |
| 18:18:38 | mriedem | melwitt: the new internal perf thing i got yesterday was they were saying claims were taking too long in the compute because of the lock | |
| 18:18:49 | mriedem | they are testing creating 70 instances on the same host at the same time | |
| 18:19:47 | melwitt | penick_: you are now leveraging soft-delete | |
| 18:19:56 | penick_ | yep | |
| 18:20:33 | melwitt | mriedem: claims in mitaka, right? the old style claim | |
| 18:20:43 | mriedem | yes | |
| 18:20:58 | mriedem | but, we still have all that code if you're using the ram/core/disk filters | |
| 18:21:03 | mriedem | and/or caching scheduler | |
| 18:21:26 | melwitt | so the COMPUTE_RESOURCE_SEMAPHORE | |
| 18:21:28 | mriedem | anyway, there was no low level profiling done | |
| 18:21:29 | mriedem | yeah | |
| 18:21:35 | dansmith | and that's all been there for a long time | |
| 18:21:41 | mriedem | i was going to see if there are things we hit in the db while we have the lock held that could be moved out | |
| 18:21:41 | openstackgerrit | sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | |
| 18:21:49 | mriedem | like getting instance pci requests | |
| 18:21:57 | melwitt | I think that's what caused some stuff in ironic to be slow too right ... the slow initial query for ironic nodes | |
| 18:22:05 | dansmith | maybe penick_ wasn't running real conductor before but now is? | |
| 18:22:36 | mriedem | i did notice this yesterday https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1473 | |
| 18:22:42 | mriedem | that will call update_usage in the RT which has the lock | |
| 18:22:49 | mriedem | and we do that for no apparent reason | |
| 18:22:58 | mriedem | the RT doesn't care about an instance setting up networking | |
| 18:23:06 | melwitt | hm | |
| 18:23:19 | mriedem | i told them to comment that line out and see what happens | |
| 18:23:27 | mriedem | and adjust their update_available_resource interval | |
| 18:24:15 | penick_ | dansmith: that's correct.. Our Ocata clusters use conductor, and our SE team is deploying the conductor service to our juno clusters today. | |
| 18:24:30 | penick_ | use the conductor service, i mean. | |
| 18:25:21 | openstackgerrit | Merged openstack/nova stable/pike: Fix sending legacy instance.update notification https://review.openstack.org/510957 | |
| 18:25:26 | openstackgerrit | Merged openstack/nova stable/pike: Keep updating allocations for Ironic https://review.openstack.org/513198 | |
| 18:25:31 | dansmith | penick_: so I'd look to see if maybe you're pegging those whereas before you were doing a bunch of direct access with lower latency? | |
| 18:26:06 | dansmith | mriedem's thing was also a single compute, so lots of contention for things in the claim process | |
| 18:26:17 | dansmith | whereas I assume penick_ has a plethora of computes | |
| 18:26:33 | dansmith | a veritable cornucopia of bountiful resources | |
| 18:26:42 | mriedem | oh yeah, i guess on _complete_deletion we are updating the rt, | |
| 18:26:45 | mriedem | which grabs the lock | |
| 18:26:48 | penick_ | The ocata clusters are all dev and qe with only a 1-5 computes each, they're sitting pretty much idle. But i'll doublecheck to make sure we don't have something else cropping up gobbling up the resources on them | |
| 18:26:52 | mriedem | and makes all of the deletes on a single node serialized | |
| 18:28:26 | ildikov | jgriffith: added some answers | |
| 18:29:04 | mriedem | so looking way back https://github.com/openstack/nova/blob/0ca98979da842f9accf0f8d8c4eb7d4f1f9eac7a/nova/compute/manager.py#L612 | |
| 18:29:07 | mriedem | before versioned objects | |
| 18:29:22 | mriedem | we always updated an instance state using that method, which also updated the RT | |
| 18:29:24 | mriedem | which grabs the lock | |
| 18:29:38 | mriedem | so we have some vestigial use of that now, updating RT unnecessarily | |