| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-30 | |||
| 20:34:56 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: remove unnecessary conf imports https://review.openstack.org/539314 | |
| 20:35:27 | melwitt | cfriesen: good question. looks like it tries to do something to cleanup volumes but it only does a volume delete if 'delete_on_termination' and doesn't do anything like detach volumes | |
| 20:35:51 | melwitt | so it seems like a better fix would be to properly handle cleanup in compute | |
| 20:37:21 | mriedem | melwitt: that's what ameeda's patch is trying to do | |
| 20:38:05 | mriedem | we do call _cleanup_allocated_networks when a build fails on the compute | |
| 20:38:10 | mriedem | which should cleanup ports | |
| 20:38:48 | melwitt | right | |
| 20:39:12 | mriedem | there could possibly be a bug there if we're using a stale network info cache | |
| 20:39:31 | mriedem | the _cleanup_volumes thing is definitely a gap in the compute cleanup code | |
| 20:39:34 | mriedem | if the volume is attached | |
| 20:39:48 | melwitt | yeah, seeing that | |
| 20:40:29 | melwitt | we're also missing a volume cleanup call for the InstanceNotFound block | |
| 20:40:44 | mriedem | well, | |
| 20:40:50 | mriedem | that means the instance was deleted concurrently | |
| 20:40:51 | mriedem | yeah? | |
| 20:41:03 | mriedem | which should go through the local delete in the API, or _shutdown_instance in the compute | |
| 20:41:51 | melwitt | yeah, concurrent | |
| 20:42:09 | melwitt | okay, just saw there's a cleanup ports call in there but no volumes call | |
| 20:42:58 | openstackgerrit | Merged openstack/nova stable/pike: Make sure that functional test triggered on sample changes https://review.openstack.org/536934 | |
| 20:43:07 | openstackgerrit | Merged openstack/nova master: Fix a comment in a notification functional test https://review.openstack.org/536262 | |
| 20:49:43 | mriedem | woot ^ should be able to start rechecking pike changes now too | |
| 20:50:25 | efried | jaypipes: All other things being equal, is there a reason you prefer a DELETE API over PUT {} ? | |
| 20:50:55 | melwitt | mriedem: how? did that resolve the libvirt keepalive timeout fails? | |
| 20:51:08 | mriedem | no | |
| 20:51:27 | mriedem | melwitt: you're right - it doesn't, need to backport the functional job timeout bump patch | |
| 20:51:29 | mriedem | want to propose that? | |
| 20:52:03 | melwitt | I can, but I was referring to the permafail of the cells v1 job on pike and ocata because of the libvirt keepalive timeout issue | |
| 20:52:38 | mriedem | melwitt: remember i removed the cellsv1 job from stable? | |
| 20:52:41 | melwitt | AFAIK we're still blocked on pike and ocata because of that | |
| 20:52:49 | melwitt | oh, I didn't know that. nevermind | |
| 20:52:58 | mriedem | https://review.openstack.org/#/c/538619/ | |
| 20:53:34 | mriedem | if we want that job back, i think we have to define it in-tree per-branch and make it non-voting on stable | |
| 20:53:55 | melwitt | cool ... wish we knew what the root cause was, but there were no real leads | |
| 20:54:53 | openstackgerrit | sean mooney proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257 | |
| 20:56:44 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Bumping functional test job timeouts https://review.openstack.org/539319 | |
| 20:56:45 | mriedem | the python2.7 update on jan 18 and the fact we're not using pike uca packages is as close as i got to 'leads' | |
| 20:56:59 | mriedem | the latter is unique to master | |
| 20:57:57 | melwitt | I see. so we could try pulling pike uca for stable/pike but not for stable/ocata, right? I assume that wouldn't be okay because it's a mismatch | |
| 20:58:15 | melwitt | do we need the func test timeout bump for ocata too? | |
| 20:59:13 | melwitt | we don't have .zuul.yaml there | |
| 20:59:21 | mriedem | melwitt: yes on the last question, that needs to go on top of https://review.openstack.org/#/c/536936/ | |
| 20:59:47 | mriedem | as for the former question, i tried that in https://review.openstack.org/#/c/536798/ but my patch to not run cellsv1 in stable landed first so the job didn't run on that patch | |
| 20:59:51 | melwitt | ah, okay. thanks | |
| 20:59:56 | mriedem | we'd have to do some depends-on chicanery there | |
| 21:00:03 | melwitt | gotcha | |
| 21:00:43 | mriedem | cfriesen: so on the perf thing, you just found out that compute was using more cpu than before? | |
| 21:01:02 | mriedem | that was likely because in newton the computes started posting inventory information to placement from every RT update | |
| 21:01:19 | mriedem | but you said server creates were taking a lot longer | |
| 21:03:37 | openstackgerrit | melanie witt proposed openstack/nova stable/ocata: Bumping functional test job timeouts https://review.openstack.org/539320 | |
| 21:05:01 | cfriesen | mriedem: yeah, on same hardware instance creation on newton was 32 sec and pike was 230. but it's hard to break down exactly what's causing it since anything audit-driven will also show higher usage. | |
| 21:06:41 | cfriesen | mriedem: it should be noted this is on an all-in-one system, so the openstack services are constrained to only two CPUs, and those were pretty much pinned | |
| 21:06:50 | cfriesen | ie at 100% usage | |
| 21:09:14 | mriedem | cfriesen: can't run osprofiler + rally or something to see at least where the majority of the time is being spent? | |
| 21:12:28 | cfriesen | mriedem: we've got traces from intel's vtune analyzer showing how much time is spent where, but just about everything shows increases. there's no single smoking gun. | |
| 21:14:53 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make the InstanceMapping marker UUID-like https://review.openstack.org/539323 | |
| 21:16:04 | mriedem | cfriesen: huh | |
| 21:18:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Make generation optional in ProviderTree https://review.openstack.org/539324 | |
| 21:20:48 | mriedem | melwitt: looks like you have real test failures in https://review.openstack.org/#/c/539013/ | |
| 21:21:42 | melwitt | mriedem: ah, thank you. I hadn't looked at the detail yet. I shall fix that up | |
| 21:21:47 | sean-k-mooney | cfriesen: dumb question but could it be related to meltdow/specter patches? | |
| 21:23:42 | cfriesen | sean-k-mooney: not so dumb. :) but no, this was a load from before the meltdown/spectre patches were applied. | |
| 21:25:40 | mriedem | cfriesen: and you've got the latest stable/pike release? | |
| 21:26:28 | mriedem | like i was wondering if any of the RequestContext changes might be related https://github.com/openstack/nova/commits/stable/pike/nova/context.py | |
| 21:26:34 | mriedem | since the context is used everywhere | |
| 21:27:51 | cfriesen | mriedem: not the latest, no. originally from 16.0.2, with some stable/pike stuff since then. not sure exactly what, I've been busy with other stuff. | |
| 21:28:46 | mriedem | cfriesen: ok would be good to know if it's still the same issue after you've got the latest pike fixes applied | |
| 21:28:57 | mriedem | https://docs.openstack.org/releasenotes/nova/pike.html | |
| 21:30:15 | mriedem | full change log http://paste.openstack.org/show/658066/ | |
| 21:30:46 | cfriesen | don't see anything related to "context" in there, got any pointers? | |
| 21:31:52 | cfriesen | last stable/pike changes to nova/context.py were from October | |
| 21:32:27 | mriedem | nothing in that changelog jumps out at me as a perf related fix | |
| 21:34:35 | cfriesen | does the upstream CI environment do performance tests of common operations? | |
| 21:35:56 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove compute nodes arg from ProviderTree init https://review.openstack.org/539330 | |
| 21:38:31 | mriedem | cfriesen: no | |
| 21:38:44 | mriedem | too much variance from node to node | |
| 21:39:04 | mriedem | not sure what can be pulled perf-trend wise from openstack-health | |
| 21:39:32 | mriedem | http://status.openstack.org/openstack-health/#/ | |
| 21:40:30 | mriedem | like, i don't know how to take that and see how long a simple tempest create server test has taken over the last 12 monhts | |
| 21:40:32 | mriedem | *months | |
| 21:40:36 | mriedem | mtreinish can maybe help | |
| 21:40:57 | openstackgerrit | melanie witt proposed openstack/nova stable/ocata: Stop globally caching host states in scheduler HostManager https://review.openstack.org/539013 | |
| 21:49:05 | mriedem | mmedvede: powerkvm ci seems pretty unhappy | |
| 21:49:07 | mriedem | is that a known issue? | |
| 21:51:30 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Provide support matrix and doc for VGPU https://review.openstack.org/539266 | |
| 21:56:07 | mriedem | dansmith: want to hit these backports? I didn't realize those weren't merged by now https://review.openstack.org/#/q/Ie70c77db753711e1449e99534d3b83669871943f+status:open | |
| 21:56:48 | mmedvede | mriedem: I do not see anything too far out of ordinary, which powerkvm ci unhappiness are you referring to? | |
| 21:57:00 | mmedvede | double checking now | |
| 21:57:12 | mriedem | mmedvede: https://review.openstack.org/#/c/538510/ | |
| 21:57:18 | mriedem | https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/10/538510/3/check/tempest-dsvm-full-xenial/41f1c6d/ | |
| 22:01:15 | edleafe | efried: PUT {} is not semantically the same as DELETE, even if in most cases the result is the same | |
| 22:04:15 | mriedem | cfriesen: speaking of perf, this is an easy fix for an RT perf issue if you're building several instances on the same compute host at once https://review.openstack.org/#/q/Ib588c31a4d2075f8730409d50c99dfb04180a9cd+status:open | |
| 22:04:32 | mriedem | our operations people were hitting perf issues with the big RT update lock | |
| 22:09:08 | dansmith | mriedem: got em | |
| 22:10:05 | efried | edleafe: Oh? How not? | |
| 22:10:26 | efried | edleafe: Oh, you mean in the general case, where None and {} aren't the same thing. | |
| 22:10:50 | prometheanfire | win 30 | |
| 22:11:51 | edleafe | efried: No. Sometimes you need to indicate if anything was in fact deleted. PUT {} can't do that; DELETE can | |
| 22:12:12 | efried | edleafe: It can? How? | |
| 22:12:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Pass the correct image to build_request_spec in conductor.rebuild_instance https://review.openstack.org/516404 | |
| 22:14:12 | mmedvede | mriedem: that pkvmci failure seems to have been a fluke on that VM, one of a kind. Sorry for the false negative. | |