Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-30
20:28:29 prometheanfire patches should flow much better through the gate now
20:28:53 mriedem cfriesen: like this? https://review.openstack.org/#/c/528385/
20:29:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional tests for traits-based scheduling https://review.openstack.org/539310
20:29:35 mriedem alex_xu: efried: ^
20:29:41 mriedem turned out it was pretty simple to write those
20:30:30 cfriesen mriedem: yep, and something similar for ports I guess.
20:34:56 openstackgerrit Matthew Edmonds proposed openstack/nova master: remove unnecessary conf imports https://review.openstack.org/539314
20:35:27 melwitt cfriesen: good question. looks like it tries to do something to cleanup volumes but it only does a volume delete if 'delete_on_termination' and doesn't do anything like detach volumes
20:35:51 melwitt so it seems like a better fix would be to properly handle cleanup in compute
20:37:21 mriedem melwitt: that's what ameeda's patch is trying to do
20:38:05 mriedem we do call _cleanup_allocated_networks when a build fails on the compute
20:38:10 mriedem which should cleanup ports
20:38:48 melwitt right
20:39:12 mriedem there could possibly be a bug there if we're using a stale network info cache
20:39:31 mriedem the _cleanup_volumes thing is definitely a gap in the compute cleanup code
20:39:34 mriedem if the volume is attached
20:39:48 melwitt yeah, seeing that
20:40:29 melwitt we're also missing a volume cleanup call for the InstanceNotFound block
20:40:44 mriedem well,
20:40:50 mriedem that means the instance was deleted concurrently
20:40:51 mriedem yeah?
20:41:03 mriedem which should go through the local delete in the API, or _shutdown_instance in the compute
20:41:51 melwitt yeah, concurrent
20:42:09 melwitt okay, just saw there's a cleanup ports call in there but no volumes call
20:42:58 openstackgerrit Merged openstack/nova stable/pike: Make sure that functional test triggered on sample changes https://review.openstack.org/536934
20:43:07 openstackgerrit Merged openstack/nova master: Fix a comment in a notification functional test https://review.openstack.org/536262
20:49:43 mriedem woot ^ should be able to start rechecking pike changes now too
20:50:25 efried jaypipes: All other things being equal, is there a reason you prefer a DELETE API over PUT {} ?
20:50:55 melwitt mriedem: how? did that resolve the libvirt keepalive timeout fails?
20:51:08 mriedem no
20:51:27 mriedem melwitt: you're right - it doesn't, need to backport the functional job timeout bump patch
20:51:29 mriedem want to propose that?
20:52:03 melwitt I can, but I was referring to the permafail of the cells v1 job on pike and ocata because of the libvirt keepalive timeout issue
20:52:38 mriedem melwitt: remember i removed the cellsv1 job from stable?
20:52:41 melwitt AFAIK we're still blocked on pike and ocata because of that
20:52:49 melwitt oh, I didn't know that. nevermind
20:52:58 mriedem https://review.openstack.org/#/c/538619/
20:53:34 mriedem if we want that job back, i think we have to define it in-tree per-branch and make it non-voting on stable
20:53:55 melwitt cool ... wish we knew what the root cause was, but there were no real leads
20:54:53 openstackgerrit sean mooney proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257
20:56:44 openstackgerrit melanie witt proposed openstack/nova stable/pike: Bumping functional test job timeouts https://review.openstack.org/539319
20:56:45 mriedem the python2.7 update on jan 18 and the fact we're not using pike uca packages is as close as i got to 'leads'
20:56:59 mriedem the latter is unique to master
20:57:57 melwitt I see. so we could try pulling pike uca for stable/pike but not for stable/ocata, right? I assume that wouldn't be okay because it's a mismatch
20:58:15 melwitt do we need the func test timeout bump for ocata too?
20:59:13 melwitt we don't have .zuul.yaml there
20:59:21 mriedem melwitt: yes on the last question, that needs to go on top of https://review.openstack.org/#/c/536936/
20:59:47 mriedem as for the former question, i tried that in https://review.openstack.org/#/c/536798/ but my patch to not run cellsv1 in stable landed first so the job didn't run on that patch
20:59:51 melwitt ah, okay. thanks
20:59:56 mriedem we'd have to do some depends-on chicanery there
21:00:03 melwitt gotcha
21:00:43 mriedem cfriesen: so on the perf thing, you just found out that compute was using more cpu than before?
21:01:02 mriedem that was likely because in newton the computes started posting inventory information to placement from every RT update
21:01:19 mriedem but you said server creates were taking a lot longer
21:03:37 openstackgerrit melanie witt proposed openstack/nova stable/ocata: Bumping functional test job timeouts https://review.openstack.org/539320
21:05:01 cfriesen mriedem: yeah, on same hardware instance creation on newton was 32 sec and pike was 230. but it's hard to break down exactly what's causing it since anything audit-driven will also show higher usage.
21:06:41 cfriesen mriedem: it should be noted this is on an all-in-one system, so the openstack services are constrained to only two CPUs, and those were pretty much pinned
21:06:50 cfriesen ie at 100% usage
21:09:14 mriedem cfriesen: can't run osprofiler + rally or something to see at least where the majority of the time is being spent?
21:12:28 cfriesen mriedem: we've got traces from intel's vtune analyzer showing how much time is spent where, but just about everything shows increases. there's no single smoking gun.
21:14:53 openstackgerrit Ed Leafe proposed openstack/nova master: Make the InstanceMapping marker UUID-like https://review.openstack.org/539323
21:16:04 mriedem cfriesen: huh
21:18:22 openstackgerrit Eric Fried proposed openstack/nova master: Make generation optional in ProviderTree https://review.openstack.org/539324
21:20:48 mriedem melwitt: looks like you have real test failures in https://review.openstack.org/#/c/539013/
21:21:42 melwitt mriedem: ah, thank you. I hadn't looked at the detail yet. I shall fix that up
21:21:47 sean-k-mooney cfriesen: dumb question but could it be related to meltdow/specter patches?
21:23:42 cfriesen sean-k-mooney: not so dumb. :) but no, this was a load from before the meltdown/spectre patches were applied.
21:25:40 mriedem cfriesen: and you've got the latest stable/pike release?
21:26:28 mriedem like i was wondering if any of the RequestContext changes might be related https://github.com/openstack/nova/commits/stable/pike/nova/context.py
21:26:34 mriedem since the context is used everywhere
21:27:51 cfriesen mriedem: not the latest, no. originally from 16.0.2, with some stable/pike stuff since then. not sure exactly what, I've been busy with other stuff.
21:28:46 mriedem cfriesen: ok would be good to know if it's still the same issue after you've got the latest pike fixes applied
21:28:57 mriedem https://docs.openstack.org/releasenotes/nova/pike.html
21:30:15 mriedem full change log http://paste.openstack.org/show/658066/
21:30:46 cfriesen don't see anything related to "context" in there, got any pointers?
21:31:52 cfriesen last stable/pike changes to nova/context.py were from October
21:32:27 mriedem nothing in that changelog jumps out at me as a perf related fix
21:34:35 cfriesen does the upstream CI environment do performance tests of common operations?
21:35:56 openstackgerrit Eric Fried proposed openstack/nova master: Remove compute nodes arg from ProviderTree init https://review.openstack.org/539330
21:38:31 mriedem cfriesen: no
21:38:44 mriedem too much variance from node to node
21:39:04 mriedem not sure what can be pulled perf-trend wise from openstack-health
21:39:32 mriedem http://status.openstack.org/openstack-health/#/
21:40:30 mriedem like, i don't know how to take that and see how long a simple tempest create server test has taken over the last 12 monhts
21:40:32 mriedem *months
21:40:36 mriedem mtreinish can maybe help
21:40:57 openstackgerrit melanie witt proposed openstack/nova stable/ocata: Stop globally caching host states in scheduler HostManager https://review.openstack.org/539013
21:49:05 mriedem mmedvede: powerkvm ci seems pretty unhappy
21:49:07 mriedem is that a known issue?
21:51:30 openstackgerrit Sylvain Bauza proposed openstack/nova master: Provide support matrix and doc for VGPU https://review.openstack.org/539266
21:56:07 mriedem dansmith: want to hit these backports? I didn't realize those weren't merged by now https://review.openstack.org/#/q/Ie70c77db753711e1449e99534d3b83669871943f+status:open
21:56:48 mmedvede mriedem: I do not see anything too far out of ordinary, which powerkvm ci unhappiness are you referring to?
21:57:00 mmedvede double checking now
21:57:12 mriedem mmedvede: https://review.openstack.org/#/c/538510/
21:57:18 mriedem https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/10/538510/3/check/tempest-dsvm-full-xenial/41f1c6d/
22:01:15 edleafe efried: PUT {} is not semantically the same as DELETE, even if in most cases the result is the same
22:04:15 mriedem cfriesen: speaking of perf, this is an easy fix for an RT perf issue if you're building several instances on the same compute host at once https://review.openstack.org/#/q/Ib588c31a4d2075f8730409d50c99dfb04180a9cd+status:open
22:04:32 mriedem our operations people were hitting perf issues with the big RT update lock
22:09:08 dansmith mriedem: got em
22:10:05 efried edleafe: Oh? How not?

Earlier   Later