Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-30
19:36:45 mriedem mgagne: huawei public cloud is using it as well
19:36:53 mriedem so yeah i have to care about that one :)
19:36:59 mgagne mriedem: wasn't it them that made a presentation at the summit about it?
19:37:21 mriedem i don't remember one, but which summit?
19:37:29 mriedem it was added by rax
19:37:44 mgagne austin
19:37:59 mgagne was Intel
19:38:00 mgagne https://www.openstack.org/videos/austin-2016/dive-into-nova-scheduler-performance-where-is-the-bottleneck
19:38:06 mriedem yeah i remember that one
19:38:15 mriedem that was about a proposal for a different scheduler
19:38:57 mgagne difference of performance is like day and night (filter vs caching)
19:38:58 mriedem placement was pretty new still around that time so a lot of the outcome of that session (there was also a related design summit session) was "placement should handle a lot of these same issues"
19:39:40 mriedem i'm mostly interested, right now, in filter scheduler + placement vs caching
19:39:48 mriedem to see if that gap is much smaller
19:40:29 mgagne yes... because if placement is not as fast or close to be as fast, I would be like: "what's the point?" =)
19:41:46 prometheanfire mriedem: it's getting to the point where we'll need an FFE for https://review.openstack.org/538070 (if horizon doesn't merge that dependant patch)
19:42:47 efried jaypipes, mriedem: The final parts of update_provider_tree going to Rocky presents an opportunity to write that design up as a separate blueprint/spec. It was never outlined in any of the placement/NRP specs (right Jay?) and it really ought to be. If you agree, I can get started on that.
19:43:33 efried (If you don't agree, I'm going to write it anyway, for my own use, and you don't get to see it.)
19:44:30 mriedem prometheanfire: i'm not sure how much i want to pursue that this late given the impact it also has to some CLIs in OSC: http://lists.openstack.org/pipermail/openstack-dev/2018-January/126741.html
19:45:40 prometheanfire mriedem: that's kinda what I thought
19:47:38 mriedem we'll just pick it up in rocky
19:47:47 mriedem nothing requires novaclient>=10.0.0 in queens
19:48:50 prometheanfire k
19:56:40 prometheanfire mriedem: k, gonna -2-W that for freeze then
19:58:00 mriedem prometheanfire: ok i left a comment in there that i'm cool with it
20:01:30 openstackgerrit Merged openstack/nova stable/pike: Set server status to ERROR if rebuild failed https://review.openstack.org/536897
20:06:25 melwitt more people are asking about https://review.openstack.org/340614 again, I've rewritten the commit message and added code comments to make it easier to review
20:08:16 mriedem i saw you dropped the revert history of shame
20:08:34 mriedem also, "people are talking" is a classic fox news tactic
20:08:41 mriedem name your sources mel
20:09:08 prometheanfire top
20:09:09 prometheanfire men
20:09:19 melwitt yeah, I had thought the history was important but I got the feeling no one could understand the point of the patch because of it
20:09:38 melwitt even I was getting confused between merge conflicts
20:09:39 prometheanfire https://www.reactiongifs.us/wp-content/uploads/2013/10/top_men_indiana_jones.gif
20:10:44 melwitt ayoung is asking about it today in #openstack-cinder
20:26:07 mriedem https://review.openstack.org/#/c/537933/ is finally merged, patches should flow much better through the gate now
20:27:45 prometheanfire mriedem: please have a piece of wood glued to your head :P
20:28:00 cfriesen melwitt: in the case of https://review.openstack.org/340614 why doesn't nova-compute do a more complete job of cleaning up at the time it sets the instance.host to None?
20:28:24 mriedem prometheanfire: ?
20:28:29 prometheanfire patches should flow much better through the gate now
20:28:53 mriedem cfriesen: like this? https://review.openstack.org/#/c/528385/
20:29:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional tests for traits-based scheduling https://review.openstack.org/539310
20:29:35 mriedem alex_xu: efried: ^
20:29:41 mriedem turned out it was pretty simple to write those
20:30:30 cfriesen mriedem: yep, and something similar for ports I guess.
20:34:56 openstackgerrit Matthew Edmonds proposed openstack/nova master: remove unnecessary conf imports https://review.openstack.org/539314
20:35:27 melwitt cfriesen: good question. looks like it tries to do something to cleanup volumes but it only does a volume delete if 'delete_on_termination' and doesn't do anything like detach volumes
20:35:51 melwitt so it seems like a better fix would be to properly handle cleanup in compute
20:37:21 mriedem melwitt: that's what ameeda's patch is trying to do
20:38:05 mriedem we do call _cleanup_allocated_networks when a build fails on the compute
20:38:10 mriedem which should cleanup ports
20:38:48 melwitt right
20:39:12 mriedem there could possibly be a bug there if we're using a stale network info cache
20:39:31 mriedem the _cleanup_volumes thing is definitely a gap in the compute cleanup code
20:39:34 mriedem if the volume is attached
20:39:48 melwitt yeah, seeing that
20:40:29 melwitt we're also missing a volume cleanup call for the InstanceNotFound block
20:40:44 mriedem well,
20:40:50 mriedem that means the instance was deleted concurrently
20:40:51 mriedem yeah?
20:41:03 mriedem which should go through the local delete in the API, or _shutdown_instance in the compute
20:41:51 melwitt yeah, concurrent
20:42:09 melwitt okay, just saw there's a cleanup ports call in there but no volumes call
20:42:58 openstackgerrit Merged openstack/nova stable/pike: Make sure that functional test triggered on sample changes https://review.openstack.org/536934
20:43:07 openstackgerrit Merged openstack/nova master: Fix a comment in a notification functional test https://review.openstack.org/536262
20:49:43 mriedem woot ^ should be able to start rechecking pike changes now too
20:50:25 efried jaypipes: All other things being equal, is there a reason you prefer a DELETE API over PUT {} ?
20:50:55 melwitt mriedem: how? did that resolve the libvirt keepalive timeout fails?
20:51:08 mriedem no
20:51:27 mriedem melwitt: you're right - it doesn't, need to backport the functional job timeout bump patch
20:51:29 mriedem want to propose that?
20:52:03 melwitt I can, but I was referring to the permafail of the cells v1 job on pike and ocata because of the libvirt keepalive timeout issue
20:52:38 mriedem melwitt: remember i removed the cellsv1 job from stable?
20:52:41 melwitt AFAIK we're still blocked on pike and ocata because of that
20:52:49 melwitt oh, I didn't know that. nevermind
20:52:58 mriedem https://review.openstack.org/#/c/538619/
20:53:34 mriedem if we want that job back, i think we have to define it in-tree per-branch and make it non-voting on stable
20:53:55 melwitt cool ... wish we knew what the root cause was, but there were no real leads
20:54:53 openstackgerrit sean mooney proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257
20:56:44 openstackgerrit melanie witt proposed openstack/nova stable/pike: Bumping functional test job timeouts https://review.openstack.org/539319
20:56:45 mriedem the python2.7 update on jan 18 and the fact we're not using pike uca packages is as close as i got to 'leads'
20:56:59 mriedem the latter is unique to master
20:57:57 melwitt I see. so we could try pulling pike uca for stable/pike but not for stable/ocata, right? I assume that wouldn't be okay because it's a mismatch
20:58:15 melwitt do we need the func test timeout bump for ocata too?
20:59:13 melwitt we don't have .zuul.yaml there
20:59:21 mriedem melwitt: yes on the last question, that needs to go on top of https://review.openstack.org/#/c/536936/
20:59:47 mriedem as for the former question, i tried that in https://review.openstack.org/#/c/536798/ but my patch to not run cellsv1 in stable landed first so the job didn't run on that patch
20:59:51 melwitt ah, okay. thanks
20:59:56 mriedem we'd have to do some depends-on chicanery there
21:00:03 melwitt gotcha
21:00:43 mriedem cfriesen: so on the perf thing, you just found out that compute was using more cpu than before?
21:01:02 mriedem that was likely because in newton the computes started posting inventory information to placement from every RT update
21:01:19 mriedem but you said server creates were taking a lot longer
21:03:37 openstackgerrit melanie witt proposed openstack/nova stable/ocata: Bumping functional test job timeouts https://review.openstack.org/539320
21:05:01 cfriesen mriedem: yeah, on same hardware instance creation on newton was 32 sec and pike was 230. but it's hard to break down exactly what's causing it since anything audit-driven will also show higher usage.
21:06:41 cfriesen mriedem: it should be noted this is on an all-in-one system, so the openstack services are constrained to only two CPUs, and those were pretty much pinned
21:06:50 cfriesen ie at 100% usage
21:09:14 mriedem cfriesen: can't run osprofiler + rally or something to see at least where the majority of the time is being spent?
21:12:28 cfriesen mriedem: we've got traces from intel's vtune analyzer showing how much time is spent where, but just about everything shows increases. there's no single smoking gun.

Earlier   Later