| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-30 | |||
| 20:54:53 | openstackgerrit | sean mooney proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257 | |
| 20:56:44 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Bumping functional test job timeouts https://review.openstack.org/539319 | |
| 20:56:45 | mriedem | the python2.7 update on jan 18 and the fact we're not using pike uca packages is as close as i got to 'leads' | |
| 20:56:59 | mriedem | the latter is unique to master | |
| 20:57:57 | melwitt | I see. so we could try pulling pike uca for stable/pike but not for stable/ocata, right? I assume that wouldn't be okay because it's a mismatch | |
| 20:58:15 | melwitt | do we need the func test timeout bump for ocata too? | |
| 20:59:13 | melwitt | we don't have .zuul.yaml there | |
| 20:59:21 | mriedem | melwitt: yes on the last question, that needs to go on top of https://review.openstack.org/#/c/536936/ | |
| 20:59:47 | mriedem | as for the former question, i tried that in https://review.openstack.org/#/c/536798/ but my patch to not run cellsv1 in stable landed first so the job didn't run on that patch | |
| 20:59:51 | melwitt | ah, okay. thanks | |
| 20:59:56 | mriedem | we'd have to do some depends-on chicanery there | |
| 21:00:03 | melwitt | gotcha | |
| 21:00:43 | mriedem | cfriesen: so on the perf thing, you just found out that compute was using more cpu than before? | |
| 21:01:02 | mriedem | that was likely because in newton the computes started posting inventory information to placement from every RT update | |
| 21:01:19 | mriedem | but you said server creates were taking a lot longer | |
| 21:03:37 | openstackgerrit | melanie witt proposed openstack/nova stable/ocata: Bumping functional test job timeouts https://review.openstack.org/539320 | |
| 21:05:01 | cfriesen | mriedem: yeah, on same hardware instance creation on newton was 32 sec and pike was 230. but it's hard to break down exactly what's causing it since anything audit-driven will also show higher usage. | |
| 21:06:41 | cfriesen | mriedem: it should be noted this is on an all-in-one system, so the openstack services are constrained to only two CPUs, and those were pretty much pinned | |
| 21:06:50 | cfriesen | ie at 100% usage | |
| 21:09:14 | mriedem | cfriesen: can't run osprofiler + rally or something to see at least where the majority of the time is being spent? | |
| 21:12:28 | cfriesen | mriedem: we've got traces from intel's vtune analyzer showing how much time is spent where, but just about everything shows increases. there's no single smoking gun. | |
| 21:14:53 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make the InstanceMapping marker UUID-like https://review.openstack.org/539323 | |
| 21:16:04 | mriedem | cfriesen: huh | |
| 21:18:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Make generation optional in ProviderTree https://review.openstack.org/539324 | |
| 21:20:48 | mriedem | melwitt: looks like you have real test failures in https://review.openstack.org/#/c/539013/ | |
| 21:21:42 | melwitt | mriedem: ah, thank you. I hadn't looked at the detail yet. I shall fix that up | |
| 21:21:47 | sean-k-mooney | cfriesen: dumb question but could it be related to meltdow/specter patches? | |
| 21:23:42 | cfriesen | sean-k-mooney: not so dumb. :) but no, this was a load from before the meltdown/spectre patches were applied. | |
| 21:25:40 | mriedem | cfriesen: and you've got the latest stable/pike release? | |
| 21:26:28 | mriedem | like i was wondering if any of the RequestContext changes might be related https://github.com/openstack/nova/commits/stable/pike/nova/context.py | |
| 21:26:34 | mriedem | since the context is used everywhere | |
| 21:27:51 | cfriesen | mriedem: not the latest, no. originally from 16.0.2, with some stable/pike stuff since then. not sure exactly what, I've been busy with other stuff. | |
| 21:28:46 | mriedem | cfriesen: ok would be good to know if it's still the same issue after you've got the latest pike fixes applied | |
| 21:28:57 | mriedem | https://docs.openstack.org/releasenotes/nova/pike.html | |
| 21:30:15 | mriedem | full change log http://paste.openstack.org/show/658066/ | |
| 21:30:46 | cfriesen | don't see anything related to "context" in there, got any pointers? | |
| 21:31:52 | cfriesen | last stable/pike changes to nova/context.py were from October | |
| 21:32:27 | mriedem | nothing in that changelog jumps out at me as a perf related fix | |
| 21:34:35 | cfriesen | does the upstream CI environment do performance tests of common operations? | |
| 21:35:56 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove compute nodes arg from ProviderTree init https://review.openstack.org/539330 | |
| 21:38:31 | mriedem | cfriesen: no | |
| 21:38:44 | mriedem | too much variance from node to node | |
| 21:39:04 | mriedem | not sure what can be pulled perf-trend wise from openstack-health | |
| 21:39:32 | mriedem | http://status.openstack.org/openstack-health/#/ | |
| 21:40:30 | mriedem | like, i don't know how to take that and see how long a simple tempest create server test has taken over the last 12 monhts | |
| 21:40:32 | mriedem | *months | |
| 21:40:36 | mriedem | mtreinish can maybe help | |
| 21:40:57 | openstackgerrit | melanie witt proposed openstack/nova stable/ocata: Stop globally caching host states in scheduler HostManager https://review.openstack.org/539013 | |
| 21:49:05 | mriedem | mmedvede: powerkvm ci seems pretty unhappy | |
| 21:49:07 | mriedem | is that a known issue? | |
| 21:51:30 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Provide support matrix and doc for VGPU https://review.openstack.org/539266 | |
| 21:56:07 | mriedem | dansmith: want to hit these backports? I didn't realize those weren't merged by now https://review.openstack.org/#/q/Ie70c77db753711e1449e99534d3b83669871943f+status:open | |
| 21:56:48 | mmedvede | mriedem: I do not see anything too far out of ordinary, which powerkvm ci unhappiness are you referring to? | |
| 21:57:00 | mmedvede | double checking now | |
| 21:57:12 | mriedem | mmedvede: https://review.openstack.org/#/c/538510/ | |
| 21:57:18 | mriedem | https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/10/538510/3/check/tempest-dsvm-full-xenial/41f1c6d/ | |
| 22:01:15 | edleafe | efried: PUT {} is not semantically the same as DELETE, even if in most cases the result is the same | |
| 22:04:15 | mriedem | cfriesen: speaking of perf, this is an easy fix for an RT perf issue if you're building several instances on the same compute host at once https://review.openstack.org/#/q/Ib588c31a4d2075f8730409d50c99dfb04180a9cd+status:open | |
| 22:04:32 | mriedem | our operations people were hitting perf issues with the big RT update lock | |
| 22:09:08 | dansmith | mriedem: got em | |
| 22:10:05 | efried | edleafe: Oh? How not? | |
| 22:10:26 | efried | edleafe: Oh, you mean in the general case, where None and {} aren't the same thing. | |
| 22:10:50 | prometheanfire | win 30 | |
| 22:11:51 | edleafe | efried: No. Sometimes you need to indicate if anything was in fact deleted. PUT {} can't do that; DELETE can | |
| 22:12:12 | efried | edleafe: It can? How? | |
| 22:12:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Pass the correct image to build_request_spec in conductor.rebuild_instance https://review.openstack.org/516404 | |
| 22:14:12 | mmedvede | mriedem: that pkvmci failure seems to have been a fluke on that VM, one of a kind. Sorry for the false negative. | |
| 22:14:56 | edleafe | efried: In cases where that distinction is important, you can return a 404 if the thing you're trying to delete is not there | |
| 22:15:17 | sean-k-mooney | the ibm powerkvm ci is still broken currently correct? | |
| 22:15:25 | edleafe | efried: Like I said, it isn't usually necessary. When you want to delete something, you usually just want it gone. | |
| 22:15:58 | efried | edleafe: Okay, then I'm specifically talking about resource provider inventories, traits, allocations, and aggregates. | |
| 22:16:42 | efried | edleafe: I can provide a good argument for why we should favor PUT <empty> over DELETE (at least DELETE as currently implemented). I'm trying to figure out if there's an argument for the other side. | |
| 22:16:49 | edleafe | efried: for those cases, I simply prefer the grammar of DELETE | |
| 22:17:07 | mmedvede | sean-k-mooney: I am confused as to why you think so. It did have a few failures, but failure rate is within normal | |
| 22:17:26 | efried | edleafe: The grammar of the request (as opposed to the (lack of) response), right? | |
| 22:17:28 | edleafe | efried: That's what DELETE is designed to do. | |
| 22:17:52 | edleafe | efried: No, PUT {} is an awkward way of saying DELETE | |
| 22:18:49 | efried | edleafe: Yeah, I get it. Is it "wrong" (in the annals of HTTP, or REST, or APIs, or whatever) for a DELETE API to return a payload? | |
| 22:20:30 | sean-k-mooney | efried: yes i belive it is not ment to have a payload generally | |
| 22:20:46 | edleafe | efried: Unless the response is 204. | |
| 22:21:01 | mmedvede | sean-k-mooney: am I missing something? Both http://ci-watch.tintri.com/project?project=nova and https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/index.html do not indicate a systemic failure on nova patches | |
| 22:21:09 | mmedvede | for powerkvm ci | |
| 22:21:40 | sean-k-mooney | mmedvede: there was a message a week or two saying it was i was not sure if it was fixed or not | |
| 22:21:56 | edleafe | efried: Othewise, you can return either a description of the deleted resource (200) or a URL to check for success/failure on a 202. | |
| 22:22:28 | efried | edleafe: So it'd be acceptable for a DELETE to return 200 with a payload? | |
| 22:23:01 | edleafe | efried: it's required | |
| 22:23:07 | mriedem | sean-k-mooney: are you thinking of the zvm ci? | |
| 22:23:23 | mriedem | or zkvm i mean | |
| 22:23:58 | efried | edleafe: Sorry, I mean I get that it's cool for DELETE to respond 204 with no content; I'm asking whether there's any restriction - standard-wise or cultural - against a DELETE responding 200 with a payload. | |
| 22:24:07 | mriedem | mmedvede: i had seen some other pkvm ci failures in stable branches, but those might have been old/transient | |
| 22:24:11 | efried | and I think you've said that's acceptable. | |
| 22:24:46 | edleafe | efried: yes, it's acceptable, although it isn't very common | |
| 22:26:14 | sean-k-mooney | mriedem: yes i was https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg115082.html | |
| 22:27:29 | mriedem | sean-k-mooney: mixing up one of the dozen ibm 3rd party CIs is grounds for pistols at dawn | |
| 22:28:02 | mmedvede | mriedem: yes, stable branches have high rate of failure unfortunately, I'll shift some time to look at those. | |
| 22:28:53 | sean-k-mooney | mriedem: haha well the grenade job is failing because of a ubuntu keyring missing on the powervm ci too but that could be intermitent | |
| 22:29:33 | mmedvede | sean-k-mooney: that is intermittent, there is a bug in ubuntu somewhere that we reported | |
| 22:29:59 | mmedvede | it happens only last 10 minutes of any hour | |
| 22:30:21 | sean-k-mooney | mmedvede: ya the patch i noticed it on is for rocky anyway so im not going to waste ci time rechecking | |
| 22:32:47 | sean-k-mooney | anyway i have fixed my unrelated ovs db socket somehow became a directory and broke everything issue with kolla so im going to head home for the evening | |