| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-27 | |||
| 15:14:18 | mriedem | there are paths in the RT that go into that code w/o consciously passing the has_ocata_computes flag | |
| 15:14:22 | mriedem | but i think it defaults to False anyway | |
| 15:15:13 | cdent | mriedem: have you got a set of logs you can make available? | |
| 15:15:18 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | |
| 15:16:02 | cdent | this doesn’t feel like something it’s going to be easy to reason about without some files to grep | |
| 15:16:48 | dansmith | cdent: it should be pretty easy to reproduce (or not) in a devstack and then you can instrument the code as needed | |
| 15:16:58 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | |
| 15:17:12 | mriedem | cdent: no, it's all local | |
| 15:17:19 | mriedem | well, in this devstack vm which is not local | |
| 15:17:28 | mriedem | but yeah i have the local.conf for the devstack if you want to reproduce | |
| 15:17:32 | cdent | mriedem: sure, but you have tar and such? | |
| 15:17:41 | mriedem | yeah | |
| 15:18:16 | mriedem | is there a standard way to tar up the journald logs? | |
| 15:18:33 | cdent | balls, I forgot about journald, meh | |
| 15:18:46 | mriedem | it's tar'ed up in devstack-gate | |
| 15:18:52 | mriedem | so i can just copy whatever we do in CI | |
| 15:19:46 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257 | |
| 15:19:47 | cdent | I can’t really look with any real attention until about 3 hours from now, but if you get a chance to do it, that’s great it will useful, if not, just the local.conf will do | |
| 15:19:57 | mriedem | https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L698-L724 | |
| 15:21:25 | mriedem | you know, i could just do this with a devstack patch | |
| 15:21:27 | mriedem | that's easier | |
| 15:21:34 | mriedem | let the ci do the work | |
| 15:25:19 | mriedem | needless to say, i'm doing a terrible job of reviewing code or specs, or writing specs | |
| 15:26:01 | mriedem | sdague: can you get this stable/pike novaclient bug fix backport? https://review.openstack.org/#/c/495901/ | |
| 15:26:12 | mriedem | pretty nasty and we need to release it | |
| 15:26:18 | sdague | mriedem: looking | |
| 15:26:55 | sdague | +A | |
| 15:27:32 | dansmith | mriedem: speaking of reviewing, I'm not sure what else to do on the base switchover patch since the difference is not measurable on my box. I'm poking at the fault thing in the later patch, but we can just strip that out and work on it in parallel | |
| 15:27:52 | mriedem | i haven't reviewed the actual change yet | |
| 15:28:04 | mriedem | was just getting 1000 active vms locally to test the fault thing we talked about last night | |
| 15:28:14 | mriedem | i agree that what we found last night, for numbers, isn't worth holding things up | |
| 15:28:28 | mriedem | sdague: thanks | |
| 15:28:34 | dansmith | mriedem: okay, like I said in the etherpad, I don't think we're doing the fault thing in that patch yet | |
| 15:28:47 | dansmith | and applying the one that does it definitely has an impact | |
| 15:29:15 | mriedem | negative impact? | |
| 15:29:37 | dansmith | yeah | |
| 15:30:05 | mriedem | because we've added a new unconditional join i suppose | |
| 15:30:19 | dansmith | it's not a new join, but it's a new query yeah | |
| 15:30:25 | mriedem | yeah, was just thinking that | |
| 15:30:43 | dansmith | I'm messing with the later patch though to see if I can squash it out | |
| 15:30:45 | mriedem | like you said, we could plumb that in the db api | |
| 15:30:53 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/newton: Updated from global requirements https://review.openstack.org/373293 | |
| 15:31:05 | mriedem | if faults is in expected_attrs, get the instances in deleted/error and include the faults on those? | |
| 15:31:05 | dansmith | that's what I'm doing yeah | |
| 15:32:04 | dansmith | yes | |
| 15:36:47 | cdent | do dansmith and mriedem have an exciting new etherpad they are willing to share? | |
| 15:37:06 | dansmith | cdent: https://etherpad.openstack.org/p/nova-instance-list | |
| 15:37:12 | dansmith | cdent: but that's just incidental to him finding the problem | |
| 15:37:21 | dansmith | it's not about the thing we were describing | |
| 15:37:35 | cdent | you know I’m a total junkie for all the info | |
| 15:37:55 | mriedem | started as getting a benchmark for testing before and after the big instance list change | |
| 15:38:10 | mriedem | but have also added todos for weird things we've seen, like the concurrent update detected thing with 500 instances | |
| 15:42:09 | mriedem | sdague: can i put [[post-config|$NOVA_CONF]] in stackrc? | |
| 15:42:44 | cdent | mriedem, dansmith, (and sean mooney): If any of you get a chance to look at the discussion on https://review.openstack.org/#/c/504540/ (limiting GET /allocation_candidates ) it’s gotten to the point where we are trying to decide what it is that we are actually optimizing for, so could do with more input | |
| 15:43:17 | sdague | mriedem: in local.conf | |
| 15:43:42 | mriedem | sdague: yeah, locally, but this is for running something through ci | |
| 15:44:51 | sdague | there isn't really anyway to jam it into stackrc, you need to do project-config changes for stuff like that | |
| 15:45:15 | sdague | if this is just for hacktastic stuff, just iniset whatever you want in lib/nova | |
| 15:45:16 | mriedem | ok, just checking, i can hack this other ways | |
| 15:45:21 | mriedem | yeah that's what i'm doing | |
| 15:55:01 | dansmith | mriedem: so with /servers/details on 2.53, we're pegging conductor real hard. With 2.1 there is no conductor interaction at all | |
| 15:55:13 | dansmith | so we must be doing something stupid in the later microversion | |
| 15:55:19 | dansmith | because we shouldn't be making rpc calls at all | |
| 15:55:47 | dansmith | also, defeating all fault loading and all tag loading doesn't make anything faster | |
| 15:56:31 | mriedem | how about services? | |
| 15:57:43 | dansmith | services is included in the 2.1 columns so I didn't try | |
| 15:59:04 | cdent | johnthetubaguy: you still want to hold your -2 on https://review.openstack.org/#/c/270116/ ? It’s got a blueprint now | |
| 16:00:41 | mriedem | oh yeah | |
| 16:00:49 | melwitt | mriedem: I'm working on the regression func test for the reschedule bug, FYI | |
| 16:00:51 | dansmith | however, I can stop compute and conductor and still do the list with no failure (and no difference in speed) | |
| 16:00:55 | dansmith | wtaf | |
| 16:01:12 | mriedem | cdent: the spec on that isn't approved | |
| 16:01:53 | mriedem | cdent: see my comment from may 26 on that patch | |
| 16:02:48 | cdent | mriedem: okay | |
| 16:03:13 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix race in delete allocation in ServerMovingTests https://review.openstack.org/507911 | |
| 16:03:31 | gibi | mriedem: I've pushed a fix for https://bugs.launchpad.net/nova/+bug/1719915 ^^ | |
| 16:03:34 | openstack | Launchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,In progress] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 16:04:36 | mriedem | thanks | |
| 16:04:59 | cdent | mriedem: I shall continue to remind the keepers of the CI | |
| 16:06:09 | dansmith | mriedem: 2.26 adds ~3s to my runtime | |
| 16:07:25 | dansmith | mriedem: 2.16 adds about 0.250s | |
| 16:08:06 | cdent | sdague: you’ve comment on the bug report related to https://review.openstack.org/#/c/501359/ , can you comment on the fix when you get a chance. _might_ have backport potential. stephenfin you willing to upgrade your +1? | |
| 16:08:27 | dansmith | mriedem: 2.26 was tags, btw | |
| 16:08:50 | dansmith | mriedem: so this is 3s with tags short-circuited at the db layer, so I think the 3s is all api overhead for empty things | |
| 16:09:38 | stephenfin | cdent: Yup, happy to +2 once someone sdague or dansmith has looked at it (I'm no expert in that area) | |
| 16:09:53 | cdent | thanks stephenfin | |
| 16:10:11 | johnthetubaguy | cdent: its normally dropped when the blueprint is approved, I don't remember how spec-less get approved now | |
| 16:10:35 | cdent | johnthetubaguy: s’okay, matt’s cleared things up: until CI is super happy the blueprint won’t get approved | |
| 16:10:54 | johnthetubaguy | cdent: ah, cool, I should read the scrollback better | |
| 16:10:54 | cdent | i hadn’t seen his comment in the middle of the stack | |
| 16:11:04 | cdent | and I should read the comments better :) | |
| 16:11:25 | johnthetubaguy | oh yeah, I see now | |
| 16:13:26 | mriedem | dansmith: cdent: this is my super hack devstack patch to try and recreate the 500 instance burst failure https://review.openstack.org/507918 | |
| 16:16:18 | mriedem | dansmith: ok so we still don't know which microversion is making instance list go back through conductor | |
| 16:16:44 | dansmith | mriedem: I have conductor stopped and nothing else is failing | |
| 16:16:46 | dansmith | which I can't explain | |
| 16:17:31 | mriedem | hmm, api going straight to db somewhere? | |
| 16:17:48 | dansmith | api should be going straight to the db everywhere | |
| 16:18:03 | dansmith | I'm not sure why conductor was doing anything during a list in the first place | |
| 16:18:11 | dansmith | I tried turning it off to see what broke and nothing did | |