Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-27
15:42:44 cdent mriedem, dansmith, (and sean mooney): If any of you get a chance to look at the discussion on https://review.openstack.org/#/c/504540/ (limiting GET /allocation_candidates ) it’s gotten to the point where we are trying to decide what it is that we are actually optimizing for, so could do with more input
15:43:17 sdague mriedem: in local.conf
15:43:42 mriedem sdague: yeah, locally, but this is for running something through ci
15:44:51 sdague there isn't really anyway to jam it into stackrc, you need to do project-config changes for stuff like that
15:45:15 sdague if this is just for hacktastic stuff, just iniset whatever you want in lib/nova
15:45:16 mriedem ok, just checking, i can hack this other ways
15:45:21 mriedem yeah that's what i'm doing
15:55:01 dansmith mriedem: so with /servers/details on 2.53, we're pegging conductor real hard. With 2.1 there is no conductor interaction at all
15:55:13 dansmith so we must be doing something stupid in the later microversion
15:55:19 dansmith because we shouldn't be making rpc calls at all
15:55:47 dansmith also, defeating all fault loading and all tag loading doesn't make anything faster
15:56:31 mriedem how about services?
15:57:43 dansmith services is included in the 2.1 columns so I didn't try
15:59:04 cdent johnthetubaguy: you still want to hold your -2 on https://review.openstack.org/#/c/270116/ ? It’s got a blueprint now
16:00:41 mriedem oh yeah
16:00:49 melwitt mriedem: I'm working on the regression func test for the reschedule bug, FYI
16:00:51 dansmith however, I can stop compute and conductor and still do the list with no failure (and no difference in speed)
16:00:55 dansmith wtaf
16:01:12 mriedem cdent: the spec on that isn't approved
16:01:53 mriedem cdent: see my comment from may 26 on that patch
16:02:48 cdent mriedem: okay
16:03:13 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fix race in delete allocation in ServerMovingTests https://review.openstack.org/507911
16:03:31 gibi mriedem: I've pushed a fix for https://bugs.launchpad.net/nova/+bug/1719915 ^^
16:03:34 openstack Launchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,In progress] - Assigned to Balazs Gibizer (balazs-gibizer)
16:04:36 mriedem thanks
16:04:59 cdent mriedem: I shall continue to remind the keepers of the CI
16:06:09 dansmith mriedem: 2.26 adds ~3s to my runtime
16:07:25 dansmith mriedem: 2.16 adds about 0.250s
16:08:06 cdent sdague: you’ve comment on the bug report related to https://review.openstack.org/#/c/501359/ , can you comment on the fix when you get a chance. _might_ have backport potential. stephenfin you willing to upgrade your +1?
16:08:27 dansmith mriedem: 2.26 was tags, btw
16:08:50 dansmith mriedem: so this is 3s with tags short-circuited at the db layer, so I think the 3s is all api overhead for empty things
16:09:38 stephenfin cdent: Yup, happy to +2 once someone sdague or dansmith has looked at it (I'm no expert in that area)
16:09:53 cdent thanks stephenfin
16:10:11 johnthetubaguy cdent: its normally dropped when the blueprint is approved, I don't remember how spec-less get approved now
16:10:35 cdent johnthetubaguy: s’okay, matt’s cleared things up: until CI is super happy the blueprint won’t get approved
16:10:54 cdent i hadn’t seen his comment in the middle of the stack
16:10:54 johnthetubaguy cdent: ah, cool, I should read the scrollback better
16:11:04 cdent and I should read the comments better :)
16:11:25 johnthetubaguy oh yeah, I see now
16:13:26 mriedem dansmith: cdent: this is my super hack devstack patch to try and recreate the 500 instance burst failure https://review.openstack.org/507918
16:16:18 mriedem dansmith: ok so we still don't know which microversion is making instance list go back through conductor
16:16:44 dansmith mriedem: I have conductor stopped and nothing else is failing
16:16:46 dansmith which I can't explain
16:17:31 mriedem hmm, api going straight to db somewhere?
16:17:48 dansmith api should be going straight to the db everywhere
16:18:03 dansmith I'm not sure why conductor was doing anything during a list in the first place
16:18:11 dansmith I tried turning it off to see what broke and nothing did
16:18:31 mriedem maybe you hit a window where a periodic was hitting conductor at the same time as you were doing the instance list?
16:18:40 dansmith could be, but I did it a few times
16:19:14 dansmith either way, I'm going to go measure the 2.26 impact with just my change (not the short-circuiting i've done) and on master and see what the diff is
16:20:46 mriedem i'm going to go preheat the oven because it's going to be pot pie time in about an hour
16:26:06 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fix race in delete allocation in ServerMovingTests https://review.openstack.org/507911
16:33:57 openstackgerrit Merged openstack/nova master: Fix IoOpsFilter test case class name. https://review.openstack.org/507205
16:34:37 openstackgerrit Merged openstack/nova master: libvirt: bandwidth param should be set in guest migrate https://review.openstack.org/497455
16:35:16 openstackgerrit Merged openstack/nova stable/ocata: Provide hints when nova-manage db sync fails to sync cell0 https://review.openstack.org/501746
16:35:38 openstackgerrit Merged openstack/nova master: Ensure errors_out_migration errors out migration https://review.openstack.org/479802
16:35:49 efried sdague got an opinion on https://review.openstack.org/#/c/488137/21/nova/conf/utils.py@85 ?
16:36:28 johnsom I have an instance booted in nova (master) that nova/neutron shows two plugged ports, but the kernel is not seeing the second network interface. It was hot-plugged with attach. Any pointers for debugging this?
16:37:19 johnsom the qemu process command line (ps -ef) only shows one interface, but I'm not sure if it should show a hot-plugged network interface or not.
16:37:43 johnsom We have been seeing this in our gates off and on during Pike, but I just had it happen local so I can debug, etc.
16:38:20 openstackgerrit Chris Dent proposed openstack/nova master: DNM: Don't monkey patch eventlet in functional https://review.openstack.org/506668
16:38:21 openstackgerrit Chris Dent proposed openstack/nova master: Do not monkey patch eventlet in unit tests https://review.openstack.org/507923
16:39:53 openstackgerrit Merged openstack/python-novaclient stable/pike: Allow boot server with multiple nics https://review.openstack.org/495901
16:46:14 johnthetubaguy johnsom: were there any nova-compute logs about why the attach failed?
16:46:38 johnsom I am looking through and collecting those, n-cpu?
16:47:43 johnthetubaguy yeah, thats the ones I think
16:47:45 johnsom nova show lists it as ACTIVE
16:48:20 johnthetubaguy it wouldn't got to an error state if it failed
16:48:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Support qemu >= 2.10 https://review.openstack.org/505673
16:48:52 johnthetubaguy AFAIK
16:48:57 johnsom Yeah, n-cpu doesn't have any ERROR level messages. I can see some vif lines related to the interface, but no ERROR
16:49:06 johnthetubaguy its probably a warn
16:49:28 johnthetubaguy do you see the OVS bits wired up?
16:49:54 johnsom Sep 27 08:48:44 devstackpy27-2 nova-compute[21517]: WARNING nova.compute.manager [None req-48a86b9a-bf96-4f0e-bc60-00682c991e35 service nova] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Received unexpected event network-vif-plugged-37ea16ee-b9bc-48c8-b23b-1221bece7c9a for instance with vm_state active and task_state None.
16:50:13 johnthetubaguy how did you do the attach?
16:50:17 johnthetubaguy via the Nova API?
16:50:20 johnsom Yes
16:51:07 johnthetubaguy its probably a case of tracing that through the code following the logs, seeing where it failed, I suspect in n-cpu but it could have been earlier
16:52:11 johnsom Sep 27 08:48:44 devstackpy27-2 devstack@n-api.service[21452]: [pid: 21460|app: 0
16:52:11 johnsom |req: 28/58] 172.21.21.140 () {62 vars in 1337 bytes} [Wed Sep 27 08:48:38 2017]
16:52:11 johnsom POST /compute/v2.1/servers/fb013f87-2e20-42d7-950d-bc9add853f2c/os-interface =>
16:52:11 johnsom generated 280 bytes in 5420 msecs (HTTP/1.1 200) 9 headers in 359 bytes (1 swit
16:52:11 johnsom ches on core 0)
16:52:16 sdague efried: the list_opts thing is fine
16:53:09 johnsom Ok, well, I am going to attempt to collect world+dog logs and info to open a bug. Just wanted to ask if there were specific things I should look at while I have a "live" system.
16:53:57 johnthetubaguy johnsom: you probably want to trace it to this code:https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5242
16:54:14 johnthetubaguy johnsom: in the n-cpu logs
16:55:04 johnthetubaguy I was expecting to see this one I think: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5268
16:55:10 johnthetubaguy but sounds like you hit a different failure
16:55:23 johnthetubaguy probably in https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5253
16:56:04 johnthetubaguy looks like an exception in there would not get logged properly
16:57:35 johnthetubaguy johnsom: sadly that means you need to look through where it got in here: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L849
16:57:53 johnthetubaguy johnsom: best of luck!
16:58:04 johnsom Thanks!
17:00:22 johnsom Sep 27 08:48:39 devstackpy27-2 nova-compute[21517]: DEBUG nova.network.neutronv2.api [None req-cfaf7680-4a74-45a7-9dc8-fd793b93fde5 admin admin] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Successfully updated port: 37ea16ee-b9bc-48c8-b23b-1221bece7c9a {{(pid=21517) _update_port /opt/stack/nova/nova/network/neutronv2/api.py:448}}
17:01:08 johnsom Yeah, this is going to take some time.
17:12:41 johnthetubaguy dansmith: traits for drivers that are not ironic, is it the driver that is meant to be reporting them upwards, or is that config, or both?
17:13:04 johnthetubaguy I guess I was meaning that as a more general question really
17:25:05 dansmith johnthetubaguy: at some point I think it'll be a little of both
17:25:13 mriedem could be an external service

Earlier   Later