Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-27
16:03:34 openstack Launchpad bug 1719915 in OpenStack Compute (nova) "test_live_migrate_delete race fail when checking allocations: MismatchError: 2 != 1" [Medium,In progress] - Assigned to Balazs Gibizer (balazs-gibizer)
16:04:36 mriedem thanks
16:04:59 cdent mriedem: I shall continue to remind the keepers of the CI
16:06:09 dansmith mriedem: 2.26 adds ~3s to my runtime
16:07:25 dansmith mriedem: 2.16 adds about 0.250s
16:08:06 cdent sdague: you’ve comment on the bug report related to https://review.openstack.org/#/c/501359/ , can you comment on the fix when you get a chance. _might_ have backport potential. stephenfin you willing to upgrade your +1?
16:08:27 dansmith mriedem: 2.26 was tags, btw
16:08:50 dansmith mriedem: so this is 3s with tags short-circuited at the db layer, so I think the 3s is all api overhead for empty things
16:09:38 stephenfin cdent: Yup, happy to +2 once someone sdague or dansmith has looked at it (I'm no expert in that area)
16:09:53 cdent thanks stephenfin
16:10:11 johnthetubaguy cdent: its normally dropped when the blueprint is approved, I don't remember how spec-less get approved now
16:10:35 cdent johnthetubaguy: s’okay, matt’s cleared things up: until CI is super happy the blueprint won’t get approved
16:10:54 johnthetubaguy cdent: ah, cool, I should read the scrollback better
16:10:54 cdent i hadn’t seen his comment in the middle of the stack
16:11:04 cdent and I should read the comments better :)
16:11:25 johnthetubaguy oh yeah, I see now
16:13:26 mriedem dansmith: cdent: this is my super hack devstack patch to try and recreate the 500 instance burst failure https://review.openstack.org/507918
16:16:18 mriedem dansmith: ok so we still don't know which microversion is making instance list go back through conductor
16:16:44 dansmith mriedem: I have conductor stopped and nothing else is failing
16:16:46 dansmith which I can't explain
16:17:31 mriedem hmm, api going straight to db somewhere?
16:17:48 dansmith api should be going straight to the db everywhere
16:18:03 dansmith I'm not sure why conductor was doing anything during a list in the first place
16:18:11 dansmith I tried turning it off to see what broke and nothing did
16:18:31 mriedem maybe you hit a window where a periodic was hitting conductor at the same time as you were doing the instance list?
16:18:40 dansmith could be, but I did it a few times
16:19:14 dansmith either way, I'm going to go measure the 2.26 impact with just my change (not the short-circuiting i've done) and on master and see what the diff is
16:20:46 mriedem i'm going to go preheat the oven because it's going to be pot pie time in about an hour
16:26:06 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fix race in delete allocation in ServerMovingTests https://review.openstack.org/507911
16:33:57 openstackgerrit Merged openstack/nova master: Fix IoOpsFilter test case class name. https://review.openstack.org/507205
16:34:37 openstackgerrit Merged openstack/nova master: libvirt: bandwidth param should be set in guest migrate https://review.openstack.org/497455
16:35:16 openstackgerrit Merged openstack/nova stable/ocata: Provide hints when nova-manage db sync fails to sync cell0 https://review.openstack.org/501746
16:35:38 openstackgerrit Merged openstack/nova master: Ensure errors_out_migration errors out migration https://review.openstack.org/479802
16:35:49 efried sdague got an opinion on https://review.openstack.org/#/c/488137/21/nova/conf/utils.py@85 ?
16:36:28 johnsom I have an instance booted in nova (master) that nova/neutron shows two plugged ports, but the kernel is not seeing the second network interface. It was hot-plugged with attach. Any pointers for debugging this?
16:37:19 johnsom the qemu process command line (ps -ef) only shows one interface, but I'm not sure if it should show a hot-plugged network interface or not.
16:37:43 johnsom We have been seeing this in our gates off and on during Pike, but I just had it happen local so I can debug, etc.
16:38:20 openstackgerrit Chris Dent proposed openstack/nova master: DNM: Don't monkey patch eventlet in functional https://review.openstack.org/506668
16:38:21 openstackgerrit Chris Dent proposed openstack/nova master: Do not monkey patch eventlet in unit tests https://review.openstack.org/507923
16:39:53 openstackgerrit Merged openstack/python-novaclient stable/pike: Allow boot server with multiple nics https://review.openstack.org/495901
16:46:14 johnthetubaguy johnsom: were there any nova-compute logs about why the attach failed?
16:46:38 johnsom I am looking through and collecting those, n-cpu?
16:47:43 johnthetubaguy yeah, thats the ones I think
16:47:45 johnsom nova show lists it as ACTIVE
16:48:20 johnthetubaguy it wouldn't got to an error state if it failed
16:48:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Support qemu >= 2.10 https://review.openstack.org/505673
16:48:52 johnthetubaguy AFAIK
16:48:57 johnsom Yeah, n-cpu doesn't have any ERROR level messages. I can see some vif lines related to the interface, but no ERROR
16:49:06 johnthetubaguy its probably a warn
16:49:28 johnthetubaguy do you see the OVS bits wired up?
16:49:54 johnsom Sep 27 08:48:44 devstackpy27-2 nova-compute[21517]: WARNING nova.compute.manager [None req-48a86b9a-bf96-4f0e-bc60-00682c991e35 service nova] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Received unexpected event network-vif-plugged-37ea16ee-b9bc-48c8-b23b-1221bece7c9a for instance with vm_state active and task_state None.
16:50:13 johnthetubaguy how did you do the attach?
16:50:17 johnthetubaguy via the Nova API?
16:50:20 johnsom Yes
16:51:07 johnthetubaguy its probably a case of tracing that through the code following the logs, seeing where it failed, I suspect in n-cpu but it could have been earlier
16:52:11 johnsom ches on core 0)
16:52:11 johnsom generated 280 bytes in 5420 msecs (HTTP/1.1 200) 9 headers in 359 bytes (1 swit
16:52:11 johnsom POST /compute/v2.1/servers/fb013f87-2e20-42d7-950d-bc9add853f2c/os-interface =>
16:52:11 johnsom |req: 28/58] 172.21.21.140 () {62 vars in 1337 bytes} [Wed Sep 27 08:48:38 2017]
16:52:11 johnsom Sep 27 08:48:44 devstackpy27-2 devstack@n-api.service[21452]: [pid: 21460|app: 0
16:52:16 sdague efried: the list_opts thing is fine
16:53:09 johnsom Ok, well, I am going to attempt to collect world+dog logs and info to open a bug. Just wanted to ask if there were specific things I should look at while I have a "live" system.
16:53:57 johnthetubaguy johnsom: you probably want to trace it to this code:https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5242
16:54:14 johnthetubaguy johnsom: in the n-cpu logs
16:55:04 johnthetubaguy I was expecting to see this one I think: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5268
16:55:10 johnthetubaguy but sounds like you hit a different failure
16:55:23 johnthetubaguy probably in https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5253
16:56:04 johnthetubaguy looks like an exception in there would not get logged properly
16:57:35 johnthetubaguy johnsom: sadly that means you need to look through where it got in here: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L849
16:57:53 johnthetubaguy johnsom: best of luck!
16:58:04 johnsom Thanks!
17:00:22 johnsom Sep 27 08:48:39 devstackpy27-2 nova-compute[21517]: DEBUG nova.network.neutronv2.api [None req-cfaf7680-4a74-45a7-9dc8-fd793b93fde5 admin admin] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Successfully updated port: 37ea16ee-b9bc-48c8-b23b-1221bece7c9a {{(pid=21517) _update_port /opt/stack/nova/nova/network/neutronv2/api.py:448}}
17:01:08 johnsom Yeah, this is going to take some time.
17:12:41 johnthetubaguy dansmith: traits for drivers that are not ironic, is it the driver that is meant to be reporting them upwards, or is that config, or both?
17:13:04 johnthetubaguy I guess I was meaning that as a more general question really
17:25:05 dansmith johnthetubaguy: at some point I think it'll be a little of both
17:25:13 mriedem could be an external service
17:25:21 dansmith johnthetubaguy: some things the compute manager probably adds to a list of virty things that the driver exposes
17:25:29 mriedem this reminds me, i was going to put something in our "nova is not a metrics gatherer" policy doc about this
17:25:42 mriedem because of the thing at the ptg where intel wanted nova-compute reporting some crazy cpu traits
17:25:46 johnthetubaguy its just for ironic it feels like the virt driver pushes those up from iroinc
17:26:03 mriedem i think ideally we don't want the ironic driver being a proxy to placement for this stuff
17:26:04 johnthetubaguy the problem is when an admin deletes a trait in ironic, how do we know to delete it in placement
17:26:13 dansmith johnthetubaguy: ironic could do it itself for sure
17:26:19 dansmith johnthetubaguy: for libvirt it'd be the virt driver
17:26:43 johnthetubaguy the problem is the nova creates the resource provider right now, using the compute node name, hashring details, etc
17:26:59 dansmith johnthetubaguy: no, the rp uuid is the ironic node uuid
17:27:15 dansmith johnthetubaguy: nova creates it if it's not there already, ironic could have done it
17:28:18 johnthetubaguy hmm, I thought it had both for some reason, I need to trace that all properly so its clear in my head
17:28:42 johnthetubaguy so I thought we said at the PTG the ironic virt driver would push this all up, but I am not totally against ironic doing that
17:29:08 Tengu hello!
17:29:41 Tengu I'm having some issues setting up host aggregation and flavor matching (i.e. "flavor m1.medium shall start only on that aggregate"
17:32:51 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146
17:34:23 openstackgerrit OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187
17:34:55 openstackgerrit melanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec https://review.openstack.org/507938
17:38:18 melwitt mriedem: ^ I wrote that test by working from nova/tests/functional/regressions/test_bug_1671648.py and just now realized I guess I could have just added an instance group to the existing test to also test this. but maybe it's better to have the tests separated
17:39:55 melwitt food for thought
17:40:26 mriedem dansmith: L135 https://etherpad.openstack.org/p/nova-instance-list are my results for 1000 active instances with your change
17:40:40 mriedem i'm pretty surprised at the improvements there
17:42:18 cdent mriedem: which job results on https://review.openstack.org/#/c/507918/ are my best target for pokage?

Earlier   Later