Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-27
16:36:28 johnsom I have an instance booted in nova (master) that nova/neutron shows two plugged ports, but the kernel is not seeing the second network interface. It was hot-plugged with attach. Any pointers for debugging this?
16:37:19 johnsom the qemu process command line (ps -ef) only shows one interface, but I'm not sure if it should show a hot-plugged network interface or not.
16:37:43 johnsom We have been seeing this in our gates off and on during Pike, but I just had it happen local so I can debug, etc.
16:38:20 openstackgerrit Chris Dent proposed openstack/nova master: DNM: Don't monkey patch eventlet in functional https://review.openstack.org/506668
16:38:21 openstackgerrit Chris Dent proposed openstack/nova master: Do not monkey patch eventlet in unit tests https://review.openstack.org/507923
16:39:53 openstackgerrit Merged openstack/python-novaclient stable/pike: Allow boot server with multiple nics https://review.openstack.org/495901
16:46:14 johnthetubaguy johnsom: were there any nova-compute logs about why the attach failed?
16:46:38 johnsom I am looking through and collecting those, n-cpu?
16:47:43 johnthetubaguy yeah, thats the ones I think
16:47:45 johnsom nova show lists it as ACTIVE
16:48:20 johnthetubaguy it wouldn't got to an error state if it failed
16:48:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Support qemu >= 2.10 https://review.openstack.org/505673
16:48:52 johnthetubaguy AFAIK
16:48:57 johnsom Yeah, n-cpu doesn't have any ERROR level messages. I can see some vif lines related to the interface, but no ERROR
16:49:06 johnthetubaguy its probably a warn
16:49:28 johnthetubaguy do you see the OVS bits wired up?
16:49:54 johnsom Sep 27 08:48:44 devstackpy27-2 nova-compute[21517]: WARNING nova.compute.manager [None req-48a86b9a-bf96-4f0e-bc60-00682c991e35 service nova] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Received unexpected event network-vif-plugged-37ea16ee-b9bc-48c8-b23b-1221bece7c9a for instance with vm_state active and task_state None.
16:50:13 johnthetubaguy how did you do the attach?
16:50:17 johnthetubaguy via the Nova API?
16:50:20 johnsom Yes
16:51:07 johnthetubaguy its probably a case of tracing that through the code following the logs, seeing where it failed, I suspect in n-cpu but it could have been earlier
16:52:11 johnsom Sep 27 08:48:44 devstackpy27-2 devstack@n-api.service[21452]: [pid: 21460|app: 0
16:52:11 johnsom |req: 28/58] 172.21.21.140 () {62 vars in 1337 bytes} [Wed Sep 27 08:48:38 2017]
16:52:11 johnsom POST /compute/v2.1/servers/fb013f87-2e20-42d7-950d-bc9add853f2c/os-interface =>
16:52:11 johnsom generated 280 bytes in 5420 msecs (HTTP/1.1 200) 9 headers in 359 bytes (1 swit
16:52:11 johnsom ches on core 0)
16:52:16 sdague efried: the list_opts thing is fine
16:53:09 johnsom Ok, well, I am going to attempt to collect world+dog logs and info to open a bug. Just wanted to ask if there were specific things I should look at while I have a "live" system.
16:53:57 johnthetubaguy johnsom: you probably want to trace it to this code:https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5242
16:54:14 johnthetubaguy johnsom: in the n-cpu logs
16:55:04 johnthetubaguy I was expecting to see this one I think: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5268
16:55:10 johnthetubaguy but sounds like you hit a different failure
16:55:23 johnthetubaguy probably in https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5253
16:56:04 johnthetubaguy looks like an exception in there would not get logged properly
16:57:35 johnthetubaguy johnsom: sadly that means you need to look through where it got in here: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L849
16:57:53 johnthetubaguy johnsom: best of luck!
16:58:04 johnsom Thanks!
17:00:22 johnsom Sep 27 08:48:39 devstackpy27-2 nova-compute[21517]: DEBUG nova.network.neutronv2.api [None req-cfaf7680-4a74-45a7-9dc8-fd793b93fde5 admin admin] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Successfully updated port: 37ea16ee-b9bc-48c8-b23b-1221bece7c9a {{(pid=21517) _update_port /opt/stack/nova/nova/network/neutronv2/api.py:448}}
17:01:08 johnsom Yeah, this is going to take some time.
17:12:41 johnthetubaguy dansmith: traits for drivers that are not ironic, is it the driver that is meant to be reporting them upwards, or is that config, or both?
17:13:04 johnthetubaguy I guess I was meaning that as a more general question really
17:25:05 dansmith johnthetubaguy: at some point I think it'll be a little of both
17:25:13 mriedem could be an external service
17:25:21 dansmith johnthetubaguy: some things the compute manager probably adds to a list of virty things that the driver exposes
17:25:29 mriedem this reminds me, i was going to put something in our "nova is not a metrics gatherer" policy doc about this
17:25:42 mriedem because of the thing at the ptg where intel wanted nova-compute reporting some crazy cpu traits
17:25:46 johnthetubaguy its just for ironic it feels like the virt driver pushes those up from iroinc
17:26:03 mriedem i think ideally we don't want the ironic driver being a proxy to placement for this stuff
17:26:04 johnthetubaguy the problem is when an admin deletes a trait in ironic, how do we know to delete it in placement
17:26:13 dansmith johnthetubaguy: ironic could do it itself for sure
17:26:19 dansmith johnthetubaguy: for libvirt it'd be the virt driver
17:26:43 johnthetubaguy the problem is the nova creates the resource provider right now, using the compute node name, hashring details, etc
17:26:59 dansmith johnthetubaguy: no, the rp uuid is the ironic node uuid
17:27:15 dansmith johnthetubaguy: nova creates it if it's not there already, ironic could have done it
17:28:18 johnthetubaguy hmm, I thought it had both for some reason, I need to trace that all properly so its clear in my head
17:28:42 johnthetubaguy so I thought we said at the PTG the ironic virt driver would push this all up, but I am not totally against ironic doing that
17:29:08 Tengu hello!
17:29:41 Tengu I'm having some issues setting up host aggregation and flavor matching (i.e. "flavor m1.medium shall start only on that aggregate"
17:32:51 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146
17:34:23 openstackgerrit OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187
17:34:55 openstackgerrit melanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec https://review.openstack.org/507938
17:38:18 melwitt mriedem: ^ I wrote that test by working from nova/tests/functional/regressions/test_bug_1671648.py and just now realized I guess I could have just added an instance group to the existing test to also test this. but maybe it's better to have the tests separated
17:39:55 melwitt food for thought
17:40:26 mriedem dansmith: L135 https://etherpad.openstack.org/p/nova-instance-list are my results for 1000 active instances with your change
17:40:40 mriedem i'm pretty surprised at the improvements there
17:42:18 cdent mriedem: which job results on https://review.openstack.org/#/c/507918/ are my best target for pokage?
17:42:40 dansmith mriedem: hmm
17:43:01 dansmith mriedem: if you roll back to the other patch does it go back to the perf you measured before?
17:43:37 dansmith mriedem: with my patch we iterate the list fewer times
17:44:01 dansmith I'd be surprised if it made that much difference, but it should make some
17:44:25 mriedem can try that in a bit
17:44:44 dansmith my microversion survey results are interesting
17:44:46 dansmith and not good
17:44:57 dansmith will be done in a few minutes
17:45:16 mriedem cdent: i'd think just the normal tempest dsvm job http://logs.openstack.org/18/507918/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d0a5723/
17:45:27 cdent roger that
17:45:29 mriedem lots of copied bash in here so i likely screwed something up
17:46:36 mriedem hmm, didn't even get to my stuff
17:48:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687
17:49:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687
17:50:07 dansmith mriedem: check that out: https://imgur.com/a/2lmiw
17:50:11 dansmith sdague: you too ^
17:50:29 mriedem jesus, graphs?!
17:51:04 mriedem well, looking at 2.46 and 2.47, i think it's 2.47 https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id41
17:51:06 dansmith 2.26 was not free but 2.47 is killing us
17:51:07 dansmith yeah
17:51:41 mriedem this is 500 error/500 active right?
17:51:46 mriedem dansmith: want to report a bug with details?
17:51:51 dansmith mriedem: yes, 500/500
17:53:04 dansmith so that was on my patch
17:53:19 dansmith I'm going to run it again on master but starting at maybe 2.24 to make sure the knees are in the same place
17:53:26 mriedem ok
17:53:28 sdague dansmith: interesting.... any idea why the display on the embedded data is causing things to go nuts
17:53:34 mriedem i did notice this w/o your change too though
17:53:50 mriedem i wonder if we're lazy-loading the flavor extra specs?
17:53:58 dansmith mriedem: yeah, I don't see differences in my numbers so I'm just doing it for completeness
17:54:02 dansmith sdague: I haven't looked yet
17:54:04 sdague mriedem: ah, right, that's probably it
17:54:20 dansmith we shouldn't be lazy-loading extra specs
17:54:28 dansmith they should be in the flavor in the instance

Earlier   Later