Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-27
16:49:06 johnthetubaguy its probably a warn
16:49:28 johnthetubaguy do you see the OVS bits wired up?
16:49:54 johnsom Sep 27 08:48:44 devstackpy27-2 nova-compute[21517]: WARNING nova.compute.manager [None req-48a86b9a-bf96-4f0e-bc60-00682c991e35 service nova] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Received unexpected event network-vif-plugged-37ea16ee-b9bc-48c8-b23b-1221bece7c9a for instance with vm_state active and task_state None.
16:50:13 johnthetubaguy how did you do the attach?
16:50:17 johnthetubaguy via the Nova API?
16:50:20 johnsom Yes
16:51:07 johnthetubaguy its probably a case of tracing that through the code following the logs, seeing where it failed, I suspect in n-cpu but it could have been earlier
16:52:11 johnsom Sep 27 08:48:44 devstackpy27-2 devstack@n-api.service[21452]: [pid: 21460|app: 0
16:52:11 johnsom |req: 28/58] 172.21.21.140 () {62 vars in 1337 bytes} [Wed Sep 27 08:48:38 2017]
16:52:11 johnsom POST /compute/v2.1/servers/fb013f87-2e20-42d7-950d-bc9add853f2c/os-interface =>
16:52:11 johnsom generated 280 bytes in 5420 msecs (HTTP/1.1 200) 9 headers in 359 bytes (1 swit
16:52:11 johnsom ches on core 0)
16:52:16 sdague efried: the list_opts thing is fine
16:53:09 johnsom Ok, well, I am going to attempt to collect world+dog logs and info to open a bug. Just wanted to ask if there were specific things I should look at while I have a "live" system.
16:53:57 johnthetubaguy johnsom: you probably want to trace it to this code:https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5242
16:54:14 johnthetubaguy johnsom: in the n-cpu logs
16:55:04 johnthetubaguy I was expecting to see this one I think: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5268
16:55:10 johnthetubaguy but sounds like you hit a different failure
16:55:23 johnthetubaguy probably in https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5253
16:56:04 johnthetubaguy looks like an exception in there would not get logged properly
16:57:35 johnthetubaguy johnsom: sadly that means you need to look through where it got in here: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L849
16:57:53 johnthetubaguy johnsom: best of luck!
16:58:04 johnsom Thanks!
17:00:22 johnsom Sep 27 08:48:39 devstackpy27-2 nova-compute[21517]: DEBUG nova.network.neutronv2.api [None req-cfaf7680-4a74-45a7-9dc8-fd793b93fde5 admin admin] [instance: fb013f87-2e20-42d7-950d-bc9add853f2c] Successfully updated port: 37ea16ee-b9bc-48c8-b23b-1221bece7c9a {{(pid=21517) _update_port /opt/stack/nova/nova/network/neutronv2/api.py:448}}
17:01:08 johnsom Yeah, this is going to take some time.
17:12:41 johnthetubaguy dansmith: traits for drivers that are not ironic, is it the driver that is meant to be reporting them upwards, or is that config, or both?
17:13:04 johnthetubaguy I guess I was meaning that as a more general question really
17:25:05 dansmith johnthetubaguy: at some point I think it'll be a little of both
17:25:13 mriedem could be an external service
17:25:21 dansmith johnthetubaguy: some things the compute manager probably adds to a list of virty things that the driver exposes
17:25:29 mriedem this reminds me, i was going to put something in our "nova is not a metrics gatherer" policy doc about this
17:25:42 mriedem because of the thing at the ptg where intel wanted nova-compute reporting some crazy cpu traits
17:25:46 johnthetubaguy its just for ironic it feels like the virt driver pushes those up from iroinc
17:26:03 mriedem i think ideally we don't want the ironic driver being a proxy to placement for this stuff
17:26:04 johnthetubaguy the problem is when an admin deletes a trait in ironic, how do we know to delete it in placement
17:26:13 dansmith johnthetubaguy: ironic could do it itself for sure
17:26:19 dansmith johnthetubaguy: for libvirt it'd be the virt driver
17:26:43 johnthetubaguy the problem is the nova creates the resource provider right now, using the compute node name, hashring details, etc
17:26:59 dansmith johnthetubaguy: no, the rp uuid is the ironic node uuid
17:27:15 dansmith johnthetubaguy: nova creates it if it's not there already, ironic could have done it
17:28:18 johnthetubaguy hmm, I thought it had both for some reason, I need to trace that all properly so its clear in my head
17:28:42 johnthetubaguy so I thought we said at the PTG the ironic virt driver would push this all up, but I am not totally against ironic doing that
17:29:08 Tengu hello!
17:29:41 Tengu I'm having some issues setting up host aggregation and flavor matching (i.e. "flavor m1.medium shall start only on that aggregate"
17:32:51 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146
17:34:23 openstackgerrit OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187
17:34:55 openstackgerrit melanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec https://review.openstack.org/507938
17:38:18 melwitt mriedem: ^ I wrote that test by working from nova/tests/functional/regressions/test_bug_1671648.py and just now realized I guess I could have just added an instance group to the existing test to also test this. but maybe it's better to have the tests separated
17:39:55 melwitt food for thought
17:40:26 mriedem dansmith: L135 https://etherpad.openstack.org/p/nova-instance-list are my results for 1000 active instances with your change
17:40:40 mriedem i'm pretty surprised at the improvements there
17:42:18 cdent mriedem: which job results on https://review.openstack.org/#/c/507918/ are my best target for pokage?
17:42:40 dansmith mriedem: hmm
17:43:01 dansmith mriedem: if you roll back to the other patch does it go back to the perf you measured before?
17:43:37 dansmith mriedem: with my patch we iterate the list fewer times
17:44:01 dansmith I'd be surprised if it made that much difference, but it should make some
17:44:25 mriedem can try that in a bit
17:44:44 dansmith my microversion survey results are interesting
17:44:46 dansmith and not good
17:44:57 dansmith will be done in a few minutes
17:45:16 mriedem cdent: i'd think just the normal tempest dsvm job http://logs.openstack.org/18/507918/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d0a5723/
17:45:27 cdent roger that
17:45:29 mriedem lots of copied bash in here so i likely screwed something up
17:46:36 mriedem hmm, didn't even get to my stuff
17:48:00 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687
17:49:28 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687
17:50:07 dansmith mriedem: check that out: https://imgur.com/a/2lmiw
17:50:11 dansmith sdague: you too ^
17:50:29 mriedem jesus, graphs?!
17:51:04 mriedem well, looking at 2.46 and 2.47, i think it's 2.47 https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id41
17:51:06 dansmith 2.26 was not free but 2.47 is killing us
17:51:07 dansmith yeah
17:51:41 mriedem this is 500 error/500 active right?
17:51:46 mriedem dansmith: want to report a bug with details?
17:51:51 dansmith mriedem: yes, 500/500
17:53:04 dansmith so that was on my patch
17:53:19 dansmith I'm going to run it again on master but starting at maybe 2.24 to make sure the knees are in the same place
17:53:26 mriedem ok
17:53:28 sdague dansmith: interesting.... any idea why the display on the embedded data is causing things to go nuts
17:53:34 mriedem i did notice this w/o your change too though
17:53:50 mriedem i wonder if we're lazy-loading the flavor extra specs?
17:53:58 dansmith mriedem: yeah, I don't see differences in my numbers so I'm just doing it for completeness
17:54:02 dansmith sdague: I haven't looked yet
17:54:04 sdague mriedem: ah, right, that's probably it
17:54:20 dansmith we shouldn't be lazy-loading extra specs
17:54:28 dansmith they should be in the flavor in the instance
17:55:56 sdague I mean, it is a bunch more data. I guess it could just be serialization cost of more data, though it seems weird.
17:56:34 mriedem well, we were always getting the flavor
17:56:35 dansmith mriedem: https://bugs.launchpad.net/nova/+bug/1719966
17:56:37 openstack Launchpad bug 1719966 in OpenStack Compute (nova) "Microversion 2.47 punches nova in its special place" [Undecided,New]
17:56:38 mriedem even before 2.27
17:56:43 mriedem ha
17:56:57 dansmith sdague: right, no difference in what we're pulling from the db across that boundary, just what we do with it in the api
17:57:36 dansmith sdague: (I checked)
17:57:40 mriedem https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L269
17:57:53 mriedem so we were always pulling it https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L263
17:58:15 mriedem and we were always joining on it in the db https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L58
17:58:26 mriedem so why is this so much slower? https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L248
17:58:33 mriedem the policy check for each instance?
17:59:42 dansmith the only thing we can lazy-load from flavor isprojects, BTW

Earlier   Later