Earlier  
Posted Nick Remark
#openstack-nova - 2017-08-01
17:42:50 mriedem dansmith: ^ destroys unit tests for the objects b/c of the fake indirection api, but is that something that's even possible? ^
17:43:26 mriedem i want to get rid of these http://logs.openstack.org/54/487954/5/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/4412f5d/logs/screen-n-cpu.txt.gz?level=TRACE#_Jul_28_19_41_38_466972
17:43:37 dansmith mriedem: no because those aren't actually the rpc calls
17:43:45 cdent Is stephenfin around this week, I sense his absence
17:43:46 dansmith well, I'm guessing that won't work because ^
17:43:57 mriedem cdent: i'm told he's back tomorrow
17:44:04 cdent thanks mriedem
17:45:48 dansmith mriedem: that line you quote is intentional log.warning in the libvirt driver
17:46:02 dansmith mriedem: and it looks like we're getting a stack trace automagically from the logging layer (?)
17:46:05 mriedem i know, but the trace is coming from oslo.messaging isn't it?
17:46:11 dansmith no I don't think so
17:47:06 dansmith mriedem: see the line below.. you need exc_info=False maybe ?
17:47:09 dansmith maybe a default changed somehow?
17:47:12 dansmith oh
17:47:14 dansmith ooooh
17:47:17 dansmith you're hitting the lower one
17:47:19 dansmith with exc_info=True
17:47:25 dansmith because it is _Remote
17:47:52 mriedem dansmith: yeah there is that,
17:47:54 dansmith so maybe in that handler, check the exception name and if it looks remoteish then you can not pass the exc_info flag?
17:47:57 mriedem but that doesn't explain this one http://logs.openstack.org/54/487954/5/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/4412f5d/logs/screen-n-cpu.txt.gz?level=TRACE#_Jul_28_19_41_40_927881
17:47:59 dansmith it's a little icky
17:48:33 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6624
17:48:40 mriedem oh well,
17:48:51 mriedem maybe we don't hit that code b/c of the same thing?
17:49:45 dansmith well, you log that message, so it seems like you are
17:49:50 dansmith not sure why you get a trace there then
17:49:53 mriedem but where does the traceback come from?
17:49:54 mriedem right
17:50:28 dansmith unless a default has changed and oslo now logs a trace if you're in exception context
17:51:22 dansmith boy that sure is ugly
17:51:51 mriedem been this way since ocata
17:52:13 trinaths hi, I'm hitting this error, when i start nova-compute, http://paste.openstack.org/show/617156/
17:52:16 openstackgerrit Sumedh Degaonkar proposed openstack/nova master: Adds 'sata' as a valid disk bus for qemu and kvm hypervisors https://review.openstack.org/459794
17:52:22 trinaths please help me resolve the same.
17:58:39 sdague mriedem: actually, looking at sfinucan's split up of docs, I'm really not convinced that https://review.openstack.org/#/c/478485/ is a good move
18:00:17 mriedem http://docs-draft.openstack.org/85/478485/16/check/gate-nova-docs-ubuntu-xenial/481415d//doc/build/html/
18:00:17 mriedem sdague: ... i said it was a massive change
18:00:36 mriedem compared to how things used to be https://docs.openstack.org/nova/ocata/
18:00:37 sdague mriedem: yeh
18:00:39 mriedem i can't even really review it
18:00:53 mriedem asettle is also working on that patch
18:00:59 sdague honestly, I mostly want to light docs like that on fire
18:01:17 sdague because you have to know what you are looking for before you can find what you are looking for
18:01:44 mriedem wait which thing do you want to light on fire?
18:02:29 sdague spliting up the process doc into 6 process docs
18:02:52 sdague http://docs-draft.openstack.org/85/478485/16/check/gate-nova-docs-ubuntu-xenial/481415d//doc/build/html/contributor/process.html
18:03:14 mriedem compared to https://docs.openstack.org/nova/ocata/process.html ?
18:03:24 sdague yes
18:06:23 mriedem idk if this is meant to be consistent with other projects, but it's not consistent with cinder's home page now
18:06:31 mriedem https://docs.openstack.org/cinder/latest/
18:06:41 mriedem or glance https://docs.openstack.org/glance/latest/
18:13:16 sdague it would be nice to figure out how we get in page toc into the sidebar instead of top of page
18:13:34 mriedem yes that's also been annoying me
18:14:28 sdague dhellman might know
18:14:34 sdague looks like glance does it
18:16:56 mriedem sdague: any idea how we figure out which log formatter/handler we're using in a devstack run?
18:17:05 mriedem like, are those configurable somewhere?
18:18:10 sdague mriedem: ??
18:18:24 mriedem well there is a json formatter, and a context formatter
18:18:33 sdague yes, we're using the context formatter
18:18:38 mriedem and a color handler, and journal and syslog handler
18:18:43 mriedem but how would someone know?
18:18:47 mriedem config?
18:19:27 sdague it's config, but I think those are the defaults
18:20:55 sdague oh, you know what it is, it's actually python logging config
18:20:59 sdague outside of our config
18:21:10 sdague so use_syslog / use_journal will tweak those things
18:21:14 sdague https://docs.openstack.org/oslo.log/latest/configuration/index.html
18:21:27 sdague but on the formatter side it's all done as python logger setup
18:23:14 mriedem ok, because what i think we're seeing with a bunch of these random stacktraces related to ComputeHostNotFound_Remote is that we're logging with an exception context,
18:23:18 mriedem and oslo.log is logging the traceback
18:23:30 mriedem automagically
18:24:20 mriedem https://www.youtube.com/watch?v=iExgnVXSAuE
18:24:24 mriedem thanks oslo.log
18:26:11 mriedem gonna push a patch to test that theory
18:27:54 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: See if we can squash ComputeHostNotFound_Remote on startup https://review.openstack.org/489683
18:27:55 mriedem sdague: dansmith: ^ is the canary
18:29:23 mriedem i think it's something to do with this: https://github.com/openstack/oslo.log/blob/3.30.0/oslo_log/formatters.py#L144
18:36:45 openstackgerrit Merged openstack/nova master: Test resize with placement api https://review.openstack.org/487958
18:36:58 mriedem yay ^
18:37:13 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_certs to instance_extra https://review.openstack.org/457711
18:54:24 jaypipes gibi: where do I change the notification sample test JSON files?
18:54:58 jaypipes gibi: nm, found it.
19:02:15 openstackgerrit Jay Pipes proposed openstack/nova master: remove provider allocs in confirm/revert resize https://review.openstack.org/488510
19:02:15 openstackgerrit Jay Pipes proposed openstack/nova master: placement: remove existing allocs when set allocs https://review.openstack.org/489273
19:02:16 openstackgerrit Jay Pipes proposed openstack/nova master: Additional assertions to resize tests https://review.openstack.org/489714
19:05:41 melwitt this is a bug fix (data corruption possible) I think we'll need for rc1, if anyone can review please https://review.openstack.org/#/c/488545/
19:07:33 openstackgerrit Ildiko Vancsa proposed openstack/nova master: Implement new attach Cinder flow https://review.openstack.org/330285
19:22:38 openstackgerrit Eric Fried proposed openstack/nova master: nova.utils.get_endpoint_data() https://review.openstack.org/488137
19:23:09 mriedem melwitt: question in there
19:23:16 mriedem the persistent vs live stuff always confuses me
19:25:25 openstackgerrit Matt Riedemann proposed openstack/nova master: Cleanup unnecessary logic in os-volume_attachments controller code https://review.openstack.org/485823
19:25:36 cfriesen mriedem: isn't "persisten" whether there is a persistent domain (unrelated to the device)?
19:26:14 melwitt mriedem: I'm not sure I understand the question. in my comment I tried to say "if DeviceNotFound was raised when we passed persistent=True, that means it wasn't found, so we should continue on to try a live detach". though I do notice I should have used "if persistent and live"
19:26:33 melwitt because if live isn't True then there's no need to continue
19:26:42 mriedem well there you go
19:27:08 melwitt is that what you were pointing out? sorry :)
19:27:15 mriedem no
19:27:39 mriedem i just wanted to ask a question to sound like i know 2 sh*ts about this code before +2ing it

Earlier   Later