Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-03
14:54:22 openstackgerrit Ed Leafe proposed openstack/nova master: Add check for invalid allocation amounts https://review.openstack.org/416752
14:54:24 figleaf cdent: ^^ Addressed your concerns
14:57:38 johnthetubaguy cdent: ah, doh, will take a peak
15:06:30 johnthetubaguy cdent: I do wonder if we have too many new lines in some of the error messages in the logs, BTW
15:07:22 johnthetubaguy cdent: its a micro nit, it might be more to do with the error message omitted by the placement API and how that looks in the logs
15:07:31 mriedem that's the nature of the requests.response isn't it?
15:09:25 johnthetubaguy it could be
15:12:22 cdent johnthetubaguy, mriedem: that could be the way that webob is formatting error responses. the json_error_formatter stuff subclasses something which puts newlines in here and there for no apparently reason
15:12:57 johnthetubaguy cdent: I was looking at that with pumaranikar the other day, I think it might be a bit new line happy
15:13:06 cdent yes
15:13:06 cdent new line happy
15:13:11 openstackgerrit Sarafraj Singh proposed openstack/nova-specs master: Send service token along with user tokens https://review.openstack.org/426933
15:13:32 cdent i'm not sure I'm keen on the decorator idea, but the current middleware idea won't work
15:13:44 cdent (but I've mostly been ignoring that)
15:16:10 cdent johnthetubaguy: in any case, the logging on the rt side shouldn't care. we should just fix it on the api side, yeah?
15:16:59 johnthetubaguy cdent: yeah, certainly if the extra new lines are from the API, which I think they are
15:17:22 johnthetubaguy cdent: it was more your patches reminded my about the niggle I spotted debugging that bug the other day
15:17:56 cdent let's make pumaranikar fix it :)
15:18:39 johnthetubaguy good idea
15:18:47 johnthetubaguy pumaranikar: we found you another placement bug
15:19:23 EmilienM mriedem: good morning, regarding the issue with Nova in WSGI in TripleO - does it make sense we stop doing that in Ocata (and revisit later)?
15:19:26 mriedem http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-conductor.txt.gz#_2017-02-03_05_17_35_826
15:19:26 mriedem so on the tripleo failures from yesterday, with a run on my debug patch, i see the instance that heat is trying to create fails
15:19:28 mriedem EmilienM: good timing
15:19:32 EmilienM wow
15:19:43 mriedem scheduler couldn't find a host
15:19:56 EmilienM do you want me to stop deploing nova api in wsgi with apache and see if it helps?
15:20:15 mriedem given we haven't really flushed out testing with that, i think it's prudent at this point in ocata
15:20:26 EmilienM ok let me do it
15:21:03 mriedem the 404 isn't because the instance was created and we just can't find it though, it's not found because it fails to build and the instance isn't created (it looks like), but the dead instance should go into cell0 and we should pull it from cell0
15:21:31 mriedem hrm, dansmith melwitt do we even look in cell0 for compute api get()?
15:21:39 cdent johnthetubaguy, pumaranikar It can probably be built right in with the json_formatter stuff because it's all of a piece
15:22:12 melwitt mriedem: we should, I think. do we not? checking
15:22:21 dansmith melwitt: yes
15:22:23 dansmith er, mriedem yes
15:22:31 dansmith mriedem: the mapping tells us what cell to look in
15:22:38 melwitt oh yeah
15:22:39 dansmith which would be cell0 if it's there
15:24:15 mriedem what's odd is in this debug run i expect to see the 'found x cells' thing from https://review.openstack.org/#/c/428360/2/nova/compute/api.py@2526
15:24:22 mriedem but it's not in http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-api.txt.gz
15:24:30 kashyap johnthetubaguy: "But asking for a friendlier looking supernova seems a bit... unscientific" :-)
15:24:52 johnthetubaguy kashyap: :)
15:25:19 openstack Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed]
15:25:19 kashyap Meanwhile, if there are any Virtualization maintainers of Ubuntu, you may want to triage this - https://bugs.launchpad.net/nova/+bug/1643911
15:25:24 kashyap lease see my last comment there.
15:25:26 melwitt mriedem: I thought we had found there weren't any 'GET .../servers' in those logs
15:25:31 kashyap (s/lease/Please/)
15:26:01 mriedem melwitt: my debug patch also did the cells thing in _get_instance()
15:26:23 melwitt oh, you added logging for _get_instance
15:26:47 mriedem melwitt: http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-api.txt.gz#_2017-02-03_05_17_30_873 is the GET
15:27:31 mriedem but it's not hitting the logging i added here https://review.openstack.org/#/c/428360/2/nova/compute/api.py
15:28:13 mriedem so i'm thinking we're trying to pull it from a cell but it's not there, which would be nuts, but i'll add some more logging
15:28:58 mriedem oh well i guess that would make sense if we pull the instance here https://review.openstack.org/#/c/428360/2/nova/compute/api.py@2269
15:29:04 mriedem because the service version is 0
15:29:41 melwitt yeah, I was about to say, we had found bc they're running through apache they're getting service version 0, or something
15:31:51 mriedem yup,
15:31:52 mriedem AND
15:31:59 mriedem http://logs.openstack.org/15/428415/1/experimental/gate-tripleo-ci-centos-7-nonha-multinode/4a76477/
15:32:02 mriedem is dan's workaround patch
15:33:24 EmilienM mriedem: https://review.openstack.org/428778 - once it pass our CI, I'll ask you you Depends-On this one, so you can continue the tests without WSGI in Apache
15:36:23 mdbooth mriedem: Thanks for looking at that NFS patch, btw. You found 2 bugs in it :)
15:36:49 mriedem mdbooth: but you told me it was rock solid!
15:37:08 mdbooth mriedem: I was thinking a good limestone.
15:37:16 mdbooth You know, pretty solid.
15:37:20 mdbooth ;)
15:37:27 mriedem heh
15:38:28 mdbooth Both would result in mounts not being unmounted, which is the opposite of the original problem of mounts being unmounted when they shouldn't be.
16:07:15 EmilienM mriedem: FYI https://review.openstack.org/#/q/topic:bug/1661360+owner:%22Emilien+Macchi+%253Cemilien%2540redhat.com%253E%22
16:09:27 mdbooth mriedem: On second thoughts, only 1 bug :) The other 1 just looks sufficiently weird that I misread it, too.
16:11:49 mriedem EmilienM: good lord that's a lot of change
16:11:54 mriedem but i don't know puppet or tripleo
16:13:57 pumaranikar johnthetubaguy, cdent sry i missed initial conversation
16:14:21 johnthetubaguy pumaranikar: no worries, was going to ping you
16:14:40 johnthetubaguy pumaranikar: thinking about the errors produced by the placement API, they have lots of big new lines in them
16:14:51 johnthetubaguy pumaranikar: it would be great if you could raise a bug for that and fix it
16:15:01 johnthetubaguy pumaranikar: seems like the json formatter needs a tweak to fix that up
16:17:32 EmilienM mriedem: it was just an fyi
16:17:36 pumaranikar johnthetubaguy, sure i will check it.
16:17:49 mriedem EmilienM: i made one very constructive comment on one patch
16:17:50 johnthetubaguy pumaranikar: thanks
16:19:30 pumaranikar johnthetubaguy, regarding json_formatter, should we try again with middleware or go with decorated function. what you suggest ?
16:21:32 openstackgerrit Ed Leafe proposed openstack/nova master: Add check for invalid allocation amounts https://review.openstack.org/416752
16:21:33 openstackgerrit Ed Leafe proposed openstack/nova master: Add check for invalid inventory amounts https://review.openstack.org/428800
16:21:41 figleaf cdent: split the inventory and allocation changes ^^
16:22:11 cdent thanks figleaf
16:24:10 cdent pumaranikar, johnthetubaguy I'd prefer middleware if we can make it work, but I haven't thought about it enough to know it can. We would need to make sure that it only engaged (processed the body) when it absolutely had to. The main issue is that anything happening "outside" of webob.dec.wsgify will not see an exception, only a response
16:33:05 osallou hi, i have activated nova notifications and connected multiple subscribes to nova exchange, however a notifications.info queue is always created and is filled indefinitly is no subscribed to drain it, is it a configuration error? I need multiple subscribers on notifications, so I cannot simply listen to the queue... thanks
16:39:19 pumaranikar cdent, johnthetubaguy yes. I think for every resource based action we have added webob.dec.wsgify
16:41:30 EmilienM mriedem: http://logs.openstack.org/86/428786/1/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/522b390/logs/nova/nova-api.txt.gz#_2017-02-03_16_15_23_470
16:41:39 EmilienM mriedem: should it be ::1:8774?
16:45:40 mriedem ?
16:45:40 mriedem and remove the []
16:45:42 mriedem idk
16:45:44 mriedem is that ipv6?
16:51:06 cdent pumaranikar: sorry was on the phone. yes that's right, thus the difficulty
16:54:59 openstackgerrit Matthew Booth proposed openstack/nova master: libvirt: Fix races with nfs volume mount/umount https://review.openstack.org/383859
16:56:53 openstackgerrit Maciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs https://review.openstack.org/428811
16:58:33 pumaranikar cdent, i will try with middleware and update you
16:58:53 openstackgerrit Maciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs https://review.openstack.org/428811
17:03:31 johnthetubaguy pumaranikar: we did that yesterday right, and it failed with middlewear?

Earlier   Later