| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 14:38:16 | bauzas | johnthetubaguy: I need to look at code, because I wonder that | |
| 14:38:42 | bauzas | johnthetubaguy: say I'm booting inst1, then deleting computeA, do I get a nova list empty? | |
| 14:38:50 | bauzas | I suppose so | |
| 14:38:58 | bauzas | so I'm wondering where the thing is done | |
| 14:44:47 | openstackgerrit | Chris Dent proposed openstack/nova master: Add more debug logging on RP inventory update failures https://review.openstack.org/414230 | |
| 14:44:47 | openstackgerrit | Chris Dent proposed openstack/nova master: Add placement request id to log when GET or POST rps https://review.openstack.org/427740 | |
| 14:44:48 | openstackgerrit | Chris Dent proposed openstack/nova master: Add placement request id to log when GET aggregates https://review.openstack.org/426874 | |
| 14:44:48 | openstackgerrit | Chris Dent proposed openstack/nova master: Add more debug logging on RP inventory delete failures https://review.openstack.org/426290 | |
| 14:45:41 | cdent | johnthetubaguy: the bottom of that stack ^ was +W (+2 from you) but failing in the gate a few times, so I've rebased but now it is all bereft of its approval | |
| 14:46:27 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Destroy the ReqSpec object when deleting the instance https://review.openstack.org/391060 | |
| 14:46:27 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Cover again the ReqSpec record creation testing https://review.openstack.org/391059 | |
| 14:54:22 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add check for invalid allocation amounts https://review.openstack.org/416752 | |
| 14:54:24 | figleaf | cdent: ^^ Addressed your concerns | |
| 14:57:38 | johnthetubaguy | cdent: ah, doh, will take a peak | |
| 15:06:30 | johnthetubaguy | cdent: I do wonder if we have too many new lines in some of the error messages in the logs, BTW | |
| 15:07:22 | johnthetubaguy | cdent: its a micro nit, it might be more to do with the error message omitted by the placement API and how that looks in the logs | |
| 15:07:31 | mriedem | that's the nature of the requests.response isn't it? | |
| 15:09:25 | johnthetubaguy | it could be | |
| 15:12:22 | cdent | johnthetubaguy, mriedem: that could be the way that webob is formatting error responses. the json_error_formatter stuff subclasses something which puts newlines in here and there for no apparently reason | |
| 15:12:57 | johnthetubaguy | cdent: I was looking at that with pumaranikar the other day, I think it might be a bit new line happy | |
| 15:13:06 | cdent | new line happy | |
| 15:13:06 | cdent | yes | |
| 15:13:11 | openstackgerrit | Sarafraj Singh proposed openstack/nova-specs master: Send service token along with user tokens https://review.openstack.org/426933 | |
| 15:13:32 | cdent | i'm not sure I'm keen on the decorator idea, but the current middleware idea won't work | |
| 15:13:44 | cdent | (but I've mostly been ignoring that) | |
| 15:16:10 | cdent | johnthetubaguy: in any case, the logging on the rt side shouldn't care. we should just fix it on the api side, yeah? | |
| 15:16:59 | johnthetubaguy | cdent: yeah, certainly if the extra new lines are from the API, which I think they are | |
| 15:17:22 | johnthetubaguy | cdent: it was more your patches reminded my about the niggle I spotted debugging that bug the other day | |
| 15:17:56 | cdent | let's make pumaranikar fix it :) | |
| 15:18:39 | johnthetubaguy | good idea | |
| 15:18:47 | johnthetubaguy | pumaranikar: we found you another placement bug | |
| 15:19:23 | EmilienM | mriedem: good morning, regarding the issue with Nova in WSGI in TripleO - does it make sense we stop doing that in Ocata (and revisit later)? | |
| 15:19:26 | mriedem | so on the tripleo failures from yesterday, with a run on my debug patch, i see the instance that heat is trying to create fails | |
| 15:19:26 | mriedem | http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-conductor.txt.gz#_2017-02-03_05_17_35_826 | |
| 15:19:28 | mriedem | EmilienM: good timing | |
| 15:19:32 | EmilienM | wow | |
| 15:19:43 | mriedem | scheduler couldn't find a host | |
| 15:19:56 | EmilienM | do you want me to stop deploing nova api in wsgi with apache and see if it helps? | |
| 15:20:15 | mriedem | given we haven't really flushed out testing with that, i think it's prudent at this point in ocata | |
| 15:20:26 | EmilienM | ok let me do it | |
| 15:21:03 | mriedem | the 404 isn't because the instance was created and we just can't find it though, it's not found because it fails to build and the instance isn't created (it looks like), but the dead instance should go into cell0 and we should pull it from cell0 | |
| 15:21:31 | mriedem | hrm, dansmith melwitt do we even look in cell0 for compute api get()? | |
| 15:21:39 | cdent | johnthetubaguy, pumaranikar It can probably be built right in with the json_formatter stuff because it's all of a piece | |
| 15:22:12 | melwitt | mriedem: we should, I think. do we not? checking | |
| 15:22:21 | dansmith | melwitt: yes | |
| 15:22:23 | dansmith | er, mriedem yes | |
| 15:22:31 | dansmith | mriedem: the mapping tells us what cell to look in | |
| 15:22:38 | melwitt | oh yeah | |
| 15:22:39 | dansmith | which would be cell0 if it's there | |
| 15:24:15 | mriedem | what's odd is in this debug run i expect to see the 'found x cells' thing from https://review.openstack.org/#/c/428360/2/nova/compute/api.py@2526 | |
| 15:24:22 | mriedem | but it's not in http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-api.txt.gz | |
| 15:24:30 | kashyap | johnthetubaguy: "But asking for a friendlier looking supernova seems a bit... unscientific" :-) | |
| 15:24:52 | johnthetubaguy | kashyap: :) | |
| 15:25:19 | kashyap | Meanwhile, if there are any Virtualization maintainers of Ubuntu, you may want to triage this - https://bugs.launchpad.net/nova/+bug/1643911 | |
| 15:25:19 | openstack | Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed] | |
| 15:25:24 | kashyap | lease see my last comment there. | |
| 15:25:26 | melwitt | mriedem: I thought we had found there weren't any 'GET .../servers' in those logs | |
| 15:25:31 | kashyap | (s/lease/Please/) | |
| 15:26:01 | mriedem | melwitt: my debug patch also did the cells thing in _get_instance() | |
| 15:26:23 | melwitt | oh, you added logging for _get_instance | |
| 15:26:47 | mriedem | melwitt: http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-api.txt.gz#_2017-02-03_05_17_30_873 is the GET | |
| 15:27:31 | mriedem | but it's not hitting the logging i added here https://review.openstack.org/#/c/428360/2/nova/compute/api.py | |
| 15:28:13 | mriedem | so i'm thinking we're trying to pull it from a cell but it's not there, which would be nuts, but i'll add some more logging | |
| 15:28:58 | mriedem | oh well i guess that would make sense if we pull the instance here https://review.openstack.org/#/c/428360/2/nova/compute/api.py@2269 | |
| 15:29:04 | mriedem | because the service version is 0 | |
| 15:29:41 | melwitt | yeah, I was about to say, we had found bc they're running through apache they're getting service version 0, or something | |
| 15:31:51 | mriedem | yup, | |
| 15:31:52 | mriedem | AND | |
| 15:31:59 | mriedem | http://logs.openstack.org/15/428415/1/experimental/gate-tripleo-ci-centos-7-nonha-multinode/4a76477/ | |
| 15:32:02 | mriedem | is dan's workaround patch | |
| 15:33:24 | EmilienM | mriedem: https://review.openstack.org/428778 - once it pass our CI, I'll ask you you Depends-On this one, so you can continue the tests without WSGI in Apache | |
| 15:36:23 | mdbooth | mriedem: Thanks for looking at that NFS patch, btw. You found 2 bugs in it :) | |
| 15:36:49 | mriedem | mdbooth: but you told me it was rock solid! | |
| 15:37:08 | mdbooth | mriedem: I was thinking a good limestone. | |
| 15:37:16 | mdbooth | You know, pretty solid. | |
| 15:37:20 | mdbooth | ;) | |
| 15:37:27 | mriedem | heh | |
| 15:38:28 | mdbooth | Both would result in mounts not being unmounted, which is the opposite of the original problem of mounts being unmounted when they shouldn't be. | |
| 16:07:15 | EmilienM | mriedem: FYI https://review.openstack.org/#/q/topic:bug/1661360+owner:%22Emilien+Macchi+%253Cemilien%2540redhat.com%253E%22 | |
| 16:09:27 | mdbooth | mriedem: On second thoughts, only 1 bug :) The other 1 just looks sufficiently weird that I misread it, too. | |
| 16:11:49 | mriedem | EmilienM: good lord that's a lot of change | |
| 16:11:54 | mriedem | but i don't know puppet or tripleo | |
| 16:13:57 | pumaranikar | johnthetubaguy, cdent sry i missed initial conversation | |
| 16:14:21 | johnthetubaguy | pumaranikar: no worries, was going to ping you | |
| 16:14:40 | johnthetubaguy | pumaranikar: thinking about the errors produced by the placement API, they have lots of big new lines in them | |
| 16:14:51 | johnthetubaguy | pumaranikar: it would be great if you could raise a bug for that and fix it | |
| 16:15:01 | johnthetubaguy | pumaranikar: seems like the json formatter needs a tweak to fix that up | |
| 16:17:32 | EmilienM | mriedem: it was just an fyi | |
| 16:17:36 | pumaranikar | johnthetubaguy, sure i will check it. | |
| 16:17:49 | mriedem | EmilienM: i made one very constructive comment on one patch | |
| 16:17:50 | johnthetubaguy | pumaranikar: thanks | |
| 16:19:30 | pumaranikar | johnthetubaguy, regarding json_formatter, should we try again with middleware or go with decorated function. what you suggest ? | |
| 16:21:32 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add check for invalid allocation amounts https://review.openstack.org/416752 | |
| 16:21:33 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add check for invalid inventory amounts https://review.openstack.org/428800 | |
| 16:21:41 | figleaf | cdent: split the inventory and allocation changes ^^ | |
| 16:22:11 | cdent | thanks figleaf | |
| 16:24:10 | cdent | pumaranikar, johnthetubaguy I'd prefer middleware if we can make it work, but I haven't thought about it enough to know it can. We would need to make sure that it only engaged (processed the body) when it absolutely had to. The main issue is that anything happening "outside" of webob.dec.wsgify will not see an exception, only a response | |
| 16:33:05 | osallou | hi, i have activated nova notifications and connected multiple subscribes to nova exchange, however a notifications.info queue is always created and is filled indefinitly is no subscribed to drain it, is it a configuration error? I need multiple subscribers on notifications, so I cannot simply listen to the queue... thanks | |
| 16:39:19 | pumaranikar | cdent, johnthetubaguy yes. I think for every resource based action we have added webob.dec.wsgify | |
| 16:41:30 | EmilienM | mriedem: http://logs.openstack.org/86/428786/1/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/522b390/logs/nova/nova-api.txt.gz#_2017-02-03_16_15_23_470 | |