| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-03 | |||
| 15:19:23 | EmilienM | mriedem: good morning, regarding the issue with Nova in WSGI in TripleO - does it make sense we stop doing that in Ocata (and revisit later)? | |
| 15:19:26 | mriedem | http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-conductor.txt.gz#_2017-02-03_05_17_35_826 | |
| 15:19:26 | mriedem | so on the tripleo failures from yesterday, with a run on my debug patch, i see the instance that heat is trying to create fails | |
| 15:19:28 | mriedem | EmilienM: good timing | |
| 15:19:32 | EmilienM | wow | |
| 15:19:43 | mriedem | scheduler couldn't find a host | |
| 15:19:56 | EmilienM | do you want me to stop deploing nova api in wsgi with apache and see if it helps? | |
| 15:20:15 | mriedem | given we haven't really flushed out testing with that, i think it's prudent at this point in ocata | |
| 15:20:26 | EmilienM | ok let me do it | |
| 15:21:03 | mriedem | the 404 isn't because the instance was created and we just can't find it though, it's not found because it fails to build and the instance isn't created (it looks like), but the dead instance should go into cell0 and we should pull it from cell0 | |
| 15:21:31 | mriedem | hrm, dansmith melwitt do we even look in cell0 for compute api get()? | |
| 15:21:39 | cdent | johnthetubaguy, pumaranikar It can probably be built right in with the json_formatter stuff because it's all of a piece | |
| 15:22:12 | melwitt | mriedem: we should, I think. do we not? checking | |
| 15:22:21 | dansmith | melwitt: yes | |
| 15:22:23 | dansmith | er, mriedem yes | |
| 15:22:31 | dansmith | mriedem: the mapping tells us what cell to look in | |
| 15:22:38 | melwitt | oh yeah | |
| 15:22:39 | dansmith | which would be cell0 if it's there | |
| 15:24:15 | mriedem | what's odd is in this debug run i expect to see the 'found x cells' thing from https://review.openstack.org/#/c/428360/2/nova/compute/api.py@2526 | |
| 15:24:22 | mriedem | but it's not in http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-api.txt.gz | |
| 15:24:30 | kashyap | johnthetubaguy: "But asking for a friendlier looking supernova seems a bit... unscientific" :-) | |
| 15:24:52 | johnthetubaguy | kashyap: :) | |
| 15:25:19 | openstack | Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed] | |
| 15:25:19 | kashyap | Meanwhile, if there are any Virtualization maintainers of Ubuntu, you may want to triage this - https://bugs.launchpad.net/nova/+bug/1643911 | |
| 15:25:24 | kashyap | lease see my last comment there. | |
| 15:25:26 | melwitt | mriedem: I thought we had found there weren't any 'GET .../servers' in those logs | |
| 15:25:31 | kashyap | (s/lease/Please/) | |
| 15:26:01 | mriedem | melwitt: my debug patch also did the cells thing in _get_instance() | |
| 15:26:23 | melwitt | oh, you added logging for _get_instance | |
| 15:26:47 | mriedem | melwitt: http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-api.txt.gz#_2017-02-03_05_17_30_873 is the GET | |
| 15:27:31 | mriedem | but it's not hitting the logging i added here https://review.openstack.org/#/c/428360/2/nova/compute/api.py | |
| 15:28:13 | mriedem | so i'm thinking we're trying to pull it from a cell but it's not there, which would be nuts, but i'll add some more logging | |
| 15:28:58 | mriedem | oh well i guess that would make sense if we pull the instance here https://review.openstack.org/#/c/428360/2/nova/compute/api.py@2269 | |
| 15:29:04 | mriedem | because the service version is 0 | |
| 15:29:41 | melwitt | yeah, I was about to say, we had found bc they're running through apache they're getting service version 0, or something | |
| 15:31:51 | mriedem | yup, | |
| 15:31:52 | mriedem | AND | |
| 15:31:59 | mriedem | http://logs.openstack.org/15/428415/1/experimental/gate-tripleo-ci-centos-7-nonha-multinode/4a76477/ | |
| 15:32:02 | mriedem | is dan's workaround patch | |
| 15:33:24 | EmilienM | mriedem: https://review.openstack.org/428778 - once it pass our CI, I'll ask you you Depends-On this one, so you can continue the tests without WSGI in Apache | |
| 15:36:23 | mdbooth | mriedem: Thanks for looking at that NFS patch, btw. You found 2 bugs in it :) | |
| 15:36:49 | mriedem | mdbooth: but you told me it was rock solid! | |
| 15:37:08 | mdbooth | mriedem: I was thinking a good limestone. | |
| 15:37:16 | mdbooth | You know, pretty solid. | |
| 15:37:20 | mdbooth | ;) | |
| 15:37:27 | mriedem | heh | |
| 15:38:28 | mdbooth | Both would result in mounts not being unmounted, which is the opposite of the original problem of mounts being unmounted when they shouldn't be. | |
| 16:07:15 | EmilienM | mriedem: FYI https://review.openstack.org/#/q/topic:bug/1661360+owner:%22Emilien+Macchi+%253Cemilien%2540redhat.com%253E%22 | |
| 16:09:27 | mdbooth | mriedem: On second thoughts, only 1 bug :) The other 1 just looks sufficiently weird that I misread it, too. | |
| 16:11:49 | mriedem | EmilienM: good lord that's a lot of change | |
| 16:11:54 | mriedem | but i don't know puppet or tripleo | |
| 16:13:57 | pumaranikar | johnthetubaguy, cdent sry i missed initial conversation | |
| 16:14:21 | johnthetubaguy | pumaranikar: no worries, was going to ping you | |
| 16:14:40 | johnthetubaguy | pumaranikar: thinking about the errors produced by the placement API, they have lots of big new lines in them | |
| 16:14:51 | johnthetubaguy | pumaranikar: it would be great if you could raise a bug for that and fix it | |
| 16:15:01 | johnthetubaguy | pumaranikar: seems like the json formatter needs a tweak to fix that up | |
| 16:17:32 | EmilienM | mriedem: it was just an fyi | |
| 16:17:36 | pumaranikar | johnthetubaguy, sure i will check it. | |
| 16:17:49 | mriedem | EmilienM: i made one very constructive comment on one patch | |
| 16:17:50 | johnthetubaguy | pumaranikar: thanks | |
| 16:19:30 | pumaranikar | johnthetubaguy, regarding json_formatter, should we try again with middleware or go with decorated function. what you suggest ? | |
| 16:21:32 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add check for invalid allocation amounts https://review.openstack.org/416752 | |
| 16:21:33 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add check for invalid inventory amounts https://review.openstack.org/428800 | |
| 16:21:41 | figleaf | cdent: split the inventory and allocation changes ^^ | |
| 16:22:11 | cdent | thanks figleaf | |
| 16:24:10 | cdent | pumaranikar, johnthetubaguy I'd prefer middleware if we can make it work, but I haven't thought about it enough to know it can. We would need to make sure that it only engaged (processed the body) when it absolutely had to. The main issue is that anything happening "outside" of webob.dec.wsgify will not see an exception, only a response | |
| 16:33:05 | osallou | hi, i have activated nova notifications and connected multiple subscribes to nova exchange, however a notifications.info queue is always created and is filled indefinitly is no subscribed to drain it, is it a configuration error? I need multiple subscribers on notifications, so I cannot simply listen to the queue... thanks | |
| 16:39:19 | pumaranikar | cdent, johnthetubaguy yes. I think for every resource based action we have added webob.dec.wsgify | |
| 16:41:30 | EmilienM | mriedem: http://logs.openstack.org/86/428786/1/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/522b390/logs/nova/nova-api.txt.gz#_2017-02-03_16_15_23_470 | |
| 16:41:39 | EmilienM | mriedem: should it be ::1:8774? | |
| 16:45:40 | mriedem | ? | |
| 16:45:40 | mriedem | and remove the [] | |
| 16:45:42 | mriedem | idk | |
| 16:45:44 | mriedem | is that ipv6? | |
| 16:51:06 | cdent | pumaranikar: sorry was on the phone. yes that's right, thus the difficulty | |
| 16:54:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: libvirt: Fix races with nfs volume mount/umount https://review.openstack.org/383859 | |
| 16:56:53 | openstackgerrit | Maciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs https://review.openstack.org/428811 | |
| 16:58:33 | pumaranikar | cdent, i will try with middleware and update you | |
| 16:58:53 | openstackgerrit | Maciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs https://review.openstack.org/428811 | |
| 17:03:31 | johnthetubaguy | pumaranikar: we did that yesterday right, and it failed with middlewear? | |
| 17:04:05 | johnthetubaguy | cdent: the exception just didn't seem to get as fair as that fault wrapper middleware | |
| 17:04:27 | johnthetubaguy | it was already converted into a response with the correct error code, at least is seemed that way | |
| 17:04:31 | pumaranikar | johnthetubaguy, yes. because we were getting valid response | |
| 17:04:34 | cdent | yeah, if you use middleware, it has to be body processing instead of exception inspecting | |
| 17:05:00 | johnthetubaguy | but the body doesn't have the data present that was in the exception I guess? | |
| 17:05:25 | cdent | if the response ststus >= 400 and <500, then if it is json, then reformat it to the structure we want | |
| 17:05:37 | johnthetubaguy | cdent: my suggestion to pumaranikar was to replace the wsgify with our own decorator, that ensure the exception gets the formatter added, its not ideal though | |
| 17:06:19 | johnthetubaguy | it seems easier to remember that adding the formatter to the error message, but that might just be me | |
| 17:06:22 | cdent | if there's a way to do that doesn't involve double decorating that might work | |
| 17:06:48 | johnthetubaguy | yeah, its just the one decorator, I pushed that up yesterday into pumaranikar's patch | |
| 17:07:29 | cdent | that's what I mean by double decorator | |
| 17:07:32 | mriedem | artom: on https://blueprints.launchpad.net/nova/+spec/fix-tag-attribute-disappearing i think i'm going to just mark that as partially completed and then we'll finish the rest in pike | |
| 17:07:43 | cdent | I think it would be better to subclass webob.dec.wsgify (which is a class) | |
| 17:07:50 | johnthetubaguy | cdent: its not the nicest thing, but it seems to work: https://review.openstack.org/#/c/395194/5/nova/api/openstack/placement/handlers/allocation.py@274 | |
| 17:07:59 | artom | mriedem, yep :) | |
| 17:08:13 | johnthetubaguy | cdent: yeah, thats nicer | |
| 17:08:21 | johnthetubaguy | pumaranikar: does that make sense? | |
| 17:08:33 | cdent | except, unfortunately the part that one would want to override is smack in a tall stack of code :( | |
| 17:08:42 | artom | mriedem, it's debatable whether the changes to tests are parts of the spec proper, in any case | |
| 17:09:10 | johnthetubaguy | cdent: ah, doh, anyways my hack appeared to work, we could always make that nicer later | |