Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-03
15:24:30 kashyap johnthetubaguy: "But asking for a friendlier looking supernova seems a bit... unscientific" :-)
15:24:52 johnthetubaguy kashyap: :)
15:25:19 kashyap Meanwhile, if there are any Virtualization maintainers of Ubuntu, you may want to triage this - https://bugs.launchpad.net/nova/+bug/1643911
15:25:19 openstack Launchpad bug 1643911 in OpenStack Compute (nova) "libvirt randomly crashes on xenial nodes with "*** Error in `/usr/sbin/libvirtd': malloc(): memory corruption:"" [Medium,Confirmed]
15:25:24 kashyap lease see my last comment there.
15:25:26 melwitt mriedem: I thought we had found there weren't any 'GET .../servers' in those logs
15:25:31 kashyap (s/lease/Please/)
15:26:01 mriedem melwitt: my debug patch also did the cells thing in _get_instance()
15:26:23 melwitt oh, you added logging for _get_instance
15:26:47 mriedem melwitt: http://logs.openstack.org/60/428360/2/experimental/gate-tripleo-ci-centos-7-nonha-multinode/7a6e5bc/logs/subnode-2/var/log/nova/nova-api.txt.gz#_2017-02-03_05_17_30_873 is the GET
15:27:31 mriedem but it's not hitting the logging i added here https://review.openstack.org/#/c/428360/2/nova/compute/api.py
15:28:13 mriedem so i'm thinking we're trying to pull it from a cell but it's not there, which would be nuts, but i'll add some more logging
15:28:58 mriedem oh well i guess that would make sense if we pull the instance here https://review.openstack.org/#/c/428360/2/nova/compute/api.py@2269
15:29:04 mriedem because the service version is 0
15:29:41 melwitt yeah, I was about to say, we had found bc they're running through apache they're getting service version 0, or something
15:31:51 mriedem yup,
15:31:52 mriedem AND
15:31:59 mriedem http://logs.openstack.org/15/428415/1/experimental/gate-tripleo-ci-centos-7-nonha-multinode/4a76477/
15:32:02 mriedem is dan's workaround patch
15:33:24 EmilienM mriedem: https://review.openstack.org/428778 - once it pass our CI, I'll ask you you Depends-On this one, so you can continue the tests without WSGI in Apache
15:36:23 mdbooth mriedem: Thanks for looking at that NFS patch, btw. You found 2 bugs in it :)
15:36:49 mriedem mdbooth: but you told me it was rock solid!
15:37:08 mdbooth mriedem: I was thinking a good limestone.
15:37:16 mdbooth You know, pretty solid.
15:37:20 mdbooth ;)
15:37:27 mriedem heh
15:38:28 mdbooth Both would result in mounts not being unmounted, which is the opposite of the original problem of mounts being unmounted when they shouldn't be.
16:07:15 EmilienM mriedem: FYI https://review.openstack.org/#/q/topic:bug/1661360+owner:%22Emilien+Macchi+%253Cemilien%2540redhat.com%253E%22
16:09:27 mdbooth mriedem: On second thoughts, only 1 bug :) The other 1 just looks sufficiently weird that I misread it, too.
16:11:49 mriedem EmilienM: good lord that's a lot of change
16:11:54 mriedem but i don't know puppet or tripleo
16:13:57 pumaranikar johnthetubaguy, cdent sry i missed initial conversation
16:14:21 johnthetubaguy pumaranikar: no worries, was going to ping you
16:14:40 johnthetubaguy pumaranikar: thinking about the errors produced by the placement API, they have lots of big new lines in them
16:14:51 johnthetubaguy pumaranikar: it would be great if you could raise a bug for that and fix it
16:15:01 johnthetubaguy pumaranikar: seems like the json formatter needs a tweak to fix that up
16:17:32 EmilienM mriedem: it was just an fyi
16:17:36 pumaranikar johnthetubaguy, sure i will check it.
16:17:49 mriedem EmilienM: i made one very constructive comment on one patch
16:17:50 johnthetubaguy pumaranikar: thanks
16:19:30 pumaranikar johnthetubaguy, regarding json_formatter, should we try again with middleware or go with decorated function. what you suggest ?
16:21:32 openstackgerrit Ed Leafe proposed openstack/nova master: Add check for invalid allocation amounts https://review.openstack.org/416752
16:21:33 openstackgerrit Ed Leafe proposed openstack/nova master: Add check for invalid inventory amounts https://review.openstack.org/428800
16:21:41 figleaf cdent: split the inventory and allocation changes ^^
16:22:11 cdent thanks figleaf
16:24:10 cdent pumaranikar, johnthetubaguy I'd prefer middleware if we can make it work, but I haven't thought about it enough to know it can. We would need to make sure that it only engaged (processed the body) when it absolutely had to. The main issue is that anything happening "outside" of webob.dec.wsgify will not see an exception, only a response
16:33:05 osallou hi, i have activated nova notifications and connected multiple subscribes to nova exchange, however a notifications.info queue is always created and is filled indefinitly is no subscribed to drain it, is it a configuration error? I need multiple subscribers on notifications, so I cannot simply listen to the queue... thanks
16:39:19 pumaranikar cdent, johnthetubaguy yes. I think for every resource based action we have added webob.dec.wsgify
16:41:30 EmilienM mriedem: http://logs.openstack.org/86/428786/1/check/gate-puppet-openstack-integration-4-scenario001-tempest-centos-7/522b390/logs/nova/nova-api.txt.gz#_2017-02-03_16_15_23_470
16:41:39 EmilienM mriedem: should it be ::1:8774?
16:45:40 mriedem and remove the []
16:45:40 mriedem ?
16:45:42 mriedem idk
16:45:44 mriedem is that ipv6?
16:51:06 cdent pumaranikar: sorry was on the phone. yes that's right, thus the difficulty
16:54:59 openstackgerrit Matthew Booth proposed openstack/nova master: libvirt: Fix races with nfs volume mount/umount https://review.openstack.org/383859
16:56:53 openstackgerrit Maciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs https://review.openstack.org/428811
16:58:33 pumaranikar cdent, i will try with middleware and update you
16:58:53 openstackgerrit Maciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs https://review.openstack.org/428811
17:03:31 johnthetubaguy pumaranikar: we did that yesterday right, and it failed with middlewear?
17:04:05 johnthetubaguy cdent: the exception just didn't seem to get as fair as that fault wrapper middleware
17:04:27 johnthetubaguy it was already converted into a response with the correct error code, at least is seemed that way
17:04:31 pumaranikar johnthetubaguy, yes. because we were getting valid response
17:04:34 cdent yeah, if you use middleware, it has to be body processing instead of exception inspecting
17:05:00 johnthetubaguy but the body doesn't have the data present that was in the exception I guess?
17:05:25 cdent if the response ststus >= 400 and <500, then if it is json, then reformat it to the structure we want
17:05:37 johnthetubaguy cdent: my suggestion to pumaranikar was to replace the wsgify with our own decorator, that ensure the exception gets the formatter added, its not ideal though
17:06:19 johnthetubaguy it seems easier to remember that adding the formatter to the error message, but that might just be me
17:06:22 cdent if there's a way to do that doesn't involve double decorating that might work
17:06:48 johnthetubaguy yeah, its just the one decorator, I pushed that up yesterday into pumaranikar's patch
17:07:29 cdent that's what I mean by double decorator
17:07:32 mriedem artom: on https://blueprints.launchpad.net/nova/+spec/fix-tag-attribute-disappearing i think i'm going to just mark that as partially completed and then we'll finish the rest in pike
17:07:43 cdent I think it would be better to subclass webob.dec.wsgify (which is a class)
17:07:50 johnthetubaguy cdent: its not the nicest thing, but it seems to work: https://review.openstack.org/#/c/395194/5/nova/api/openstack/placement/handlers/allocation.py@274
17:07:59 artom mriedem, yep :)
17:08:13 johnthetubaguy cdent: yeah, thats nicer
17:08:21 johnthetubaguy pumaranikar: does that make sense?
17:08:33 cdent except, unfortunately the part that one would want to override is smack in a tall stack of code :(
17:08:42 artom mriedem, it's debatable whether the changes to tests are parts of the spec proper, in any case
17:09:10 johnthetubaguy cdent: ah, doh, anyways my hack appeared to work, we could always make that nicer later
17:09:12 cdent hmm, maybe override call_func
17:09:32 johnthetubaguy cdent: did you see this way: https://review.openstack.org/#/c/395194/5/nova/api/openstack/placement/handlers/allocation.py@274 ?
17:09:37 cdent pumaranikar: have a look at the wsgify class in webob/dec.py, may be some ideas
17:10:13 cdent johnthetubaguy: yeah, saw that
17:10:52 cdent I think that will work, but it feels icky for a reason I can't put my finger on
17:11:06 johnthetubaguy cdent: totally with you on that
17:11:08 pumaranikar cdent, looking at wsgify class.
17:11:19 cdent it might, however, be less messy than what I'm thinking. won't really know until we try it
17:11:26 johnthetubaguy cdent: +1
17:11:34 cdent pumaranikar: I think overriding the call_fun method might work
17:15:55 johnthetubaguy cdent: I was curious on your gabbi thoughts too, this seemed the simplest way to ensure the formatter was present: https://review.openstack.org/#/c/395194/5/nova/tests/functional/api/openstack/placement/gabbits/allocations.yaml@158
17:16:45 cdent johnthetubaguy: yeah, pretty much that's it. look for a key and value that wouldn't be there if the formatter was not swinging
17:18:09 EmilienM mriedem: yes it's IPv6. Do you test nova in ipv6?
17:18:40 EmilienM osapi_compute_listen=[::1] doesn't seem to work
17:19:44 mriedem EmilienM: not sure off hand
17:19:50 mriedem sdague or sc68cal might know
17:19:58 EmilienM I'll try with ::1
17:20:30 EmilienM mriedem: triple jobs have ipv6 coverage, fyi
17:21:42 openstackgerrit Dan Smith proposed openstack/nova master: Update the upgrades part of devref https://review.openstack.org/428824
17:21:43 EmilienM mriedem: btw, I saw your comment on https://review.openstack.org/#/c/428424/ - and I think we should have the job asap since things are broken for us, it would be helpful on both sides to have the job in place without having to run "check experimental".

Earlier   Later