| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-19 | |||
| 14:08:59 | jroll | ah, I see your +Ws now, thank you! :) | |
| 14:11:40 | openstack | Launchpad bug 1636489 in OpenStack Compute (nova) "Volume attachment fails for all the available instances after running different volume operations for 1-2 hours or more." [Undecided,New] | |
| 14:11:40 | Sudha | Hi all , ANy thoughts on https://bugs.launchpad.net/nova/+bug/1636489 ? | |
| 14:12:33 | sambetts | johnthetubaguy, jroll: awesome, I hope we can fix it with the atomic stuff next cycle :) its really edge casey anyway | |
| 14:12:43 | jroll | yeah, +1 | |
| 14:14:07 | johnthetubaguy | sambetts: jroll: sorry for the firedrill, mostly getting a heap of stuff re-discovered and straightened out in my head | |
| 14:14:20 | jroll | johnthetubaguy: no worries, I learned a thing today \o/ | |
| 14:14:41 | sambetts | johnthetubaguy: \o/ yeah big TIL moment | |
| 14:22:48 | bauzas | do we have something wonkey with the py35 unit tests ? | |
| 14:23:26 | bauzas | that looks weirdo | |
| 14:24:02 | bauzas | dims: have we recently merged something for py35 ? | |
| 14:24:11 | archies | @Nova team, I have a case where I need your help to understand better. | |
| 14:24:13 | bauzas | it seems we have a critical issue | |
| 14:24:23 | dims | nope bauzas | |
| 14:24:33 | bauzas | lemme look at logstash | |
| 14:24:33 | archies | @all, In one of my case, Nova's "detach_volume" fires the detach method into libvirt, which claims success, but the device is still attached as per the guest XML file. | |
| 14:24:40 | sdague | bauzas: is there a link to a fail? | |
| 14:24:50 | bauzas | sdague: eg. http://logs.openstack.org/34/418134/7/check/gate-nova-python35-db/578aaff/testr_results.html.gz | |
| 14:25:00 | bauzas | zuul tells me all nova checks are red | |
| 14:25:44 | sdague | bauzas: https://pypi.python.org/pypi/mox3/0.19.0 ... | |
| 14:26:00 | bauzas | sdague: opening a bug | |
| 14:28:21 | sdague | bauzas: there are only a couple of commits - http://git.openstack.org/cgit/openstack/mox3/commit/?id=4a85ff0ec4ceb98ad87bec6002f6e67788b06d00 | |
| 14:28:51 | openstack | Launchpad bug 1657757 in OpenStack Compute (nova) "py35 unit tests are hugely failing" [Critical,New] | |
| 14:28:51 | bauzas | sdague: https://bugs.launchpad.net/nova/+bug/1657757 | |
| 14:29:04 | jroll | s/hugely/bigly/ | |
| 14:29:19 | sdague | bauzas: there are only 4 patches since the last release, the inspect one is the only of consequence | |
| 14:29:34 | mriedem | new release broke everything? | |
| 14:30:04 | bauzas | mriedem: sdague: trying to bisect | |
| 14:30:30 | sdague | bauzas: it has to be that patch I posted | |
| 14:32:19 | bauzas | sdague: probably, I just wanted to see the logstash query | |
| 14:32:55 | bauzas | mriedem: sdague: tried with the query "project:openstack/nova AND build_status:FAILURE AND build_name:gate-nova-python35-db" but not sure that one is good | |
| 14:36:09 | bauzas | sdague: mriedem: trying to run the job locally with an older mox release | |
| 14:36:24 | sdague | https://review.openstack.org/422651 is the revert | |
| 14:37:11 | openstackgerrit | Andrey Kurilin proposed openstack/python-novaclient: Pass relevant parameters to Token based authentication https://review.openstack.org/419441 | |
| 14:41:58 | mriedem | sdague: https://review.openstack.org/#/c/422653/ | |
| 14:42:05 | mriedem | should stem the bleeding | |
| 14:42:16 | mriedem | i need to follow up with a g-r blacklist | |
| 14:43:21 | sdague | mriedem: I was doing that right now | |
| 14:44:29 | sdague | mriedem: you need the block, otherwise this is going to be broken within an hour with updates | |
| 14:44:41 | sdague | mriedem: https://review.openstack.org/422656 instead | |
| 14:46:24 | bhagyashris | jaypipes: cdent: Hi | |
| 14:49:58 | mriedem | need a +W on this https://review.openstack.org/#/c/421482/ so oslo.serialization can get their final release in for ocata today | |
| 14:51:01 | mriedem | harlowja: ^ it is done | |
| 14:52:26 | vladikr | mriedem, could you please remove your -2 from https://review.openstack.org/#/c/405951 ? - It's all rewritten now | |
| 14:52:51 | mriedem | done | |
| 14:53:00 | vladikr | mriedem, thanks | |
| 14:53:08 | mriedem | vladikr: so you were able to condense that whole series into 2 changes now? | |
| 14:53:53 | vladikr | mriedem, looks like it. removing all the classes in the middle, there is not much left | |
| 14:54:15 | openstackgerrit | Eric Fried proposed openstack/nova: Introduce base PowerVM driver (disk drv) https://review.openstack.org/409401 | |
| 14:57:48 | bauzas | sdague: got a sec for a problem related to the placement call from the scheduler ? | |
| 14:58:14 | bauzas | sdague: it seems we default calling the placement service from the reportclient by not using the latest microversion | |
| 14:58:16 | bauzas | http://logs.openstack.org/61/417961/10/check/gate-tempest-dsvm-cells-ubuntu-xenial/8a974f7/logs/screen-n-sch.txt.gz?level=TRACE#_2017-01-19_14_17_38_980 | |
| 14:58:47 | bauzas | sdague: since cdent is on PTO today, do you know how could I add this header ? | |
| 15:00:12 | bauzas | sdague: I mean, in https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L172 | |
| 15:00:57 | mriedem | bauzas: jaypipes has a patch that does that, sec | |
| 15:01:10 | bauzas | mriedem: oh cool | |
| 15:01:25 | mriedem | https://review.openstack.org/#/c/407309/13/nova/scheduler/client/report.py | |
| 15:01:32 | mriedem | the problem there is he's requesting 'latest', | |
| 15:01:49 | mriedem | we don't want the report client to request 'latest', we want a specific version that we're opting into | |
| 15:02:02 | mriedem | because at some point latest in pike might break an ocata client | |
| 15:02:52 | bauzas | mriedem: I can try to amend that patch | |
| 15:03:11 | bauzas | mriedem: we're having a max version for our novaclient, right? | |
| 15:03:19 | mriedem | bauzas: yes | |
| 15:03:27 | mriedem | there is a max version in novaclient for what it knows how to understand | |
| 15:03:45 | bauzas | yeah, I remember | |
| 15:03:48 | mriedem | by default the python api in novaclient always requests the minimum | |
| 15:03:49 | bauzas | and a min one too | |
| 15:04:16 | bauzas | yeah, unless you're specifically asking for another version in your resource call | |
| 15:04:23 | bauzas | I remember that too | |
| 15:05:28 | bauzas | mriedem: I could be doing that, like calling MIN_VERSION by default and just ask for 1.4 for my placement call in the scheduler | |
| 15:05:37 | mriedem | bauzas: looks like you can just past the headers kwarg to session.get() https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L485 | |
| 15:05:49 | mriedem | *pass | |
| 15:06:10 | bauzas | yeah | |
| 15:06:12 | bauzas | saw that | |
| 15:06:20 | mriedem | the question is what if the minimum available isn't new enough for what you need? | |
| 15:06:21 | bauzas | but I'd prefer to have our reportclient having that | |
| 15:07:05 | bauzas | mriedem: I think the safe_connect deco should say sorry then | |
| 15:07:05 | mriedem | now that i'm thinking about this, don't people usually upgrade the api last? | |
| 15:07:21 | mriedem | yeah but what does that mean for the filter scheduler? | |
| 15:07:25 | openstackgerrit | Sam Betts proposed openstack/nova: Change order of _cleanup_deploy and _unprovision in Ironic virt https://review.openstack.org/422678 | |
| 15:07:57 | bauzas | mriedem: it would mean that you'd have to run the Ocata placement service before upgrading nova-scheduler | |
| 15:08:10 | bauzas | EmilienM: ^ | |
| 15:09:00 | EmilienM | bauzas: that kind of orchestration is not easy to do, specially when you deploy in production | |
| 15:09:09 | bauzas | I know | |
| 15:09:40 | mriedem | bauzas: but that also means upgrading nova-api to ocata doesn't it? unless you package them as separate services | |
| 15:10:52 | mriedem | do i need to start a ML thread to think this through? | |
| 15:10:53 | bauzas | mriedem: that indeed depends on how it is packed | |
| 15:10:59 | bauzas | mriedem: that looks important | |
| 15:11:09 | bauzas | and we're 1 week away from FF | |
| 15:11:13 | bauzas | which makes me nervous | |
| 15:11:39 | mriedem | well let me start a thread first and we can stew in there | |
| 15:12:33 | mriedem | johnthetubaguy: keep me honest about the order of upgrade, with n-api coming last | |
| 15:12:55 | mriedem | it seems this changes every time i talk about it, might be n-cond -> n-api -> n-cpus | |
| 15:13:05 | bauzas | in general yes | |
| 15:13:26 | mriedem | http://docs.openstack.org/developer/nova/upgrade.html | |
| 15:13:32 | mriedem | " It is safest to start nova-conductor first and nova-api last." | |
| 15:13:35 | bauzas | we did the modification with dan's change so that newton computes auto-heal once they find a placement service | |
| 15:13:47 | mriedem | yeah i'm not worried about n-cpu | |
| 15:13:59 | bauzas | but we haven't tell'm what version of placement they should expect | |
| 15:15:30 | johnthetubaguy | mriedem: so it coming last is kind of a new thing | |