| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-11 | |||
| 14:26:23 | dtantsur | folks, can someone please take a look at https://review.openstack.org/#/c/508555/ ? it fixes our CI apparently | |
| 14:26:28 | dtantsur | mriedem, jaypipes ^^^ | |
| 14:26:55 | jaypipes | dtantsur: yes, will do. | |
| 14:27:08 | cdent | dtantsur: so that was it? I’m surprised it didn’t show up a bit sooner. | |
| 14:27:25 | dtantsur | cdent: all I can say that a patch with Depends-On it has passed the CI :) | |
| 14:28:07 | cdent | well it presumably needs to happen anyway, so motion in the right direction | |
| 14:28:16 | dtantsur | yep | |
| 14:28:42 | openstackgerrit | Merged openstack/python-novaclient master: Remove deprecated command in nova.rst https://review.openstack.org/526569 | |
| 14:29:02 | jaypipes | mriedem: yeah, that's sort of an odd place for the deadlock to occur. that basically means there are two greenthreads that are attempting to simultaneously (or near-simultaneously) calling action_event_finish() on an action with the *same* request ID. | |
| 14:29:20 | jaypipes | mriedem: which doesn't seem like it should happen, no? | |
| 14:29:41 | jaypipes | mriedem: i.e. there should not be >1 greenthread attempting to "finish" a particular task/event at the same time. | |
| 14:30:04 | jaypipes | mriedem: unless request ID is not a UUID? | |
| 14:30:16 | jaypipes | mriedem: if request ID is an auto-inc, it could happen. | |
| 14:30:58 | mriedem | request ID should be unique per user request | |
| 14:31:19 | mriedem | generated by some keystone or oslo middleware i tihnk | |
| 14:31:32 | jaypipes | mriedem: hmm, not necessarily... | |
| 14:31:33 | mriedem | https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/request_id.py | |
| 14:31:58 | mriedem | the failures all seem to happen on a resize test | |
| 14:32:04 | jaypipes | mriedem: see the comment and code on line 6302-6310 | |
| 14:32:37 | diga | jaypipes: Hi | |
| 14:32:38 | jaypipes | mriedem: seems that on a service restart, the action is retrieved by getting the "last created by instance UUID" | |
| 14:32:57 | jaypipes | mriedem: I'm wondering if that is returning an action ID that another greenthread might be operating on? | |
| 14:33:15 | mriedem | jaypipes: maybe, | |
| 14:33:15 | jaypipes | diga: hey. just saw your email. will email you an answer, ok? | |
| 14:33:22 | mriedem | we wouldn't have a compute restart in the middle of this tempest run | |
| 14:33:22 | diga | jaypipes: I have my setup ready but how to reproduce the issue is the big changel I am facing | |
| 14:33:24 | diga | jaypipes: sure | |
| 14:33:35 | mriedem | jaypipes: but we probably shouldn't update those kinds of actions anyway | |
| 14:33:37 | mriedem | so good spot | |
| 14:34:15 | mriedem | jaypipes: this is failing resize testing where we're rpc'ing back and forth with the same user context, | |
| 14:34:28 | mriedem | to decorated methods like prep_resize, resize_instance, finish_resize, etc | |
| 14:34:34 | mriedem | so it's possible we are hitting this quickly | |
| 14:35:54 | mriedem | jaypipes: yeah i think that's probably what's happening here | |
| 14:36:54 | mriedem | prep_resize updates it on the dest, rpc casts to resize_instance on the source which updates it again, which rpc casts back to the dest to finish_resize which updates the action again | |
| 14:36:59 | mriedem | 3 updates within a short window | |
| 14:39:37 | mriedem | jaypipes: left comments, thanks for the help | |
| 14:40:07 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707 | |
| 14:40:07 | openstackgerrit | Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357 | |
| 14:40:08 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 14:40:08 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 14:40:10 | edleafe | mriedem: ^^ rebased | |
| 14:40:13 | mriedem | yar | |
| 14:40:49 | jaypipes | mriedem: not sure I really helped, but ok :) | |
| 14:41:06 | mriedem | you greased my brain gears | |
| 14:42:18 | openstackgerrit | Dan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation https://review.openstack.org/507638 | |
| 14:42:30 | superdan | mriedem: that xen thing merged and this is fixed up ^ | |
| 14:42:33 | superdan | whoa | |
| 14:51:38 | mriedem | dansmith: service version needs to be fixed | |
| 14:51:43 | mriedem | otherwise lgtm | |
| 14:51:49 | dansmith | eh I bumped it? | |
| 14:52:04 | mriedem | rebase fail | |
| 14:52:11 | dansmith | oh doh | |
| 14:52:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation https://review.openstack.org/507638 | |
| 14:52:54 | dansmith | dammit, hang on | |
| 14:53:06 | dansmith | sheesh | |
| 14:53:08 | openstackgerrit | Dan Smith proposed openstack/nova master: Make live migration hold resources with a migration allocation https://review.openstack.org/507638 | |
| 14:53:22 | dansmith | did half of that on friday and then completed before max caffeine absorption = fail | |
| 14:57:30 | mriedem | +2 | |
| 14:58:06 | mriedem | gibi: you've been on this series before https://review.openstack.org/#/c/507638/ | |
| 14:58:09 | mriedem | ^ is the last patch | |
| 15:00:39 | jaypipes | mriedem: easy one. https://review.openstack.org/#/c/526973/ | |
| 15:05:02 | mriedem | jaypipes: this must be a response sample? https://review.openstack.org/#/c/526973/1/placement-api-ref/source/update-resource_provider.json | |
| 15:05:48 | mriedem | yeah i guess http://logs.openstack.org/73/526973/1/check/build-placement-api-ref/cbc7192/html/#id11 | |
| 15:06:04 | jaypipes | yessir | |
| 15:06:39 | mriedem | it's weird that we have 2 source files for resource providers for api-ref | |
| 15:06:44 | mriedem | once has create/list, the other has show/update | |
| 15:06:51 | mriedem | *and delete | |
| 15:08:47 | mriedem | anyway something takashin could fix up later, | |
| 15:08:48 | mriedem | +W | |
| 15:09:35 | mriedem | lyarwood: did you say you're back now? | |
| 15:09:49 | lyarwood | mriedem: I am indeed | |
| 15:10:01 | mriedem | lyarwood: ok, we have a bunch of pike backports which need a final look https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/pike | |
| 15:10:02 | mriedem | if you would | |
| 15:10:09 | mriedem | i need to do a release by end of day | |
| 15:10:18 | lyarwood | mriedem: ack, did a few at the end of the list earlier, I'll start on these now | |
| 15:10:38 | mriedem | thanks | |
| 15:11:24 | lyarwood | mriedem: should I take anything that you have pushed as already having a +2 btw? Noticed you did that with something of mine the other day.. | |
| 15:11:52 | lyarwood | mriedem: I'd rather not tbh, just wondering if that's now policy on stable or something | |
| 15:12:53 | mriedem | lyarwood: no you don't have to do that unless it's trivial - i was just referring to the other +2'ed patches in pike right now | |
| 15:12:58 | mriedem | dansmith did a bunch of those +2s already | |
| 15:13:13 | lyarwood | mriedem: kk cool | |
| 15:14:38 | mriedem | andreas_s: zkvm ci failed in 15 minutes on a patch on master http://sng01.objectstorage.softlayer.net/v1/AUTH_1940ea10-6e82-4501-b2f9-eb236510e575/ibmzkvmci/production/507638/29/check-tempest-dsvm-neutron-full-ubuntu-xenial-s390x/4e9ba43/logs/devstacklog.txt.gz | |
| 15:14:42 | mriedem | 2017-12-11 15:06:27.360 | cp: cannot stat '/opt/stack/new/devstack/files/etcd-v3.1.7-linux-s390x/etcdctl': No such file or directory | |
| 15:15:30 | andreas_s | mriedem: hi, saw it earlier the day - a new nodepool image with the updated binary has been build 20 minutes ago, hopefully new test runs will suceed... | |
| 15:15:41 | mriedem | ok | |
| 15:34:18 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: introduce framework for RFB authentication https://review.openstack.org/345397 | |
| 15:34:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: introduce the VeNCrypt RFB authentication scheme https://review.openstack.org/345398 | |
| 15:34:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: Provide an RFB security proxy implementation https://review.openstack.org/345399 | |
| 15:35:29 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Document TLS security setup for noVNC proxy https://review.openstack.org/500544 | |
| 15:35:34 | openstackgerrit | Jay Pipes proposed openstack/nova master: handle traits with sharing providers https://review.openstack.org/517119 | |
| 15:35:35 | openstackgerrit | Jay Pipes proposed openstack/nova master: Test alloc_cands with non overlapping sharing RPs https://review.openstack.org/519380 | |
| 15:35:35 | openstackgerrit | Jay Pipes proposed openstack/nova master: Test alloc_cands with one RP shared between two RPs https://review.openstack.org/519617 | |
| 15:35:35 | openstackgerrit | Jay Pipes proposed openstack/nova master: Test allocation candidates: multiple aggregates https://review.openstack.org/518633 | |
| 15:35:36 | openstackgerrit | Jay Pipes proposed openstack/nova master: Test helper: validate provider summaries https://review.openstack.org/518982 | |
| 15:35:36 | openstackgerrit | Jay Pipes proposed openstack/nova master: Fix accumulated nits in refactor series https://review.openstack.org/521189 | |
| 15:35:37 | openstackgerrit | Jay Pipes proposed openstack/nova master: Add aggregates check in allocation candidates https://review.openstack.org/522407 | |
| 15:35:37 | openstackgerrit | Jay Pipes proposed openstack/nova master: Fix missing rps in allocation candidates https://review.openstack.org/522409 | |
| 15:35:38 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: func tests for multiple shared RPs https://review.openstack.org/498737 | |
| 15:52:19 | gibi | mriedem: went through https://review.openstack.org/#/c/507638/ and +W-d it | |
| 15:52:40 | mriedem | gibi: thanks | |