| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-20 | |||
| 15:01:26 | zioproto | I am building the ubuntu packages for 14.0.5 ... but I will wait a 14.0.6 if something still needs to be changed for the database migrations | |
| 15:01:38 | lyarwood | mdbooth: AFAIK it's in and we have some config support for it in the libvirt driver but it's not wired up at the moment | |
| 15:01:47 | dansmith | zioproto: so I'm going to give you another patch to try and see if you get farther, but it's not the fix I want, so I'll ask you to test the real fix as well, okay? | |
| 15:01:51 | lyarwood | mdbooth: https://github.com/openstack/cinder/blob/master/cinder/api/contrib/volume_actions.py#L192-L208 | |
| 15:01:56 | mriedem | lyarwood: mdbooth: terminate_connection in cinder is an RPC call to the cinder-volume backend | |
| 15:01:58 | zioproto | dansmith: sure ! | |
| 15:02:04 | mdbooth | lyarwood: Sweet, because the real solution here is to stop doing this and use qemu. | |
| 15:02:09 | mriedem | lyarwood: that's not what you want | |
| 15:02:24 | alex_xu | mriedem: at least that match the requirement from the operator | |
| 15:02:28 | mriedem | lyarwood: https://github.com/openstack/cinder/blob/master/cinder/volume/api.py#L749 | |
| 15:02:54 | mriedem | https://github.com/openstack/cinder/blob/master/cinder/volume/rpcapi.py#L217 | |
| 15:03:11 | mriedem | alex_xu: ok so i guess i'll just go with that then, and unblock takashin | |
| 15:03:20 | alex_xu | mriedem: thanks | |
| 15:03:23 | mriedem | i can fix the os-migrations api-ref docs (i've been meaning to do that for awhile now), | |
| 15:03:24 | mdbooth | mriedem: It's a call, which is nice. | |
| 15:03:27 | lyarwood | mriedem: yeah I was just getting there, thanks for the links :) | |
| 15:03:31 | takashin | mriedem: alex_xu: Thank you. | |
| 15:03:36 | mriedem | and add --server as a filter option to nova migration-list | |
| 15:03:52 | mriedem | mdbooth: well normally a block rpc call from the api is not nice | |
| 15:03:55 | mriedem | *blocking | |
| 15:04:09 | dansmith | zioproto: https://pastebin.com/yHueQi4s | |
| 15:05:06 | alex_xu | mriedem: I can help you on those | |
| 15:05:08 | mdbooth | mriedem: Nice for me :) | |
| 15:05:15 | mdbooth | But yeah | |
| 15:05:18 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Microversion 2.42 - Fix tag attribute disappearing https://review.openstack.org/429512 | |
| 15:05:35 | mriedem | sdague: another one if you have time https://review.openstack.org/#/c/429512/ - unblocks novaclient for new microversions in pike | |
| 15:06:45 | zioproto | dansmith: should I revert the previous patch with the 'raise' ? | |
| 15:06:56 | dansmith | zioproto: no | |
| 15:07:10 | dansmith | zioproto: the raise lets us see if I typo'd this one :P | |
| 15:07:33 | zioproto | dansmith: https://pastebin.com/eLsiFn9W | |
| 15:08:08 | kashyap | mdbooth: So, no, upstream QEMU's built-in iSCSI initiator doesn't support multipathing yet | |
| 15:08:29 | kashyap | mdbooth: And I haven't seen any related to it for over 2 years | |
| 15:09:12 | dansmith | zioproto: hmm, that ignoring line should mean we're skipping the thing that gives us the error | |
| 15:09:30 | dansmith | zioproto: I hate to ask, but did you apply with patch or paste it in by hand? | |
| 15:09:46 | mdbooth | kashyap: Sucks :( | |
| 15:09:52 | zioproto | I pasted by hand ! | |
| 15:10:04 | dansmith | zioproto: any chance you missed the 'continue' line under the log? | |
| 15:10:14 | dansmith | zioproto: L35 in the paste | |
| 15:10:28 | lyarwood | kashyap / mdbooth ; AFAIK the lack of multipath support is the reason it was never wired up in Nova | |
| 15:10:43 | zioproto | and you are right ! | |
| 15:10:44 | mdbooth | lyarwood: Right. | |
| 15:10:48 | kashyap | mdbooth: Hey, on the brighter hand, QEMU 2.9 just released, with some "robust" improvements | |
| 15:11:58 | zioproto | dansmith: https://pastebin.com/LeH3HudT | |
| 15:12:05 | zioproto | sorry for missing the 'continue' | |
| 15:12:16 | openstackgerrit | STEW TY proposed openstack/nova master: Transform instance.unrescue notifications https://review.openstack.org/388275 | |
| 15:12:39 | dansmith | zioproto: no problem :) | |
| 15:13:10 | dansmith | zioproto: okay, so you have a bunch of instances with keypairs that have since been deleted... | |
| 15:13:26 | zioproto | also the instances have been deleted I guess | |
| 15:13:26 | mriedem | efried: one new question from me inline https://review.openstack.org/#/c/438598/12 | |
| 15:13:37 | efried | mriedem Ack, looking... | |
| 15:13:39 | openstackgerrit | Steve Noyes proposed openstack/nova master: Update detach to use V3 Cinder API https://review.openstack.org/438750 | |
| 15:13:41 | zioproto | dansmith: I think we are talking about the instances that are marked as deleted | |
| 15:14:10 | zioproto | dansmith: are you going to submit this patch for review in master ? | |
| 15:14:17 | dansmith | zioproto: no, let me get you an example | |
| 15:14:29 | mriedem | efried: same question about exception handling in power_off | |
| 15:14:37 | mriedem | we try to avoid handling Exception if possible | |
| 15:14:39 | dansmith | zioproto: 2017-04-20 15:11:10.584 787 WARNING nova.objects.instance [req-d1cbe1e7-1f03-46b6-9009-bba793eb73ba - - - - -] Instance b750d102-4fbb-4401-80b2-cc36888c9749 keypair c_rally_5aabcf4c_Xg4N7es6 not found | |
| 15:14:49 | efried | mriedem Roger that. Responding. | |
| 15:14:50 | dansmith | zioproto: yeah, we'll fix in master and backport | |
| 15:15:01 | sdague | mriedem: so, given that powervm driver is pretty self contained, is there particular primary concerns before merging? | |
| 15:15:06 | bauzas | oh man, /me needs to get drugs in order to think about https://review.openstack.org/#/c/437424/ | |
| 15:15:16 | bauzas | let's prepare a new battle | |
| 15:15:27 | zioproto | dansmith: ok, please give me the review number, I will cherry-pick this patch in my deployment or I cant upgrade next week | |
| 15:15:36 | openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for resource providers https://review.openstack.org/457532 | |
| 15:15:37 | openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for allocations https://review.openstack.org/457534 | |
| 15:15:37 | openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for inventories https://review.openstack.org/457533 | |
| 15:15:38 | openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: tests: add a hook for functional testing in the gate https://review.openstack.org/452122 | |
| 15:15:38 | openstackgerrit | Roman Podoliaka proposed openstack/osc-placement master: CLI for usages https://review.openstack.org/457535 | |
| 15:15:46 | dansmith | zioproto: yeah, but we need to get the problem fully fixed first :) | |
| 15:15:51 | mriedem | sdague: i had questions in the 2nd patch | |
| 15:16:23 | zioproto | I am available to test more patches :) | |
| 15:16:37 | zioproto | dansmith: should I open a bug on LP ? | |
| 15:16:49 | dansmith | zioproto: yep, working on a new patch | |
| 15:17:09 | dansmith | zioproto: yeah, please do file a bug, we'll need it for the review | |
| 15:17:56 | efried | sdague Not sure if you've been aware, but FYI we have a CI running (a subset of) tempest on the in-tree patches, and publishing to gerrit. | |
| 15:18:12 | sdague | efried: while you are changing things, I think you misunderstand LOG.exception(e) :) | |
| 15:18:22 | efried | sdague Entirely possible ;-) | |
| 15:18:28 | efried | Please enlighten me. | |
| 15:19:00 | imacdonn | mdbooth, you asked earlier if a rescan would make the device(s) go away .. if I'm interpreting your question properly, the answer is "no" ... rescan picks up new LUNs, but doesn't do anything about ones that have become unresponsive | |
| 15:19:12 | efried | Thought it was LOG.error + stack trace | |
| 15:19:18 | sdague | efried: right | |
| 15:19:22 | mdbooth | imacdonn: Ok, thanks | |
| 15:19:49 | sdague | efried: comment left | |
| 15:19:54 | efried | sdague Thanks. | |
| 15:20:52 | openstack | bugzilla.redhat.com bug 1367805 in openstack-nova "Previously removed iSCSI multipath volumes can be rediscovered by future disconnect_volume calls before being unmapped on the backend by os-terminate_connection." [High,Closed: insufficient_data] - Assigned to lyarwood | |
| 15:20:52 | lyarwood | mdbooth / imacdonn ; https://bugzilla.redhat.com/show_bug.cgi?id=1367805 is a dup of this that I could never reproduce correctly FWIW | |
| 15:21:06 | mdbooth | imacdonn lyarwood: So what I think we need to achieve here is that after calling disconnect nothing is subsequently allowed to connect until the associated terminate connection has defintely resulted in the device having been killed by cinder | |
| 15:21:22 | imacdonn | lyarwood, \o/ I'm not alone ;) | |
| 15:22:00 | openstackgerrit | STEW TY proposed openstack/nova master: Transform instance.unrescue notifications https://review.openstack.org/388275 | |
| 15:22:21 | lyarwood | mdbooth: yes and the locking in your change would do that now | |
| 15:22:33 | imacdonn | mdbooth, right .. and I've sortof hacked that by putting a lock on things that cover that window .. like _shutdown_instance() in compute.manager | |
| 15:23:04 | imacdonn | mdbooth, and the same lock on connect_volume() in virt.libvirt.volume.iscsi | |
| 15:23:10 | efried | sdague So I don't need to pass the exception itself to LOG.exception - it picks it up automatically? | |
| 15:23:25 | mriedem | efried: yeah | |
| 15:23:31 | efried | Beautiful, thanks. | |
| 15:23:33 | mriedem | LOG.exception() just takes a message if you want to say something | |
| 15:23:38 | mriedem | and it dumps the current stacktrace | |
| 15:23:56 | mdbooth | lyarwood: Assuming we can verify the sync behaviour of terminate connection all the way down | |
| 15:24:07 | sdague | yeh, it's probably worth logging the instance there, I put a suggested string in a comment right now | |
| 15:24:41 | imacdonn | mdbooth, lyarwood AFAIK, it's sync as far as nova-compute is concerned - it's making an API call to cinder ... it's an RPC call within cinder, but nova-compute doesn't know about that | |