| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-10 | |||
| 16:46:19 | lyarwood | jgriffith: yeah as above I can't see how adding reserved helps with this non-multiattach race tbh | |
| 16:46:50 | lyarwood | jgriffith: we only expect available or downloading in that case right? | |
| 16:46:57 | mriedem | yeah | |
| 16:47:17 | jgriffith | lyarwood: yes | |
| 16:47:20 | mriedem | and yes i think adding 'reserved' would only be for racing to attach the same multiattach volume to separate instances | |
| 16:47:29 | jgriffith | mriedem: +1 | |
| 16:47:31 | lyarwood | kk | |
| 16:47:37 | lyarwood | just checking, thanks | |
| 16:47:42 | mriedem | so if just the volume refresh fixes it, then i could remove the 'reserved' part of this patch, and that can be done later if it's a problem | |
| 16:52:56 | lyarwood | mriedem: yup I'd drop it for now tbh but it's really up to the cinder folks | |
| 16:53:41 | mriedem | lyarwood: if it fixes your issue for non-multiattach volumes then i'm happy to simplify the patch | |
| 16:53:52 | mriedem | i need some time to write a test anyway | |
| 17:08:59 | openstack | Launchpad bug 1696125 in OpenStack Compute (nova) "Detach interface failed - timeout waiting to detach tap device in linuxbridge job (pike)" [High,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:08:59 | cfriesen_ | mriedem: did you ever get any further with https://bugs.launchpad.net/nova/+bug/1696125 ? I think we're seeing it too, though it showed up in the guise of a stalled heat stack deletion. | |
| 17:11:24 | openstackgerrit | Sam Yaple proposed openstack/nova stable/pike: Fix wrapping of neutron forbidden error https://review.openstack.org/560087 | |
| 17:11:40 | openstackgerrit | Sam Yaple proposed openstack/nova stable/ocata: Fix wrapping of neutron forbidden error https://review.openstack.org/560088 | |
| 17:12:04 | cfriesen_ | mriedem: and we're not using linuxbridge | |
| 17:16:13 | mriedem | cfriesen_: no, slipped out of mind since we're not hitting it in the gate anymore | |
| 17:17:26 | mriedem | SamYaple: you have to first backport that to stable/queens | |
| 17:17:33 | mriedem | oh wait | |
| 17:17:37 | SamYaple | mriedem: its in stable queens | |
| 17:17:38 | SamYaple | i checked | |
| 17:17:42 | mriedem | yeah :) | |
| 17:17:43 | mriedem | nvm | |
| 17:20:22 | SamYaple | yea it got me too. its just a 5 month old branch, when queens was still master | |
| 17:20:28 | SamYaple | 5 month old patch* | |
| 17:21:02 | kashyap | dansmith: Just read the scroll. Thanks for the explanation | |
| 17:21:56 | kashyap | dansmith: And yes, I did realize I had to remove 'test.nested'. Just didn't commit to it in the paste-bin. /me tinkers a bit | |
| 17:26:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Make ResourceClass.normalize_name handle sharp S https://review.openstack.org/560093 | |
| 17:26:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Test case: ResourceClass.normalize_name with ß https://review.openstack.org/560092 | |
| 17:30:14 | melwitt | efried: how did you find that? ^ | |
| 17:30:29 | efried | melwitt: Nice one, right? | |
| 17:30:37 | melwitt | yeah o.O | |
| 17:30:52 | efried | I actually found it while working on https://review.openstack.org/#/c/556628/ which I'm now going to abandon (because reasons I'm posting here is a minute) | |
| 17:31:43 | efried | (but mostly because jaypipes is right) | |
| 17:33:12 | mriedem | we might have a new dbdeadlock http://logs.openstack.org/14/554314/1/gate/tempest-full-py3/1aa1216/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_10_13_49_39_294710 | |
| 17:38:12 | artom | (null) | |
| 18:01:25 | openstackgerrit | Eric Fried proposed openstack/os-traits master: normalize_name helper https://review.openstack.org/560107 | |
| 18:15:11 | openstackgerrit | Ed Leafe proposed openstack/nova-specs master: Add Generation to Consumers https://review.openstack.org/556971 | |
| 18:17:44 | edleafe | dansmith: ^^ Addressed your comments | |
| 18:18:31 | efried | edleafe: Thanks, that lgtm | |
| 18:22:16 | simondodsley | anyone help with https://review.openstack.org/#/c/558530 that breaks pep8 with a C901 'LibvirtConfigGuestDisk.format_dom' is too complex (40) error? My Python skills are not up to re-coding that module with confidence | |
| 18:22:23 | openstackgerrit | Jimmy McCrory proposed openstack/nova master: Pass user context detaching volume on exceptions https://review.openstack.org/560115 | |
| 18:22:31 | openstackgerrit | Eric Fried proposed openstack/os-traits master: normalize_name helper https://review.openstack.org/560107 | |
| 18:22:32 | dansmith | efried: well, I still wanted more detail on the userid history thing, but I guess I'm the only one | |
| 18:23:22 | efried | dansmith: Oh, yeah, edleafe did you want to address that? | |
| 18:23:45 | efried | I would volunteer to write it up, but I'm not 100% sure I understand it well enough to articulate it. | |
| 18:25:35 | melwitt | simondodsley: I think that just means the code you added pushed the format_dom method length over the limit for "complexity". so you just need to break some of the code in format_dom into a private helper method and call it from format_dom | |
| 18:28:13 | simondodsley | melwitt: Yeah - I get what the error is, but the whole section of that module is repeating if statements. I'm not sure I want to modify code that I didn't add for my first attempt at messing with core Nova code :) | |
| 18:32:42 | edleafe | dansmith: I didn't work on the user_id changes, so my knowledge of the whys and wherefores is limited | |
| 18:33:17 | melwitt | simondodsley: oh, I see. I think what I would do is add a refactor patch to stack under your patch that adds a method like "iotune = self._get_iotune()" and puts all of the iotune.append stuff in there. then your additions would go in _get_iotune | |
| 18:34:02 | efried | edleafe: Is http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-04-10.log.html#t2018-04-10T15:24:58 roughly correct to your understanding? | |
| 18:34:42 | melwitt | that said, it looks like you need some review on your spec https://review.openstack.org/#/c/340168 (I will take a look) | |
| 18:35:11 | openstackgerrit | Merged openstack/nova-specs master: Add Generation to Consumers https://review.openstack.org/556971 | |
| 18:36:04 | edleafe | efried: yeah, that's pretty much it | |
| 18:36:23 | efried | I can have a crack at an edit. | |
| 18:36:40 | efried | Because when dansmith is happy, everyone's happy. | |
| 18:37:02 | efried | Or maybe it's the inverse. | |
| 18:37:14 | edleafe | efried: wasn't there some discussion about trying to backfill those values, using the consumer_uuid to find the project/user for that VM | |
| 18:37:30 | efried | oh, sheesh, I don't remember any of that. | |
| 18:37:35 | edleafe | but that that wouldn't work for non-VM consumers | |
| 18:39:28 | edleafe | yeah, that was a while ago, well before the consumer generation stuff | |
| 18:43:44 | eandersson | cfriesen_, do you think the numa migration (not live) issues are fixed in newer releases? | |
| 18:44:04 | eandersson | I haven't had time to build a non-mitaka lab yet to run the same tests | |
| 19:08:16 | cfriesen_ | eandersson: I don't regularly run migration/evacuate/resize tests on upstream master, so I can't say for sure. I'm also confused why it's not working for you. Did you ever get a chance to test with debug logging enabled to figure out why the claim logs weren't coming out? | |
| 19:08:37 | cfriesen_ | eandersson: generally I'd expect more recent versions to do a better job of resource tracking | |
| 19:08:40 | eandersson | I think I pasted you anything relavent | |
| 19:08:59 | eandersson | Yea - I'll try to set up a lab with pike or newer this week | |
| 19:09:08 | eandersson | need to do some research for our upgrade path anyway | |
| 19:30:05 | cfriesen_ | dansmith: in https://git.openstack.org/cgit/openstack/nova/commit/?id=804215017f38300ac429e4401ce9c23f0c8ff3c6 you have a comment suggesting that failing to detach a volume is expected and reasonable. Was there a fix to keep heat from waiting forever if this happens when deleting a stack? | |
| 19:53:32 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add columns for generation to consumers https://review.openstack.org/557958 | |
| 19:53:33 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add handlers for consumer_generation https://review.openstack.org/557959 | |
| 19:53:34 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add a microversion for consumer generation https://review.openstack.org/558958 | |
| 19:53:47 | edleafe | efried: ^^ new revs for consumer generation. Still have a couple of functional tests failing in the last patch, but otherwise they're ready for review | |
| 19:54:09 | efried | edleafe: ack | |
| 19:54:43 | dansmith | cfriesen_: that is about the instance not releasing the disk, it still raises | |
| 19:55:02 | dansmith | cfriesen_: but no, if heat is going to try to detach things before it deletes an instance, then it's going to keep waiting | |
| 19:55:23 | dansmith | if it's doing a stack delete, then it should just kill the instance and not wait for volume detach which is never going to happen | |
| 19:58:55 | cfriesen_ | dansmith: apparently heat has no way of knowing that the detach failed from nova's perspective, so currently it waits around forever. And I suspect the volume was a separate heat resource and it was trying to tear it down cleanly. (Looks like a stack update could also result in volume detach, which would be subject to the same problem.) | |
| 20:00:03 | dansmith | cfriesen_: I think we log an instance action for it, no? | |
| 20:00:05 | dansmith | but regardless, | |
| 20:00:34 | dansmith | at a minimum it should time out and then try shutting the instance down before doing the detach (or just delete it while it's off) | |
| 20:00:49 | dansmith | actually just deleting it while it's running should work I would think | |
| 20:01:24 | cfriesen_ | dansmith: delete the instance? what about the stack update case where we just want to detach and replace the volume? | |
| 20:01:38 | dansmith | you said stack delet | |
| 20:01:44 | dansmith | and that's where we hit the issue as well | |
| 20:01:56 | dansmith | but, if you're doing an update, then power off and detach | |
| 20:02:20 | dansmith | the deal is the instance didn't let go of the volume, so there's really nothing else to do other than shut it down, or tell the instance to get over itself | |
| 20:02:46 | cfriesen_ | dansmith: fair enough. you mentioned an instance action, where would that get logged? | |
| 20:02:56 | dansmith | in instance actions | |
| 20:04:15 | cfriesen_ | I mean where in the detach volume exception path would we report the failure to detach as an instance action? | |
| 20:04:33 | dansmith | I'm not positive we do, I'm just saying I think we would | |
| 20:04:39 | dansmith | I can look, but so can you :) | |
| 20:05:03 | cfriesen_ | yep..was hoping you knew off the top of your head. Looking. | |
| 20:07:11 | dansmith | nope | |
| 20:08:39 | dansmith | ah yeah, | |
| 20:08:53 | dansmith | cfriesen_: @wrap_instance_event on compute manager detach_volume | |
| 20:08:57 | cfriesen_ | dansmith: looks like ComputeManager.detach_volume() has wrap_instance_fault and wrap_instance_event decorators | |
| 20:09:01 | dansmith | yeah | |
| 20:09:01 | dansmith | I think that decorator ends up with it | |