Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-10
17:30:14 melwitt efried: how did you find that? ^
17:30:29 efried melwitt: Nice one, right?
17:30:37 melwitt yeah o.O
17:30:52 efried I actually found it while working on https://review.openstack.org/#/c/556628/ which I'm now going to abandon (because reasons I'm posting here is a minute)
17:31:43 efried (but mostly because jaypipes is right)
17:33:12 mriedem we might have a new dbdeadlock http://logs.openstack.org/14/554314/1/gate/tempest-full-py3/1aa1216/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_10_13_49_39_294710
17:38:12 artom (null)
18:01:25 openstackgerrit Eric Fried proposed openstack/os-traits master: normalize_name helper https://review.openstack.org/560107
18:15:11 openstackgerrit Ed Leafe proposed openstack/nova-specs master: Add Generation to Consumers https://review.openstack.org/556971
18:17:44 edleafe dansmith: ^^ Addressed your comments
18:18:31 efried edleafe: Thanks, that lgtm
18:22:16 simondodsley anyone help with https://review.openstack.org/#/c/558530 that breaks pep8 with a C901 'LibvirtConfigGuestDisk.format_dom' is too complex (40) error? My Python skills are not up to re-coding that module with confidence
18:22:23 openstackgerrit Jimmy McCrory proposed openstack/nova master: Pass user context detaching volume on exceptions https://review.openstack.org/560115
18:22:31 openstackgerrit Eric Fried proposed openstack/os-traits master: normalize_name helper https://review.openstack.org/560107
18:22:32 dansmith efried: well, I still wanted more detail on the userid history thing, but I guess I'm the only one
18:23:22 efried dansmith: Oh, yeah, edleafe did you want to address that?
18:23:45 efried I would volunteer to write it up, but I'm not 100% sure I understand it well enough to articulate it.
18:25:35 melwitt simondodsley: I think that just means the code you added pushed the format_dom method length over the limit for "complexity". so you just need to break some of the code in format_dom into a private helper method and call it from format_dom
18:28:13 simondodsley melwitt: Yeah - I get what the error is, but the whole section of that module is repeating if statements. I'm not sure I want to modify code that I didn't add for my first attempt at messing with core Nova code :)
18:32:42 edleafe dansmith: I didn't work on the user_id changes, so my knowledge of the whys and wherefores is limited
18:33:17 melwitt simondodsley: oh, I see. I think what I would do is add a refactor patch to stack under your patch that adds a method like "iotune = self._get_iotune()" and puts all of the iotune.append stuff in there. then your additions would go in _get_iotune
18:34:02 efried edleafe: Is http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2018-04-10.log.html#t2018-04-10T15:24:58 roughly correct to your understanding?
18:34:42 melwitt that said, it looks like you need some review on your spec https://review.openstack.org/#/c/340168 (I will take a look)
18:35:11 openstackgerrit Merged openstack/nova-specs master: Add Generation to Consumers https://review.openstack.org/556971
18:36:04 edleafe efried: yeah, that's pretty much it
18:36:23 efried I can have a crack at an edit.
18:36:40 efried Because when dansmith is happy, everyone's happy.
18:37:02 efried Or maybe it's the inverse.
18:37:14 edleafe efried: wasn't there some discussion about trying to backfill those values, using the consumer_uuid to find the project/user for that VM
18:37:30 efried oh, sheesh, I don't remember any of that.
18:37:35 edleafe but that that wouldn't work for non-VM consumers
18:39:28 edleafe yeah, that was a while ago, well before the consumer generation stuff
18:43:44 eandersson cfriesen_, do you think the numa migration (not live) issues are fixed in newer releases?
18:44:04 eandersson I haven't had time to build a non-mitaka lab yet to run the same tests
19:08:16 cfriesen_ eandersson: I don't regularly run migration/evacuate/resize tests on upstream master, so I can't say for sure. I'm also confused why it's not working for you. Did you ever get a chance to test with debug logging enabled to figure out why the claim logs weren't coming out?
19:08:37 cfriesen_ eandersson: generally I'd expect more recent versions to do a better job of resource tracking
19:08:40 eandersson I think I pasted you anything relavent
19:08:59 eandersson Yea - I'll try to set up a lab with pike or newer this week
19:09:08 eandersson need to do some research for our upgrade path anyway
19:30:05 cfriesen_ dansmith: in https://git.openstack.org/cgit/openstack/nova/commit/?id=804215017f38300ac429e4401ce9c23f0c8ff3c6 you have a comment suggesting that failing to detach a volume is expected and reasonable. Was there a fix to keep heat from waiting forever if this happens when deleting a stack?
19:53:32 openstackgerrit Ed Leafe proposed openstack/nova master: Add columns for generation to consumers https://review.openstack.org/557958
19:53:33 openstackgerrit Ed Leafe proposed openstack/nova master: Add handlers for consumer_generation https://review.openstack.org/557959
19:53:34 openstackgerrit Ed Leafe proposed openstack/nova master: Add a microversion for consumer generation https://review.openstack.org/558958
19:53:47 edleafe efried: ^^ new revs for consumer generation. Still have a couple of functional tests failing in the last patch, but otherwise they're ready for review
19:54:09 efried edleafe: ack
19:54:43 dansmith cfriesen_: that is about the instance not releasing the disk, it still raises
19:55:02 dansmith cfriesen_: but no, if heat is going to try to detach things before it deletes an instance, then it's going to keep waiting
19:55:23 dansmith if it's doing a stack delete, then it should just kill the instance and not wait for volume detach which is never going to happen
19:58:55 cfriesen_ dansmith: apparently heat has no way of knowing that the detach failed from nova's perspective, so currently it waits around forever. And I suspect the volume was a separate heat resource and it was trying to tear it down cleanly. (Looks like a stack update could also result in volume detach, which would be subject to the same problem.)
20:00:03 dansmith cfriesen_: I think we log an instance action for it, no?
20:00:05 dansmith but regardless,
20:00:34 dansmith at a minimum it should time out and then try shutting the instance down before doing the detach (or just delete it while it's off)
20:00:49 dansmith actually just deleting it while it's running should work I would think
20:01:24 cfriesen_ dansmith: delete the instance? what about the stack update case where we just want to detach and replace the volume?
20:01:38 dansmith you said stack delet
20:01:44 dansmith and that's where we hit the issue as well
20:01:56 dansmith but, if you're doing an update, then power off and detach
20:02:20 dansmith the deal is the instance didn't let go of the volume, so there's really nothing else to do other than shut it down, or tell the instance to get over itself
20:02:46 cfriesen_ dansmith: fair enough. you mentioned an instance action, where would that get logged?
20:02:56 dansmith in instance actions
20:04:15 cfriesen_ I mean where in the detach volume exception path would we report the failure to detach as an instance action?
20:04:33 dansmith I'm not positive we do, I'm just saying I think we would
20:04:39 dansmith I can look, but so can you :)
20:05:03 cfriesen_ yep..was hoping you knew off the top of your head. Looking.
20:07:11 dansmith nope
20:08:39 dansmith ah yeah,
20:08:53 dansmith cfriesen_: @wrap_instance_event on compute manager detach_volume
20:08:57 cfriesen_ dansmith: looks like ComputeManager.detach_volume() has wrap_instance_fault and wrap_instance_event decorators
20:09:01 dansmith yeah
20:09:01 dansmith I think that decorator ends up with it
20:09:34 efried dansmith: You wanna finish up the paperwork on the bp here https://blueprints.launchpad.net/nova/+spec/add-consumer-generation
20:09:50 cfriesen_ dansmith: So heat should maybe poll those for updates as well as looking for the state change on the volume?
20:10:29 dansmith efried: I hate paperwork
20:10:44 dansmith cfriesen_: I'm guessin' that'd be smart :)
20:11:21 efried dansmith: :*
20:12:07 cfriesen_ time to go check the captured database logs and see if the appropriate info was there.
20:18:11 openstackgerrit Merged openstack/nova stable/queens: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/559700
20:18:51 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/559702
20:24:26 openstackgerrit Surya Seetharaman proposed openstack/nova master: Cleanup RP and HM records while deleting a compute service. https://review.openstack.org/554920
20:29:41 openstackgerrit Merged openstack/nova master: Pick next minimum libvirt / QEMU versions for "Stein" https://review.openstack.org/558171
20:30:13 openstackgerrit Merged openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771
20:31:56 cfriesen_ dansmith: I see relevent entries in "instance_actions_events" and "instance_faults", but I don't know of any API to get either of those. Would the fault entry show up when doing a GET
20:31:57 cfriesen_ GET
20:31:59 cfriesen_ /servers/{server_id}
20:32:06 cfriesen_ whoops, paste error
20:32:07 dansmith cfriesen_: the instance actions api
20:32:15 mriedem GET /os-instance-actions
20:32:23 mriedem the fault is stored on the server
20:32:35 mriedem https://developer.openstack.org/api-ref/compute/#servers-actions-servers-os-instance-actions
20:32:40 dansmith cfriesen_: https://developer.openstack.org/api-ref/compute/#list-actions-for-server
20:32:43 dansmith yeah
20:32:44 mriedem https://developer.openstack.org/api-guide/compute/faults.html
20:32:56 mriedem ^ might be stale
20:33:02 mriedem most things in the compute api guide are old
20:33:45 mriedem e.g. https://developer.openstack.org/api-guide/compute/faults.html#instance-faults
20:33:50 mriedem "However, there is currently no API to retrieve this information."
20:33:52 mriedem that's totally bogus
20:34:19 mriedem GET /server/{id} has a 'fault' key in the response
20:34:20 mriedem "A fault object. Only displayed in the failed response. Default keys are code, created, and message (response code, created time, and message respectively). In addition, the key details (stack trace) is available if you have the administrator privilege."
20:38:21 cfriesen_ thanks, looks like heat should be able to get the necessary info

Earlier   Later