Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-10
18:36:23 efried I can have a crack at an edit.
18:36:40 efried Because when dansmith is happy, everyone's happy.
18:37:02 efried Or maybe it's the inverse.
18:37:14 edleafe efried: wasn't there some discussion about trying to backfill those values, using the consumer_uuid to find the project/user for that VM
18:37:30 efried oh, sheesh, I don't remember any of that.
18:37:35 edleafe but that that wouldn't work for non-VM consumers
18:39:28 edleafe yeah, that was a while ago, well before the consumer generation stuff
18:43:44 eandersson cfriesen_, do you think the numa migration (not live) issues are fixed in newer releases?
18:44:04 eandersson I haven't had time to build a non-mitaka lab yet to run the same tests
19:08:16 cfriesen_ eandersson: I don't regularly run migration/evacuate/resize tests on upstream master, so I can't say for sure. I'm also confused why it's not working for you. Did you ever get a chance to test with debug logging enabled to figure out why the claim logs weren't coming out?
19:08:37 cfriesen_ eandersson: generally I'd expect more recent versions to do a better job of resource tracking
19:08:40 eandersson I think I pasted you anything relavent
19:08:59 eandersson Yea - I'll try to set up a lab with pike or newer this week
19:09:08 eandersson need to do some research for our upgrade path anyway
19:30:05 cfriesen_ dansmith: in https://git.openstack.org/cgit/openstack/nova/commit/?id=804215017f38300ac429e4401ce9c23f0c8ff3c6 you have a comment suggesting that failing to detach a volume is expected and reasonable. Was there a fix to keep heat from waiting forever if this happens when deleting a stack?
19:53:32 openstackgerrit Ed Leafe proposed openstack/nova master: Add columns for generation to consumers https://review.openstack.org/557958
19:53:33 openstackgerrit Ed Leafe proposed openstack/nova master: Add handlers for consumer_generation https://review.openstack.org/557959
19:53:34 openstackgerrit Ed Leafe proposed openstack/nova master: Add a microversion for consumer generation https://review.openstack.org/558958
19:53:47 edleafe efried: ^^ new revs for consumer generation. Still have a couple of functional tests failing in the last patch, but otherwise they're ready for review
19:54:09 efried edleafe: ack
19:54:43 dansmith cfriesen_: that is about the instance not releasing the disk, it still raises
19:55:02 dansmith cfriesen_: but no, if heat is going to try to detach things before it deletes an instance, then it's going to keep waiting
19:55:23 dansmith if it's doing a stack delete, then it should just kill the instance and not wait for volume detach which is never going to happen
19:58:55 cfriesen_ dansmith: apparently heat has no way of knowing that the detach failed from nova's perspective, so currently it waits around forever. And I suspect the volume was a separate heat resource and it was trying to tear it down cleanly. (Looks like a stack update could also result in volume detach, which would be subject to the same problem.)
20:00:03 dansmith cfriesen_: I think we log an instance action for it, no?
20:00:05 dansmith but regardless,
20:00:34 dansmith at a minimum it should time out and then try shutting the instance down before doing the detach (or just delete it while it's off)
20:00:49 dansmith actually just deleting it while it's running should work I would think
20:01:24 cfriesen_ dansmith: delete the instance? what about the stack update case where we just want to detach and replace the volume?
20:01:38 dansmith you said stack delet
20:01:44 dansmith and that's where we hit the issue as well
20:01:56 dansmith but, if you're doing an update, then power off and detach
20:02:20 dansmith the deal is the instance didn't let go of the volume, so there's really nothing else to do other than shut it down, or tell the instance to get over itself
20:02:46 cfriesen_ dansmith: fair enough. you mentioned an instance action, where would that get logged?
20:02:56 dansmith in instance actions
20:04:15 cfriesen_ I mean where in the detach volume exception path would we report the failure to detach as an instance action?
20:04:33 dansmith I'm not positive we do, I'm just saying I think we would
20:04:39 dansmith I can look, but so can you :)
20:05:03 cfriesen_ yep..was hoping you knew off the top of your head. Looking.
20:07:11 dansmith nope
20:08:39 dansmith ah yeah,
20:08:53 dansmith cfriesen_: @wrap_instance_event on compute manager detach_volume
20:08:57 cfriesen_ dansmith: looks like ComputeManager.detach_volume() has wrap_instance_fault and wrap_instance_event decorators
20:09:01 dansmith I think that decorator ends up with it
20:09:01 dansmith yeah
20:09:34 efried dansmith: You wanna finish up the paperwork on the bp here https://blueprints.launchpad.net/nova/+spec/add-consumer-generation
20:09:50 cfriesen_ dansmith: So heat should maybe poll those for updates as well as looking for the state change on the volume?
20:10:29 dansmith efried: I hate paperwork
20:10:44 dansmith cfriesen_: I'm guessin' that'd be smart :)
20:11:21 efried dansmith: :*
20:12:07 cfriesen_ time to go check the captured database logs and see if the appropriate info was there.
20:18:11 openstackgerrit Merged openstack/nova stable/queens: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/559700
20:18:51 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/559702
20:24:26 openstackgerrit Surya Seetharaman proposed openstack/nova master: Cleanup RP and HM records while deleting a compute service. https://review.openstack.org/554920
20:29:41 openstackgerrit Merged openstack/nova master: Pick next minimum libvirt / QEMU versions for "Stein" https://review.openstack.org/558171
20:30:13 openstackgerrit Merged openstack/nova master: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/505771
20:31:56 cfriesen_ dansmith: I see relevent entries in "instance_actions_events" and "instance_faults", but I don't know of any API to get either of those. Would the fault entry show up when doing a GET
20:31:57 cfriesen_ GET
20:31:59 cfriesen_ /servers/{server_id}
20:32:06 cfriesen_ whoops, paste error
20:32:07 dansmith cfriesen_: the instance actions api
20:32:15 mriedem GET /os-instance-actions
20:32:23 mriedem the fault is stored on the server
20:32:35 mriedem https://developer.openstack.org/api-ref/compute/#servers-actions-servers-os-instance-actions
20:32:40 dansmith cfriesen_: https://developer.openstack.org/api-ref/compute/#list-actions-for-server
20:32:43 dansmith yeah
20:32:44 mriedem https://developer.openstack.org/api-guide/compute/faults.html
20:32:56 mriedem ^ might be stale
20:33:02 mriedem most things in the compute api guide are old
20:33:45 mriedem e.g. https://developer.openstack.org/api-guide/compute/faults.html#instance-faults
20:33:50 mriedem "However, there is currently no API to retrieve this information."
20:33:52 mriedem that's totally bogus
20:34:19 mriedem GET /server/{id} has a 'fault' key in the response
20:34:20 mriedem "A fault object. Only displayed in the failed response. Default keys are code, created, and message (response code, created time, and message respectively). In addition, the key details (stack trace) is available if you have the administrator privilege."
20:38:21 cfriesen_ thanks, looks like heat should be able to get the necessary info
20:38:59 openstackgerrit Merged openstack/nova master: Don't persist RequestSpec.retry https://review.openstack.org/559447
20:39:30 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/560142
20:39:31 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Don't persist RequestSpec.retry https://review.openstack.org/560143
20:44:32 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/560145
20:44:33 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Don't persist RequestSpec.retry https://review.openstack.org/560146
21:10:10 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
21:10:11 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:15:15 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/560145
21:15:16 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Don't persist RequestSpec.retry https://review.openstack.org/560146
21:16:16 jackie-truong I have a patch that adds documentation for the certificate validation feature. Should that be linked with the current nova-validate-certificates patches? I'll still include the bp link in the commit message
21:17:02 cfriesen_ mriedem: dansmith: have you ever seen anything like this? http://paste.openstack.org/show/718865/ It seems like we ask nova to detach, but libvirt doesn't recognize the device.
21:17:39 mriedem yes
21:17:46 mriedem cfriesen_: you should be talking to lyarwood and melwitt
21:17:51 cfriesen_ cool, will do
21:17:54 mriedem they've been in the bowels of that code a few times
21:17:57 openstackgerrit Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158
21:18:30 mriedem cfriesen_: which branch are you on? do you have all the latest fixes?
21:19:05 cfriesen_ this would be Pike, I'd have to check if it was totally up to date
21:19:59 cfriesen_ gotta run, parent-teacher interview
21:20:52 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
21:20:53 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
21:20:54 openstackgerrit Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158
21:22:57 mriedem cfriesen_away: i was thinking of https://review.openstack.org/#/c/488545/ but that's in pike GA
21:41:25 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/560162
21:53:33 openstackgerrit Eric Berglund proposed openstack/nova master: PowerVM Driver: Localdisk https://review.openstack.org/549300

Earlier   Later