Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-14
13:41:38 efried This soft delete thing: can a guy "undelete" a soft-deleted instance?
13:43:18 mriedem efried: yes
13:43:19 mriedem restore
13:43:34 mriedem efried: https://developer.openstack.org/api-ref/compute/#restore-soft-deleted-instance-restore-action
13:43:43 efried mriedem (melwitt) And by current design, does the quota get re-hit at that time?
13:43:57 mriedem efried: no
13:44:05 mriedem quota is decremented when we actually delete
13:44:16 mriedem it's one of the reasons the local delete code in the api is terrible
13:44:34 mriedem because we have to create a quota reservation to decrement in the api and then either roll that back or commit it in the compute
13:44:37 efried Ohh, but according to comments on https://review.openstack.org/#/c/386756/ it looks like we *also* decrement the quota when we do the soft delete.
13:44:39 mriedem if we lose the compute, well, oops!
13:46:17 efried melwitt dansmith Am I misunderstanding ^^ ?
13:46:44 mriedem dan is out today
13:46:55 mriedem it's 6:46am for mel
13:46:59 mriedem and she's not a morning person i gather
13:47:02 efried :)
13:49:09 mriedem i left a comment in the patch
13:51:32 mriedem that change needs to be rebased anyway
13:51:36 mriedem the compute api code in there is wayyyyy old
13:51:41 openstackgerrit Matt Riedemann proposed openstack/nova master: Change soft delete resources to prevent DOS https://review.openstack.org/386756
13:53:21 efried mriedem Cool, thanks.
13:53:40 efried I keep hoping that reviewing stuff I don't understand will eventually lead to some kind of enlightenment.
13:54:44 efried So far, it's like that flashlight in a horror movie. Dim... dimmer... gone.
13:55:45 mriedem well, that's true if you're reviewing anything related to local delete in the api or quotas
13:56:03 mriedem efried: you could triage some new bugs!?
13:56:06 efried mriedem BTW, I'm picking things to review based on the "query_high_prio_bug_fixes.py" result in the Pike Review Focus etherpad.
13:56:18 efried mriedem Sure, was gonna ask earlier if I could help with that.
13:56:27 efried But wasn't sure I'd be qualified - see flashlight analogy above.
13:56:30 mriedem like i came across this today https://bugs.launchpad.net/nova/+bug/1682060
13:56:30 openstack Launchpad bug 1682060 in kolla-ansible "empty nova service and hypervisor list" [Undecided,New]
13:56:42 mriedem ^ is a thing that just landed which broke how kolla does cell setup
13:56:53 mriedem yay
13:56:57 efried mriedem Okay, looking. (And will have to find out wtf kolla is...)
13:57:14 mriedem kolla is openstack on top of k8s
13:57:20 mriedem deploy openstack with k8s
13:57:34 efried And k8s is...?
13:57:38 mriedem kubernetes
13:58:15 mriedem dude, i'm still reeling from the shelve/unshelve stuff i was digging into for a few hours yesterday
13:58:26 mriedem i find new and terrible things every day it seems
13:58:54 mriedem well, "find" as in new to me, and then i find out someone else found the same thing out over a year ago
13:58:57 mriedem and it's not fixed yet
14:01:12 mriedem cdent: back to your point about dropping code,
14:01:15 mriedem i think we can do that after a release
14:01:28 mriedem cdent: so you request the new microversion and if it's there, great, else fallback,
14:01:47 mriedem cdent: and then in queens, we require the higher microversion - we've been signalling that in nova-status
14:01:57 cdent mriedem: okay, no problem, that's what I'm doing now: trying the new way. kablooie? okay do the old way
14:02:08 mriedem e.g. filter scheduler requires placement>=1.4 in pike, and nova-status checks that too
14:02:31 mriedem yeah https://github.com/openstack/nova/blob/ae637e70a72166067e09f3bbc3acb54e5d92771e/nova/cmd/status.py#L208
14:02:33 mriedem cool
14:02:53 mriedem huh, never realized we were using pkg_resources there
14:03:44 cdent mriedem: that's a relatively recent bugfix
14:03:48 cdent it was doing floats initially
14:03:53 cdent which would have broken relatively soon
14:04:31 mriedem i remember seeing something about that
14:06:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Clarify wording in compute API _delete method https://review.openstack.org/456910
14:13:23 sdague mriedem: you know, shelve/unshelve being terrible might be a feature and not a bug, to convince no one to ever use it
14:15:07 mriedem sdague: well, it's also fodder for why nova/openstack is no good
14:15:17 mriedem i'd rather drop things that don't work
14:16:01 sdague mriedem: I agree on the drop front
14:16:33 mriedem i think fixing the volume terminate_connection thing is simple, but that only fixes some backends, because of how cinder works
14:16:42 mriedem which is unfortunate, and what stalled fixing it over a year ago
14:17:05 mriedem because what really needs to be called is this internal remove_export function which only os-detach calls, but that's exactly what we don't want to use for shelve becaues we want the volume to stay attached
14:17:17 sdague mriedem: it would be nice if there was a way we could actually get some real data on whether anyone actually uses this bit
14:17:44 smcginnis +1
14:17:47 mriedem so cinder exposes a new api, or changes terminate_connection to also call remove_export, of nova calls detach but then immediately also calls reserve_volume so the instance doesn't lose the volume, all of which aren't great
14:18:00 mriedem option 2 there is probably my favorite
14:18:04 mriedem since it abstracts everything within cinder
14:18:14 mriedem sdague: yeah but i don't really know how to get that
14:18:31 sdague mriedem: yeh, me either
14:18:32 mriedem polling the operators, i could ask if they disable it via policy
14:18:48 mriedem can't really ask users though
14:19:20 mriedem i also don't know if operators track incoming request metrics and could figure out how many calls are made to that api in a week/month
14:19:22 sdague mriedem: so... an idea I was thinking about the other day was to actually build something into nova that builds API stats in a format that's easy to annonymously dump back out
14:19:22 cdent turn if off and wait for someone to complain :D
14:19:40 sdague cdent: if the feedback cycle wasn't 1.5 years, that would work :)
14:20:07 mriedem we send an instance.shelve notification, which i'd think telemetry is consuming, but i'm not sure,
14:20:09 cdent we've all become quite patient for other reasons...this one more thing, no problem
14:20:19 mriedem but that also assumes people are using telemetry, which is a bad assumption
14:20:28 cdent yes it is a bad assumption
14:20:39 cdent it's also a bad assumption that even if they were, that they were recording that particular event
14:20:48 cdent s/recording/persisting/
14:20:51 openstackgerrit Alex Xu proposed openstack/nova master: Use plain routes list for flavors-access endpoint instead of stevedore https://review.openstack.org/456589
14:20:51 openstackgerrit Alex Xu proposed openstack/nova master: Use plain routes list for flavor endpoint instead of stevedore[1] https://review.openstack.org/456549
14:20:52 openstackgerrit Alex Xu proposed openstack/nova master: Use plain routes list for '/os-aggregates' endpoint instead of stevedore https://review.openstack.org/456825
14:20:52 openstackgerrit Alex Xu proposed openstack/nova master: Use plain routes list for flavors-extraspecs endpoint instead of stevedore https://review.openstack.org/456585
14:20:53 openstackgerrit Alex Xu proposed openstack/nova master: Use plain routes list for '/os-keypairs' endpoint instead of stevedore https://review.openstack.org/456478
14:21:21 sdague mriedem: so, just based on past interactions with folks, I honestly just wonder if marking shelve/unshelve as experimental in the API docs, say there are a number of issues with it, and move on might be best
14:21:31 sdague because it really really seems far on the edge
14:21:52 sdague and it gets a disproportunate amount of debugging time relative to use
14:23:08 mriedem i was hoping we could just pull from INFO logs in n-api for when a POST request to shelve comes in, but because it's an 'action' API and the bit we care about is in the body, we don't have that info...
14:24:09 mriedem i'm fine with noting limitations in the api, we do that for some other things as well, or should: https://review.openstack.org/#/c/452970/
14:24:19 mriedem https://review.openstack.org/#/c/452970/2/api-ref/source/servers-admin-action.inc
14:24:31 sdague mriedem: yep, that's one of those issues. I do think it's elsewhere though. Anyway, need to afk for a bit.
14:26:00 feeblepipes mriedem: happen to know Jan Gutter's IRC nick?
14:26:07 mriedem jangutter
14:26:14 feeblepipes mriedem: k, thx.
14:26:30 mriedem sdague: on "an idea I was thinking about the other day was to actually build something into nova that builds API stats in a format that's easy to annonymously dump back out" - thinking about that, i think we already have that, it's instance actions
14:27:15 feeblepipes mriedem: pls see my -1 and comments from that netronome spec. I'm cool with it, but have concerns about the PCI device allocation code proposed for neutronv2/api.py module.
14:27:15 mriedem so compute_shelve_instance is an action event
14:31:03 mriedem feeblepipes: they just mean this i think https://review.openstack.org/#/c/456481/1/nova/network/neutronv2/api.py

Earlier   Later