Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-04
15:48:01 mriedem since we can't reschedule, i guess i won't ask about that
15:48:20 mriedem so when we shelve offload an instance, do we deallocate for that consumer on it's resource providers
15:48:41 mriedem or when we delete the instance, and the compute host is down so we can't rpc cast to the compute to do the delete, do we deallocate from the api
15:48:44 mriedem i think the answer to both is no
15:49:16 jaypipes yeah, I suspect you are correct about that. but would be easy enough to fix in a bug report.
15:50:08 mriedem i may be wrong about the shelve offload case, because i remember alaski pointing out a super gross tight coupling of instance vm_state checking in the RT
15:50:21 mriedem yup ALLOW_RESOURCE_REMOVAL = [DELETED, SHELVED_OFFLOADED]
15:50:30 mriedem RT._update_usage_from_instance
15:50:32 mriedem checks that
15:50:57 jaypipes ok
15:51:18 mriedem which passes sign=-1 to update_instance_allocation to call _delete_allocation_for_instance
15:51:19 mriedem cool
15:51:29 mriedem we still fail the local delete case probably
15:51:39 mriedem which is maybe related to dansmith's bug about evacuate
16:01:04 dansmith sorry I was on a call, and have only skimmed the scrollback
16:01:06 openstack Launchpad bug 1661312 in OpenStack Compute (nova) "Evacuation will corrupt instance allocations" [Undecided,New]
16:01:06 mriedem dansmith: in https://bugs.launchpad.net/nova/+bug/1661312 where does step 4 happen?
16:01:19 cdent jaypipes, bauzas, mriedem I left a longer comment on the spec to remind us of some of these things
16:01:19 openstack Launchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Undecided,New]
16:01:19 mriedem dansmith: jaypipes: cdent: created https://bugs.launchpad.net/nova/+bug/1679750
16:01:21 dansmith jaypipes: if we end up off the compute node due to shelve or evacuate, won't we heal our way to deleting the allocations after a bit?
16:01:39 dansmith jaypipes: also for local delete, I would think
16:01:51 cdent thanks mriedem
16:01:58 jaypipes dansmith: need to look into it. we *may* but wasn't 100% sure.
16:02:11 dansmith jaypipes: I would think we do
16:02:21 dansmith jaypipes: it might take a bit, but.. should happen I think
16:02:30 cdent the periodic job in the RT is always trying to heal if it sees a gap
16:02:39 dansmith cdent: right
16:02:41 cdent so the problem comes up when there is no RT
16:02:47 cdent s/the/a/
16:04:58 dansmith you mean in a scheduler-claims world?
16:05:41 cdent dansmith: no, in the world where we want the representation of available resources to be correct
16:06:27 cdent it doesn't always have practical impact if a compute nodes is down but in the placement db but it does mean that our global total of eg vcpus is wrong
16:07:01 cdent but since they self register
16:07:01 cdent it would be nice that when a compute node went down , it was removed from the placement db
16:07:18 cdent We don't currently have anything in place to clean up when they die, do we?
16:07:27 dansmith I dunno about that.. for instances that aren't evacuated, those are still valid allocations, yes?
16:08:01 dansmith if you think about when we do quota by counting things in placement,
16:08:06 cdent I was thinking more in the sense of full on node crash
16:08:13 dansmith one compute node going offline shouldn't free up quota for you and then get you in the red once it comes back
16:08:44 dansmith even if the compute node is off and cold, if there are unevacuated (or undeleted) instances there, the allocations should stay, IMHO
16:08:59 mriedem dansmith: i agree with that
16:09:00 cdent the quota case nicely complicates things
16:09:01 dansmith i.e. I wouldn't think placement should know about liveness
16:09:05 openstack Launchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Undecided,New]
16:09:05 mriedem https://bugs.launchpad.net/nova/+bug/1679750 is about when you delete an instance on a dead node
16:09:17 dansmith if nova has an instance, then placement has a record of allocations.. nothing more than that
16:09:34 cdent because if we are using allocations for usage, is a cold node supposed to count against my quota? that's not my (the user's) fault and I want to run some instances
16:09:45 cdent s/usage/quota measurment/
16:09:48 dansmith cdent: it should IMHO yeah
16:10:01 dansmith cdent: you're free to delete the instance and get the quota back
16:10:05 dansmith the flip side is,
16:10:13 dansmith if a compute node goes down, you get more quota and boot a thing,
16:10:31 dansmith when the compute node comes back, should the provider be free to nuke the instance that was running on that node since you'd now be over quota?
16:10:33 cdent hmmm. yeah. weird.
16:10:41 cdent blech
16:10:51 mriedem i think when i get some time i'll write a functional test for the local delete /allocations bug
16:11:44 mriedem back to spec review
16:19:51 openstackgerrit Artom Lifshitz proposed openstack/nova-specs master: SPICE native client support https://review.openstack.org/442040
16:22:55 openstackgerrit Balazs Gibizer proposed openstack/nova-specs master: Show persisted scheduler hints in server details https://review.openstack.org/440580
16:25:29 gibi mriedem, sdague, cdent, jaypipes, johnthetubaguy: hi! I updated the spec https://review.openstack.org/#/c/440580
16:26:08 gibi mriedem, sdague, cdent, jaypipes, johnthetubaguy: I tried to answer all the concerns comments
16:26:20 jaypipes gibi: thx Balasz!
16:26:41 cdent gibi++
16:28:24 mriedem mgagne: thanks for this https://review.openstack.org/#/c/337819/
16:28:36 mriedem mgagne: not many people take the time to update spec docs after the fact
16:29:09 mgagne mriedem: hmm, didn't get notification for that one, checking
16:29:11 johnthetubaguy cdent: you are right about limiting the crust at ingress, I think that was the question for me, should we do that first.
16:30:06 johnthetubaguy mriedem: ++ on the thank you and your comments there, it came up the other day in a bug I think
16:31:04 mgagne cool, glad to see it got merged =)
17:03:19 openstackgerrit Lee Yarwood proposed openstack/nova master: encryptors: Always default to the NoOpEncrytor https://review.openstack.org/453265
17:26:36 mriedem melwitt: on https://review.openstack.org/#/c/453041/ i know it's just a re-proposal, but most of what that spec is talking about is the old justification for doing it, i.e. rpc versioning and the like. the reason you're re-proposing it is a bit different, for cells v2. i'd like to see something in there about why this is something we want for cells v2 - what cells v2 problem does it solve, etc?
17:27:01 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: Libvirt: Native LUKS file and host device decryption by QEMU https://review.openstack.org/437070
17:27:17 melwitt mriedem: okay. so add that and remove Previously-approved?
17:27:53 openstackgerrit Mathieu Gagné proposed openstack/nova-specs master: Nova support for attached volume extend https://review.openstack.org/453272
17:28:06 mriedem melwitt: previously-approved is fine
17:28:15 melwitt k
17:28:15 mriedem i'm just looking for something about cells v2 in the problem description
17:28:24 openstackgerrit Mathieu Gagné proposed openstack/nova-specs master: Nova support for attached volume extend https://review.openstack.org/453272
17:28:33 mriedem maybe that's L53... "Because the new tokens will go in the database we need to consider
17:28:34 mriedem cells v2."
17:29:19 mriedem melwitt: so maybe that paragraph answers it
17:29:25 mriedem the url contains the instance uuid
17:29:36 mriedem weren't we going to use that to lookup which cell the token was in?
17:32:11 melwitt yeah, I'm trying to see if that change happened already yet (looking at the old merged vs abandoned patches)
17:33:25 melwitt okay, so that change didn't make it https://review.openstack.org/#/c/325382/
17:37:05 mriedem do we have to worry about old connection urls that don't have the instance uuid so we can't do the lookup for the cell?
17:39:57 openstackgerrit Gábor Antal proposed openstack/nova master: Transform instance.volume_attach notification https://review.openstack.org/401992
17:44:48 openstackgerrit Gábor Antal proposed openstack/nova master: Transform instance.volume_attach notification https://review.openstack.org/401992
17:44:48 melwitt mriedem: so it looks like the idea during the transition period is to have both the consoleauth service running and have new tokens stored in cell databases. so old connection urls will be served by the consoleauth service, which has the tokens globally in memcached
17:47:51 openstackgerrit Gábor Antal proposed openstack/nova master: Transform instance.volume_attach notification https://review.openstack.org/401992
17:49:56 melwitt mriedem: and then I think through normal token expiration, eventually all of the old memcached tokens won't be used anymore, and when that happens it would be safe to stop running the consoleauth service since no outstanding connection urls will be without the server uuid
17:52:26 macsz bugs team meeting (#openstack-meeting-4) in 10 minutes, if anyone is interested
18:00:40 openstackgerrit Dan Peschman proposed openstack/nova master: Add uuid to Service model https://review.openstack.org/453025
18:05:10 mriedem melwitt: ok is that all in the spec? i didn't read the whole thing b/c i'm lazy
18:05:47 mriedem ah reading your reply
18:06:08 oomichi gmann: thanks, LGTM now
18:07:44 melwitt mriedem: it is, but it's not as detailed as my comment. I sort of derived it from looking at the abandoned patches plus the spec, since I'm not so familiar with consoles. the spec language appears to assume more knowledge
18:07:59 mriedem yeah same here
18:09:01 mriedem you need a phd like paul murray

Earlier   Later