Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-26
16:06:32 dansmith melwitt: that kinda defeats the purpose,in that it becomes a thing that can get out of sync and that we can't refresh without being able to contact the other cell
16:06:45 dansmith melwitt: okay well we could count instances in the api db to mitigate that
16:07:06 johnthetubaguy melwitt: maybe we hard fail on that
16:07:29 dansmith melwitt: so maybe we should get a read from operator(s) about how bad it would be be to not be if ram/cores leaks while a whole cell is offline
16:07:37 johnthetubaguy melwitt: if we know something is there
16:07:44 dansmith johnthetubaguy: hard fail to create instances anywhere if one cell is offline? that'd be bad :)
16:07:52 pkoniszewski_ mdbooth: AFAIR nova does not assume that there will only be live migration from N-1 to N (N-2 to N must also be possible and safe)
16:07:58 melwitt dansmith: yeah. I haven't thought about it much but it seemed like it makes sense for it to be stale if the cell is gone. but if it went away permanently, then yeah that would be the refresh/sync case. so nevermind that idea
16:08:29 mdbooth pkoniszewski_: I understood the contract was that N and N-2 never exist in the same deployment, right?
16:08:34 melwitt dansmith, johnthetubaguy: yeah, we could fall back to InstanceMappings for the instance count but cores/ram would be unknown
16:08:37 dansmith melwitt: well, no I meant even if it's offline temporarily, you wouldn't be able to refresh that value from N-1 cells because you don't know how much of it is in the offline cell, unless you keep more data
16:08:41 mdbooth So no migration between them is implicit.
16:08:45 mdbooth dansmith: ^^^ ?
16:08:55 dansmith mdbooth: correct
16:09:20 bauzas mriedem: dansmith: cdent: rev'd https://review.openstack.org/417961
16:09:38 melwitt dansmith: oh, right
16:09:40 dansmith mdbooth: pkoniszewski you shouldn't break your RPC api between N-2 and N, but we also don't support that kind of deployment, so higher level assumptions might not be possible between those two
16:09:48 cdent bauzas: cool, will look asap, currently double booked in two meetings ...
16:10:14 mriedem bauzas: ok. i think we should move the compute/service version checking stuff into a separate patch for sanity
16:10:21 mriedem bauzas: but sounds like you're going to be afk for awhile
16:10:22 dansmith bauzas: that service version and fail should be a separate patch
16:10:25 dansmith what mriedem said
16:10:50 bauzas mriedem: yup, I could potentially lightweight the change
16:10:56 pkoniszewski_ dansmith: got it
16:11:20 melwitt dansmith: actually I guess the long term answer was we're going to ask for cores/ram count from placement when that's doable
16:11:30 bauzas mriedem: dansmith: gimme just 20 mins for my daughter's homework and I'm back
16:11:33 dansmith melwitt: ah, that's true
16:11:42 dansmith melwitt: I forgot about that part
16:12:13 mdbooth pkoniszewski_: Either way, it can be deleted as soon as we can assume the other end is doing the update. Worth adding a comment that we added the remote end in N.
16:12:34 pkoniszewski_ mdbooth: yes, i'll add comment there
16:14:14 mfeoktistov sfinucan: i never thought that "ploop" sounds bad, until i heard the pronunciation of other developers not from virtuozzo. Yes, i agree it sounds strange, to say the least .
16:14:51 mdbooth pkoniszewski_: This code is hairy as hell, btw. Thanks for taking it on.
16:15:38 mfeoktistov sfinucan: we pronounce like pe + loop. "Parallels loop" . Parallels is previous name of our company
16:16:12 mdbooth mfeoktistov: diana_clarke particularly enjoys it, btw. IIRC it's her favourite disk format.
16:16:25 sfinucan mfeoktistov: Aha - that nuance escaped me :)
16:17:39 mdbooth sfinucan: Had you previous used gee-you-eye?
16:17:54 diana_clarke Ploop! There it is! https://www.youtube.com/watch?v=Z-FPimCmbX8
16:18:06 mdbooth diana_clarke: lol
16:19:14 sfinucan mdbooth: In my blessed ignorance, yes
16:19:43 alaski dansmith: melwitt just caught part of scrollback from last night. I wanted BuildRequest.instance to be orphaned because I didn't want it treated like an instance, it's a holding place for what an instance should look like when it's created but isn't an instance itself. It's a subtle distinction made complicated by using an actual instance object there. I thought thinking of it that way would lead to
16:19:44 alaski fewer bugs.
16:20:02 dansmith alaski: right, I remember
16:20:07 sfinucan We got odd looks for wanting to play Call of Duty (COD) in past years too
16:20:13 dansmith alaski: I even went looking for where we had that convo, but it must have been in IRC
16:20:26 alaski dansmith: cool. I also said I would defer to those who were doing the work :)
16:20:29 mfeoktistov diana_clarke: ahaha! good track for presentation
16:20:36 dansmith alaski: s'all good now
16:22:16 cfriesen mdbooth: thanks for taking a look at the live-migration claims patch. It'd be good to finally get that stuff in.
16:22:18 melwitt alaski: yeah, once we realized what was happening, it was clear what to do
16:24:58 alaski dansmith: melwitt excellent
16:25:14 melwitt and I could see allowing it to masquerade as a normal Instance object could lead to a lot of subtle problems, if anything were missing from its fields
16:25:38 dansmith on the other hand, in reality,
16:25:48 dansmith we used that to smooth over where we got the instance,
16:26:02 dansmith so down the stack we've lost context (heh) about why it might not be a real one
16:26:54 melwitt yeah. I suspect if it had a real context, the problem would travel further down the stack and be really puzzling
16:27:16 melwitt *even further
16:27:24 dansmith I dunno, the refresh should work if we have uuid and context
16:27:24 johnthetubaguy melwitt: sorry, got distracted, which bits where you counting when you needed the cells again?
16:28:22 dansmith johnthetubaguy: the counting patches counts instances, ram and cores from the cell db
16:28:36 melwitt johnthetubaguy: the ram and cores, and I tally the instances while counting ram/cores. ram and cores are being counted currently by pulling the instances and going through their flavor extra specs
16:28:44 dansmith johnthetubaguy: if we lose connectivity with a cell for ten minutes, a user suddenly gets their quota back until we reconnect and could burn it in other cells
16:29:00 johnthetubaguy yeah, that makes sense
16:29:33 melwitt johnthetubaguy: the real plan is to call the placement api to get cores/ram count and use InstanceMappings for instances, but we can't get that from placement quite yet
16:30:01 johnthetubaguy melwitt: ah, true, well that to make placement do quota too ;)
16:30:09 johnthetubaguy s/to/or
16:30:48 johnthetubaguy melwitt: FWIW, its tempting to fail hard if a cell is down, and we know there is something in there we should be counting
16:30:57 dansmith johnthetubaguy: how does that work?
16:31:11 dansmith johnthetubaguy: if one cell is down, we can't do anything? that kinda defeats the purpose no?
16:31:21 dansmith well, s/do/create/ but ..
16:31:36 melwitt yeah, that seems like it would draw much ire
16:31:40 dansmith yeah
16:31:52 dansmith johnthetubaguy: so, we can count instances in the api database,
16:32:04 dansmith johnthetubaguy: which would leave just cores and ram as leaky if a whole cell went down
16:32:22 johnthetubaguy yeah, thats fair, if we did it that way, it feels less of a big deal
16:32:22 dansmith johnthetubaguy: which should be rare, and maybe some quota leakage isn't the worst thing in that case, given we will someday have placement do better
16:32:23 dansmith but I dunno
16:32:40 dansmith I feel like maybe that needs to be a question to the ops, but we have no time for that
16:32:46 johnthetubaguy yeah
16:33:28 johnthetubaguy honestly, feels like fail hard and don't care will be both be requested, depending on how much RAM and CPU quota matters to you
16:33:51 johnthetubaguy I like that the instance count one works just fine though
16:35:53 melwitt usually maybe the cores/ram leak stays bounded by the instances quota, but if someone has set unlimited instances and limited cores/ram, that would be a lot bigger problem
16:36:50 dansmith yeah
16:36:57 dansmith that's why asking the operators would be good
16:37:31 sfinucan mriedem: If /you're/ looking for something to do, could you take a look at https://review.openstack.org/#/c/405954 at some point?
16:37:48 mriedem currently reviewing https://review.openstack.org/#/c/404472
16:37:58 mriedem but i already had https://review.openstack.org/#/c/405954 in a tab
16:38:14 dansmith sfinucan: you know mriedem is on vacation right? I know, it's hard to tell
16:38:27 sfinucan a tab will do nicely
16:38:29 jaypipes_ edleafe: are you handling the change to https://review.openstack.org/#/c/404472/17/nova/compute/resource_tracker.py that you said you were looking at yesterday?
16:38:33 jaypipes_ edleafe: if not, I will do so.
16:38:41 jaypipes_ and good morning fellas.
16:38:46 sfinucan dansmith: I do. I've already given out to him for being around but he won't listen
16:38:56 dansmith sfinucan: he's not very good at vacation
16:39:11 sfinucan You Americans and your pesky work ethic
16:39:50 sfinucan We clearly need a Greek PTL
16:39:55 jaypipes_ lol
16:40:11 jaypipes_ my idea of vacation is doing code reviews by a pool.
16:40:46 dansmith mriedem: bauzas: that change is failing because it can't find placement.password
16:41:05 bauzas dansmith: yup, I'm working on a new PS
16:41:05 kashyap sfinucan: Quit giving those non-apologies

Earlier   Later