Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-26
15:57:56 mdbooth pkoniszewski_: I described the race. Can you point out the step I got wrong?
15:58:13 dansmith mriedem: I meant the last statement was a hedge, not a "we've already got a second mortgage, why not go for a third"
15:58:37 dansmith mriedem: let me look through it and see how I feel after that
15:58:43 mdbooth pkoniszewski_: I think cfriesen(?) said they'd hit it in practise, too.
15:58:43 mriedem hey, the dow was over 20K so i think that's a safe bet right now
15:58:46 mriedem what could possibly go wrong
15:58:48 dansmith heh
15:58:58 mfeoktistov mriedem: thanks for discussion in mailing list
15:59:13 pkoniszewski_ mdbooth: live migration will always be set to 'completed', there is no chance we will end up with 'failed' or 'finished' state
15:59:26 dansmith johnthetubaguy: are you still around?
15:59:28 pkoniszewski_ mdbooth: the RPC has been changed to call
15:59:31 pkoniszewski_ mdbooth: https://review.openstack.org/#/c/389687/
15:59:35 johnthetubaguy dansmith: I am
15:59:48 bauzas there it is
16:00:02 dansmith johnthetubaguy: how are you feeling about the counting quotas stuff?
16:00:06 openstackgerrit Sylvain Bauza proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961
16:00:40 mdbooth pkoniszewski_: So it has, I must have been looking at an old version when looking at the old commit.
16:00:54 mdbooth Well... for the reasons stated I also think that's a bad idea :)
16:01:02 mdbooth But yeah, that means there's no race.
16:01:15 pkoniszewski_ mdbooth: but there is still an issue with rollback, thats a very good point
16:02:30 melwitt mriedem, dansmith, johnthetubaguy: one thing I realized about the counting quotas approach is that once we go multicell, if we lose contact with a cell db for whatever reason, we lose the ability to count resources in it, which would basically free up a person's resource usage for the time period the cell db is inaccessible
16:02:35 sfinucan mfeoktistov: +W'd
16:02:43 sfinucan also, worst IRC nick ever :)
16:02:54 sfinucan ...and mine is bad
16:03:47 dansmith melwitt: yeah, that would be bad..
16:05:05 dansmith melwitt: that would be for what kind of resources?
16:05:18 dansmith melwitt: we can count instances and server groups and keypairs all in the api db
16:05:30 mdbooth pkoniszewski_: Incidentally, you could add a comment to the 'completed' code that it can be safely deleted in Queens, because we can guarantee the destination will have already updated migration status.
16:05:41 melwitt dansmith: it seems like we'd need to have some sort of caching table for the usage that gets updated anytime we actually count
16:06:10 melwitt dansmith: currently, I think instances, cores, ram. but that's because I rolled the instance tally into the cores and ram count
16:06:32 dansmith melwitt: that kinda defeats the purpose,in that it becomes a thing that can get out of sync and that we can't refresh without being able to contact the other cell
16:06:45 dansmith melwitt: okay well we could count instances in the api db to mitigate that
16:07:06 johnthetubaguy melwitt: maybe we hard fail on that
16:07:29 dansmith melwitt: so maybe we should get a read from operator(s) about how bad it would be be to not be if ram/cores leaks while a whole cell is offline
16:07:37 johnthetubaguy melwitt: if we know something is there
16:07:44 dansmith johnthetubaguy: hard fail to create instances anywhere if one cell is offline? that'd be bad :)
16:07:52 pkoniszewski_ mdbooth: AFAIR nova does not assume that there will only be live migration from N-1 to N (N-2 to N must also be possible and safe)
16:07:58 melwitt dansmith: yeah. I haven't thought about it much but it seemed like it makes sense for it to be stale if the cell is gone. but if it went away permanently, then yeah that would be the refresh/sync case. so nevermind that idea
16:08:29 mdbooth pkoniszewski_: I understood the contract was that N and N-2 never exist in the same deployment, right?
16:08:34 melwitt dansmith, johnthetubaguy: yeah, we could fall back to InstanceMappings for the instance count but cores/ram would be unknown
16:08:37 dansmith melwitt: well, no I meant even if it's offline temporarily, you wouldn't be able to refresh that value from N-1 cells because you don't know how much of it is in the offline cell, unless you keep more data
16:08:41 mdbooth So no migration between them is implicit.
16:08:45 mdbooth dansmith: ^^^ ?
16:08:55 dansmith mdbooth: correct
16:09:20 bauzas mriedem: dansmith: cdent: rev'd https://review.openstack.org/417961
16:09:38 melwitt dansmith: oh, right
16:09:40 dansmith mdbooth: pkoniszewski you shouldn't break your RPC api between N-2 and N, but we also don't support that kind of deployment, so higher level assumptions might not be possible between those two
16:09:48 cdent bauzas: cool, will look asap, currently double booked in two meetings ...
16:10:14 mriedem bauzas: ok. i think we should move the compute/service version checking stuff into a separate patch for sanity
16:10:21 mriedem bauzas: but sounds like you're going to be afk for awhile
16:10:22 dansmith bauzas: that service version and fail should be a separate patch
16:10:25 dansmith what mriedem said
16:10:50 bauzas mriedem: yup, I could potentially lightweight the change
16:10:56 pkoniszewski_ dansmith: got it
16:11:20 melwitt dansmith: actually I guess the long term answer was we're going to ask for cores/ram count from placement when that's doable
16:11:30 bauzas mriedem: dansmith: gimme just 20 mins for my daughter's homework and I'm back
16:11:33 dansmith melwitt: ah, that's true
16:11:42 dansmith melwitt: I forgot about that part
16:12:13 mdbooth pkoniszewski_: Either way, it can be deleted as soon as we can assume the other end is doing the update. Worth adding a comment that we added the remote end in N.
16:12:34 pkoniszewski_ mdbooth: yes, i'll add comment there
16:14:14 mfeoktistov sfinucan: i never thought that "ploop" sounds bad, until i heard the pronunciation of other developers not from virtuozzo. Yes, i agree it sounds strange, to say the least .
16:14:51 mdbooth pkoniszewski_: This code is hairy as hell, btw. Thanks for taking it on.
16:15:38 mfeoktistov sfinucan: we pronounce like pe + loop. "Parallels loop" . Parallels is previous name of our company
16:16:12 mdbooth mfeoktistov: diana_clarke particularly enjoys it, btw. IIRC it's her favourite disk format.
16:16:25 sfinucan mfeoktistov: Aha - that nuance escaped me :)
16:17:39 mdbooth sfinucan: Had you previous used gee-you-eye?
16:17:54 diana_clarke Ploop! There it is! https://www.youtube.com/watch?v=Z-FPimCmbX8
16:18:06 mdbooth diana_clarke: lol
16:19:14 sfinucan mdbooth: In my blessed ignorance, yes
16:19:43 alaski dansmith: melwitt just caught part of scrollback from last night. I wanted BuildRequest.instance to be orphaned because I didn't want it treated like an instance, it's a holding place for what an instance should look like when it's created but isn't an instance itself. It's a subtle distinction made complicated by using an actual instance object there. I thought thinking of it that way would lead to
16:19:44 alaski fewer bugs.
16:20:02 dansmith alaski: right, I remember
16:20:07 sfinucan We got odd looks for wanting to play Call of Duty (COD) in past years too
16:20:13 dansmith alaski: I even went looking for where we had that convo, but it must have been in IRC
16:20:26 alaski dansmith: cool. I also said I would defer to those who were doing the work :)
16:20:29 mfeoktistov diana_clarke: ahaha! good track for presentation
16:20:36 dansmith alaski: s'all good now
16:22:16 cfriesen mdbooth: thanks for taking a look at the live-migration claims patch. It'd be good to finally get that stuff in.
16:22:18 melwitt alaski: yeah, once we realized what was happening, it was clear what to do
16:24:58 alaski dansmith: melwitt excellent
16:25:14 melwitt and I could see allowing it to masquerade as a normal Instance object could lead to a lot of subtle problems, if anything were missing from its fields
16:25:38 dansmith on the other hand, in reality,
16:25:48 dansmith we used that to smooth over where we got the instance,
16:26:02 dansmith so down the stack we've lost context (heh) about why it might not be a real one
16:26:54 melwitt yeah. I suspect if it had a real context, the problem would travel further down the stack and be really puzzling
16:27:16 melwitt *even further
16:27:24 johnthetubaguy melwitt: sorry, got distracted, which bits where you counting when you needed the cells again?
16:27:24 dansmith I dunno, the refresh should work if we have uuid and context
16:28:22 dansmith johnthetubaguy: the counting patches counts instances, ram and cores from the cell db
16:28:36 melwitt johnthetubaguy: the ram and cores, and I tally the instances while counting ram/cores. ram and cores are being counted currently by pulling the instances and going through their flavor extra specs
16:28:44 dansmith johnthetubaguy: if we lose connectivity with a cell for ten minutes, a user suddenly gets their quota back until we reconnect and could burn it in other cells
16:29:00 johnthetubaguy yeah, that makes sense
16:29:33 melwitt johnthetubaguy: the real plan is to call the placement api to get cores/ram count and use InstanceMappings for instances, but we can't get that from placement quite yet
16:30:01 johnthetubaguy melwitt: ah, true, well that to make placement do quota too ;)
16:30:09 johnthetubaguy s/to/or
16:30:48 johnthetubaguy melwitt: FWIW, its tempting to fail hard if a cell is down, and we know there is something in there we should be counting
16:30:57 dansmith johnthetubaguy: how does that work?
16:31:11 dansmith johnthetubaguy: if one cell is down, we can't do anything? that kinda defeats the purpose no?
16:31:21 dansmith well, s/do/create/ but ..
16:31:36 melwitt yeah, that seems like it would draw much ire

Earlier   Later