Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-09
17:20:01 dansmith bauzas: we can abort on startup if they have specified something that can't be reshaped, requiring them to move some stuff off before doing it, but requiring clearing every single compute node (over time) to turn that on really sucks
17:20:44 bauzas dansmith: okay so you're in favor of some automation
17:20:50 bauzas lemme reconsider that
17:20:52 mordred dansmith: just adopt cloud native design principles and stop writing files to disk. solves all your problems
17:21:15 dansmith mordred: heh
17:21:38 gmann mriedem: tssurya locked_by in valid sort/filter list was mistake. we can improve that in tssurya spec i think.
17:22:06 mriedem mistake because it's a field we don't even expose today?
17:22:11 tssurya gmann, mriedem: I will add locked" as a sort/filter key
17:22:36 mriedem i know at one point there was a comment in the code near these fields that said "don't use locked anymore" becaues locked_by implied locked
17:22:38 tssurya but as for the existing "locked_by" key in the whitelist, isn't that just a refactor of code ? since we don't expose it anyways ?
17:23:18 gmann mriedem: yes, we did not expose that and had in valid sort/fitler
17:23:34 mriedem tssurya: refactor of code?
17:23:45 gmann i tried to check the reason in original change but did not find any rational to add in whitelist
17:23:47 mriedem tssurya: as i mentioned in the spec, supporting filtering on locked_by could get weird,
17:24:11 tssurya mriedem: no I don't want to support locked_by based filtering as well
17:24:14 tssurya its useless IMO
17:24:18 mriedem becaues if we expose locked_by=admin or other, but only store admin or owner, then filtering on locked_by=other means we have to do something like "select * from instances where locked_by!='admin'"
17:24:38 dansmith filtering in the api on a non-db field isn't *super* terrible, but sorting is
17:24:47 gmann this one - https://review.openstack.org/#/c/408571/
17:24:57 bauzas dansmith: just to make sure https://review.openstack.org/#/c/555081/22/specs/train/approved/cpu-resources.rst@725 requires a reshape, right?
17:25:26 mriedem dansmith: i don't think sorting is a problem really here
17:25:29 bauzas ie. libvirt sees the options be set, raises a ReshapeNeeded, gets the inventories and allocations, move them
17:25:38 mriedem filtering would have to be special cased though, and locked_by is in the db
17:26:05 dansmith bauzas: properly representing allocations of dedicated cpus as PCPU right? Not even migration will fix those instances, so I would think reshape is the *only* way to fix that, no?
17:27:10 mriedem my pizza is getting dangerously cold
17:27:15 dansmith bauzas: unless you depend on scheduler/conductor to generate a new allocation for a properly hierarchical instance I guess, but.. that's just really super expensive when all you want to do is move some numbers around
17:27:32 bauzas dansmith: well, in a world before pcpu_shared_set, all instances are seen as using shared set of CPUs, so they need to be VCPU anyway, but the problem is that if we reserve, say CPU1, then we need to arbitratly assign instances using this CPU1 as PCPU
17:28:00 dansmith mriedem_away: I know we can filter on locked_by in the db, but I'm saying if we allow filtering on locked and then drop locked in the future, we can continue honoring that behavior from other fields
17:28:01 bauzas dansmith: but then there is an allocation ratio problem, since we don't want to oversubscribe on PCPU
17:28:32 dansmith bauzas: but L725 is talking about instances that already have dedicated cpus, but represented as VCPU right?
17:28:40 cdent efried: I'm moving on to the cross project nova+placement etherpad for my pre-ptg emails. a) cool with that? b) okay with the process I've been following c) anything especially I should or should not skip?
17:29:10 bauzas dansmith: yup, but the above example is still valid, nope?
17:29:20 bauzas it's just undocumented in the spec
17:29:55 bauzas that's what I tried to address in https://review.openstack.org/#/c/555081/22/specs/train/approved/cpu-resources.rst@181
17:30:27 bauzas actually, my wife yells at me because I need to go eating melted cheese
17:30:50 bauzas I need to disappear
17:31:18 bauzas https://en.wikipedia.org/wiki/Vacherin#Mont_d'Or
17:32:23 gmann mriedem_away: mordred let me check those fields from request point of view.
17:33:20 sean-k-mooney bauzas: stephenfin can ye review https://review.openstack.org/#/c/649882/ its related to both the cpus in placemnt spec and numa in placement
17:37:40 bauzas dansmith: left another comment trying to describe my concerns in https://review.openstack.org/#/c/555081/22/specs/train/approved/cpu-resources.rst@181
17:37:44 bauzas hopefully it will help
17:37:50 bauzas now I need to bail out
17:37:52 dansmith bauzas: I just commented summarizing my view
17:37:54 dansmith maybe that will help
17:37:56 dansmith go eat cheese
17:38:30 bauzas dansmith: okay, i'll join later
17:41:53 efried cdent: a) Sure, if you're done with nova reviews :P b) Yes, me likey. (Note that I am reading all of them, but only responding when I have something to say; I'm not sure there's positive value in an empty ack.); c) other than things you already mentioned (perhaps those should be re/moved from the main part of the etherpad?)
17:43:28 jmlowe cdent: Got a quick question about placement and vgpus
17:43:34 efried cdent: also on a) and b), I am likely to hold off on any further responses until tomorrow. Trying to focus solely on nova spec reviews today (though the gods are conspiring to make that challenging).
17:44:13 cdent a) I clicked on every link on the spec etherpad this morning and left reviews where I felt comfortable. sometimes I did not because my reaction was "oh look, yet more complexity for the five people who care" and didn't think that was too helpful, so moved on to something else, b) cool, c) I'm trying to make sure the etherpad exposes what has thread
17:44:20 cdent jmlowe: hola
17:45:05 cdent jmlowe: I can try to help, but the specifics of how the vgpus are being managed is not much in my wheelhouse
17:47:12 mriedem tssurya: so maybe a question that you don't want to hear right now, but do we really need to expose that locked_by field? it seems to be causing a lot more unnecessary churn on this spec than is needed
17:47:27 mriedem if the point of the spec is just expose a reason field, can we just....do that?
17:48:29 jmlowe cdent: So we got 6 nodes with 4 nvidia v100's each for a total of 24, the thing that's confusing me is that in the docs the example is "openstack flavor set vgpu_1 --property "resources:VGPU=1"", I can slice these 8 ways, how do I manipulate the resources in placement so that I have 4*8 vgpus? Or do I have a GCE when it comes to resources, filter scheduler, and placement?
17:50:56 jmlowe seems like a major omission, how do I tell placement how many of some arbitrary thing I have?
17:51:15 openstackgerrit Surya Seetharaman proposed openstack/nova-specs master: Support adding the reason behind a server lock https://review.openstack.org/638629
17:51:53 cdent jmlowe: what release are you on these days?
17:52:01 jmlowe Rocky
17:52:04 sean-k-mooney jmlowe: bauzas will be able to answer but you should not need too
17:52:22 tssurya mriedem: just saw your comment here :/
17:52:36 mriedem i left them on the spec as well, you can answer whenever
17:52:55 tssurya hmm, ok if we are going to expose "locked" after all then locked_by is not needed
17:53:14 sean-k-mooney jmlowe: there are some limitation in what we currently supprot but as of stien we shoudl now have a nested resoucp provider per phyical gpu
17:53:35 sean-k-mooney each phyical gpu will have an inventory of 8 vGPU reources
17:53:50 jmlowe any chance this is explained in the Stein admin guide?
17:54:15 cdent jmlowe: the issue you've got is that what's possible in Stein is a lot different from what's possible in Rocky
17:54:21 sean-k-mooney jmlowe: in rocky i know we did not support multiple vgpus type on the same host but we might have supported multipel pysical gpus
17:54:45 jmlowe *looking ahead to stein admin guide
17:55:16 sean-k-mooney jmlowe: i think bauzas wrote something up yes but not sure if its in the admin guide
17:55:45 mriedem tssurya: we already expose locked :)
17:56:38 jmlowe ok, this is starting to make more sense now that I understand the limitations of <= Rocky
17:56:46 mriedem https://docs.openstack.org/nova/rocky/admin/virtual-gpu.html
17:56:50 mriedem that's the vgpu admin guide for rocky
17:57:11 mriedem https://docs.openstack.org/nova/stein/admin/virtual-gpu.html#checking-allocations-and-inventories-for-virtual-gpus is new content for stein
17:57:33 mriedem the placement CLI examples in there may or may not be useful to you in rocky
17:57:40 sean-k-mooney ya so in rocky we supporte 1 gpu type per node but i belive we had supported muplip phyical gpus that were reported in 1 inventory on the compute node RP
17:57:46 mriedem the main difference is in stein there are nested providers for VGPU inventory and allocations
17:58:00 sean-k-mooney and in stien we reshap them into child resouce providers
17:58:09 cdent jmlowe: this may be informative: https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#checking-allocations-and-inventories-for-virtual-gpus
17:58:32 cdent oh, sorry, mriedem just did that
17:58:54 cdent was off looking.
18:00:11 cdent Is dinner time
18:02:32 tssurya efried: wait wait I am updating that spec to remove that part which you just commented
18:02:47 efried tssurya: Okay. Don't worry, I wasn't merging it :)
18:02:57 efried you're going to just not display locked_by at all?
18:03:06 efried Sigh, I guess that's one way to skin this cat.
18:03:44 efried And you would know better than I whether a user would want to report on "locked by me" and/or "locked by not-me".
18:03:58 tssurya efried: yea
18:04:06 tssurya I am going to kick locked_by
18:04:13 tssurya that's causing all the confusion
18:04:26 efried And I guess that could be added later, though it would seem like a lot of work to spin a whole new blueprint/spec/microversion just for that.
18:04:43 tssurya efried: yea that's why I had intially added it here
18:04:52 tssurya but I hope nobody would want locked_by ?
18:05:01 efried that's the thing
18:05:38 tssurya when the user tries to unlock they would know if it was "locked by me" or not
18:05:43 efried like I'm doing maintenance, so I lock a bunch of things, and then my maint is done, so I want to unlock the things. But I forgot to keep track of the ones I locked. So I want a report that says "hey, what did I lock?"
18:06:00 dansmith I was told that the way to tell if it's locked by me is to unlock it :P
18:06:15 efried yeah. I could script to unlock "everything" and tolerate the failures ^
18:06:17 dansmith which seems silly to me

Earlier   Later