Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-27
17:19:43 jroll More than one endpoint exists with the name 'placement'. <- not cool, devstack, not cool
17:19:56 figleaf johnthetubaguy: cdent: what it looks like is that we need to make sure that the update_instance_allocation() calls are all checked to make sure that they aren't called for new ironic nodes
17:20:19 mriedem johnthetubaguy: i'll go through PS38 too
17:20:36 johnthetubaguy figleaf: but there are no new calls right now, so surely we need someone to claim the new resource class?
17:21:36 figleaf johnthetubaguy: I only see two places where update_instance_allocations() is called in the RT.
17:21:44 johnthetubaguy figleaf: I would rather update_instance_allocation() was able to know from the instance what resources it should be claiming, and it just claims the correct ones, for now thats a bit hard coded I guess, but eventually its the same logic the scheduler has to do when requesting the correct resource class
17:22:24 figleaf johnthetubaguy: yeah, this is all transitional code
17:22:59 figleaf johnthetubaguy: we wanted to get ironic handled before we are able to handle all types of RPs
17:23:21 figleaf johnthetubaguy: can you think of a test that would fail now with ironic?
17:23:42 johnthetubaguy figleaf: add resource class on ironic node, boot instance that will land there, resource claim fails
17:23:50 johnthetubaguy figleaf: at least, thats the way it looks to me
17:23:56 cdent figleaf, johnthetubaguy my read of the code in the rt ironic inventory patch is that johnthetubaguy is right: it is possible to reache update_instance_allocation with a compute node and an instance and depending on what "compute_node" is, the allocations will either fail, or apply to the nova-compute node, incorrectly
17:24:19 cdent because the data at that point is about resource usages (vcpu etc)
17:24:49 johnthetubaguy yeah, instead of allocating for vcpu, it should be for the custom resource class, I think
17:25:21 johnthetubaguy oh balls...
17:25:21 cdent which should mean that the ironic "instance" is not include in the loop that is calling update_instance_allocation
17:25:27 cdent because it isn't really an instance
17:25:36 cdent (in that context)
17:25:45 johnthetubaguy if we merge the code to query placement, we will never land on an iroinc node in the first place, unless it is reporting VCPU resources
17:26:02 figleaf johnthetubaguy: that's a known issue
17:26:11 cdent "oh balls" means we've passed the "bugger" state of affairs
17:26:23 johnthetubaguy cdent: heh, ack
17:26:25 figleaf johnthetubaguy: which is why ironic has to keep reporting vm-type resources until Pike
17:26:45 johnthetubaguy figleaf: but this code doesn't do that, I thought
17:27:00 cdent brb
17:27:21 figleaf johnthetubaguy: digging into the claim stuff - haven't looked at that in a while
17:28:43 johnthetubaguy figleaf: actually, I am changing my mind on the fix here, I think its probably just that we should still be reporting VCPUs, we can't drop those till we schedule based on the new resources
17:29:25 figleaf johnthetubaguy: yes, that's been the plan
17:30:09 johnthetubaguy figleaf: but the code only reports the custom resource classes, or did I read that wrongly?
17:30:35 cdent johnthetubaguy, figleaf: can we restate the problem again (I know we keep doing that, but I think it is useful): If we released this code today without any change it would be wrong because:
17:31:31 figleaf cdent: ...once an ironic node reports resources with a custom resource class, all the VCPU etc. stuff gets removed
17:31:49 figleaf cdent: ...and we have no way to then select that ironic node
17:32:04 figleaf johnthetubaguy: cdent: Does that sound right?
17:32:06 openstackgerrit John Garbutt proposed openstack/nova: Block starting compute unless placement conf is provided https://review.openstack.org/425806
17:32:12 cdent "reports resources" == writes inventory?
17:32:20 figleaf yeah
17:34:06 figleaf cdent: the _cleanup_ironic_legacy_allocations() method removes all the old-style inventory
17:34:48 cdent (if there was any)
17:35:04 mriedem the funny thing is, ironic + placement in newton doesn't work anyway
17:35:13 figleaf cdent: well, unless the node is newly-added, there will be
17:35:20 mriedem i.e. https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/newton+topic:bug/1647316
17:35:51 cdent figleaf: right, but when a node is newly added the same problem exists: there's no way to allocate or claim inventory?
17:35:51 figleaf mriedem: yeah - "funny"
17:36:10 johnthetubaguy mriedem: its almost like we need some ironic testing
17:36:27 mriedem johnthetubaguy: we have it in ocata
17:36:32 mriedem just not in newton where placement was optional
17:36:41 johnthetubaguy mriedem: ah
17:36:46 mriedem ironic + placement ci was broken for a few days in master
17:36:54 figleaf cdent: if a node is added and has a custom RC, its inventory will be added new-style
17:36:56 mriedem which is why i'm backporting that series of fixes
17:37:05 cdent figleaf: yes, but you can't allocate against it
17:37:06 jroll bah, no '-' in custom resource classes? :|
17:37:08 figleaf cdent: and we have no way to select that node now
17:37:16 cdent and that too
17:37:24 cdent feh
17:37:30 mriedem jroll: _ is far superior
17:37:49 cdent jroll: just upperletters and numbers
17:37:55 jroll and _
17:38:19 jroll kinda lame IMO, but alas
17:38:26 cdent jroll: yeah, but that's just to keep you conused so best to just leave it out ...
17:38:55 johnthetubaguy figleaf: cdent: so the summary, either we have VCPUs, claim them, and schedule on them, OR we have to schedule on custom_resource classes and claim them, current state: schedule VCPU, report custom, claim VCPU (except for existing instances that we tidy up to claim the new resource class)
17:38:56 cdent I _think_ the constraint came into play when we didn't even have custom resource classes, not sure.
17:39:01 openstackgerrit Matt Riedemann proposed openstack/nova: Block starting compute unless placement conf is provided https://review.openstack.org/425806
17:41:17 jroll well, unless I did something awful wrong, the ironic code gives me 2017-01-27 17:39:56.654 TRACE nova.compute.manager AttributeError: 'SchedulerClient' object has no attribute 'set_provider_inventory'
17:42:19 jroll nope, I'm at b6339a876a440831903445bd4b10afb732e91335 which is latest here
17:42:29 johnthetubaguy hmm, interesting
17:43:16 cdent this sounds like a job for functional tests
17:43:17 figleaf jroll: there are some places I found where SchedulerClient and SchedulerClient.reportclient are treated as the same
17:43:27 figleaf It's in reportclient
17:44:44 jroll figleaf: just reporting this code is broken :)
17:44:57 jroll cdent: I'm working on integration stuff atm
17:45:28 figleaf jroll: heh, can relate to that
17:46:00 cdent jroll: which is great, but there's something in the middle: more functional testing while we are developing the features (just setting a bit for future efforts, not moaning)
17:46:40 jroll cdent: sure, don't disagree
17:47:46 cdent jroll: shit bro, it's not you who I think should have done functional tests on this stuff. It's me and figleaf and jaypipes
17:48:33 jroll cdent: heh. it's all of us. I just want to help get this working
17:49:12 cdent figleaf: are you working on dealing with the traceback jroll's reported, or should I do that?
17:49:21 jroll figleaf: okay, that worked, now I should see an inventory record that is mapped to that custom resource class, yeah?
17:49:53 figleaf jroll: sorry, I'm looking at 6 different bits of code, trying to keep them straight :(
17:50:06 figleaf jroll: what was it that you were doing when you hit the bug
17:50:07 figleaf ?
17:50:23 cdent traceback on the review
17:50:30 jroll figleaf: so I've got devstack. ironic stuff has resource classes. using this code.
17:50:40 jroll fixed traceback as mentioned (leaving a comment on that now)
17:51:02 jroll looks like things go through but I don't see anything in inventory table that references the custom RP
17:51:07 jroll er, s/RP/RC/
17:56:55 figleaf jroll: ugh - looks like there are more of those all over the place
18:01:01 cdent figleaf, jroll, johnthetubaguy: sorry guys I gotta go. I haven't eaten enough and starting to fade badly. If there's something I can do on sunday or in the UK morning monday, let me know, either by email to os-dev or direct to me.
18:02:01 jroll cdent: no worries man, I'll just leave anything I find on gerrit
18:02:09 jroll looks like it submits inventory correctly
18:02:10 johnthetubaguy cdent: figleaf: I am going to run soon too, similar comments about monday morning
18:02:28 cdent best of luck
18:02:42 johnthetubaguy jroll: well, not sure, once we get bauzas's change in, you will no longer be able to pick iroinc nodes if the current proposal merges
18:02:59 johnthetubaguy figleaf: I attempted to add a summary comment about what I think we should do
18:03:02 jroll johnthetubaguy: ... O_o
18:03:09 figleaf johnthetubaguy: cdent: ok, thanks for opening up this can of worms! (Seriously! It would have really sucked to let this out like this)
18:03:50 johnthetubaguy there may be 40% nonsense in my current suggestion, but lets see if that makes things clearer or not
18:05:17 jroll johnthetubaguy: lgtm, I think
18:06:22 johnthetubaguy figleaf: are you on board with the plan I sketched out in https://review.openstack.org/#/c/404472?
18:13:21 openstackgerrit John Garbutt proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961

Earlier   Later