| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-27 | |||
| 17:23:50 | johnthetubaguy | figleaf: at least, thats the way it looks to me | |
| 17:23:56 | cdent | figleaf, johnthetubaguy my read of the code in the rt ironic inventory patch is that johnthetubaguy is right: it is possible to reache update_instance_allocation with a compute node and an instance and depending on what "compute_node" is, the allocations will either fail, or apply to the nova-compute node, incorrectly | |
| 17:24:19 | cdent | because the data at that point is about resource usages (vcpu etc) | |
| 17:24:49 | johnthetubaguy | yeah, instead of allocating for vcpu, it should be for the custom resource class, I think | |
| 17:25:21 | johnthetubaguy | oh balls... | |
| 17:25:21 | cdent | which should mean that the ironic "instance" is not include in the loop that is calling update_instance_allocation | |
| 17:25:27 | cdent | because it isn't really an instance | |
| 17:25:36 | cdent | (in that context) | |
| 17:25:45 | johnthetubaguy | if we merge the code to query placement, we will never land on an iroinc node in the first place, unless it is reporting VCPU resources | |
| 17:26:02 | figleaf | johnthetubaguy: that's a known issue | |
| 17:26:11 | cdent | "oh balls" means we've passed the "bugger" state of affairs | |
| 17:26:23 | johnthetubaguy | cdent: heh, ack | |
| 17:26:25 | figleaf | johnthetubaguy: which is why ironic has to keep reporting vm-type resources until Pike | |
| 17:26:45 | johnthetubaguy | figleaf: but this code doesn't do that, I thought | |
| 17:27:00 | cdent | brb | |
| 17:27:21 | figleaf | johnthetubaguy: digging into the claim stuff - haven't looked at that in a while | |
| 17:28:43 | johnthetubaguy | figleaf: actually, I am changing my mind on the fix here, I think its probably just that we should still be reporting VCPUs, we can't drop those till we schedule based on the new resources | |
| 17:29:25 | figleaf | johnthetubaguy: yes, that's been the plan | |
| 17:30:09 | johnthetubaguy | figleaf: but the code only reports the custom resource classes, or did I read that wrongly? | |
| 17:30:35 | cdent | johnthetubaguy, figleaf: can we restate the problem again (I know we keep doing that, but I think it is useful): If we released this code today without any change it would be wrong because: | |
| 17:31:31 | figleaf | cdent: ...once an ironic node reports resources with a custom resource class, all the VCPU etc. stuff gets removed | |
| 17:31:49 | figleaf | cdent: ...and we have no way to then select that ironic node | |
| 17:32:04 | figleaf | johnthetubaguy: cdent: Does that sound right? | |
| 17:32:06 | openstackgerrit | John Garbutt proposed openstack/nova: Block starting compute unless placement conf is provided https://review.openstack.org/425806 | |
| 17:32:12 | cdent | "reports resources" == writes inventory? | |
| 17:32:20 | figleaf | yeah | |
| 17:34:06 | figleaf | cdent: the _cleanup_ironic_legacy_allocations() method removes all the old-style inventory | |
| 17:34:48 | cdent | (if there was any) | |
| 17:35:04 | mriedem | the funny thing is, ironic + placement in newton doesn't work anyway | |
| 17:35:13 | figleaf | cdent: well, unless the node is newly-added, there will be | |
| 17:35:20 | mriedem | i.e. https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/newton+topic:bug/1647316 | |
| 17:35:51 | cdent | figleaf: right, but when a node is newly added the same problem exists: there's no way to allocate or claim inventory? | |
| 17:35:51 | figleaf | mriedem: yeah - "funny" | |
| 17:36:10 | johnthetubaguy | mriedem: its almost like we need some ironic testing | |
| 17:36:27 | mriedem | johnthetubaguy: we have it in ocata | |
| 17:36:32 | mriedem | just not in newton where placement was optional | |
| 17:36:41 | johnthetubaguy | mriedem: ah | |
| 17:36:46 | mriedem | ironic + placement ci was broken for a few days in master | |
| 17:36:54 | figleaf | cdent: if a node is added and has a custom RC, its inventory will be added new-style | |
| 17:36:56 | mriedem | which is why i'm backporting that series of fixes | |
| 17:37:05 | cdent | figleaf: yes, but you can't allocate against it | |
| 17:37:06 | jroll | bah, no '-' in custom resource classes? :| | |
| 17:37:08 | figleaf | cdent: and we have no way to select that node now | |
| 17:37:16 | cdent | and that too | |
| 17:37:24 | cdent | feh | |
| 17:37:30 | mriedem | jroll: _ is far superior | |
| 17:37:49 | cdent | jroll: just upperletters and numbers | |
| 17:37:55 | jroll | and _ | |
| 17:38:19 | jroll | kinda lame IMO, but alas | |
| 17:38:26 | cdent | jroll: yeah, but that's just to keep you conused so best to just leave it out ... | |
| 17:38:55 | johnthetubaguy | figleaf: cdent: so the summary, either we have VCPUs, claim them, and schedule on them, OR we have to schedule on custom_resource classes and claim them, current state: schedule VCPU, report custom, claim VCPU (except for existing instances that we tidy up to claim the new resource class) | |
| 17:38:56 | cdent | I _think_ the constraint came into play when we didn't even have custom resource classes, not sure. | |
| 17:39:01 | openstackgerrit | Matt Riedemann proposed openstack/nova: Block starting compute unless placement conf is provided https://review.openstack.org/425806 | |
| 17:41:17 | jroll | well, unless I did something awful wrong, the ironic code gives me 2017-01-27 17:39:56.654 TRACE nova.compute.manager AttributeError: 'SchedulerClient' object has no attribute 'set_provider_inventory' | |
| 17:42:19 | jroll | nope, I'm at b6339a876a440831903445bd4b10afb732e91335 which is latest here | |
| 17:42:29 | johnthetubaguy | hmm, interesting | |
| 17:43:16 | cdent | this sounds like a job for functional tests | |
| 17:43:17 | figleaf | jroll: there are some places I found where SchedulerClient and SchedulerClient.reportclient are treated as the same | |
| 17:43:27 | figleaf | It's in reportclient | |
| 17:44:44 | jroll | figleaf: just reporting this code is broken :) | |
| 17:44:57 | jroll | cdent: I'm working on integration stuff atm | |
| 17:45:28 | figleaf | jroll: heh, can relate to that | |
| 17:46:00 | cdent | jroll: which is great, but there's something in the middle: more functional testing while we are developing the features (just setting a bit for future efforts, not moaning) | |
| 17:46:40 | jroll | cdent: sure, don't disagree | |
| 17:47:46 | cdent | jroll: shit bro, it's not you who I think should have done functional tests on this stuff. It's me and figleaf and jaypipes | |
| 17:48:33 | jroll | cdent: heh. it's all of us. I just want to help get this working | |
| 17:49:12 | cdent | figleaf: are you working on dealing with the traceback jroll's reported, or should I do that? | |
| 17:49:21 | jroll | figleaf: okay, that worked, now I should see an inventory record that is mapped to that custom resource class, yeah? | |
| 17:49:53 | figleaf | jroll: sorry, I'm looking at 6 different bits of code, trying to keep them straight :( | |
| 17:50:06 | figleaf | jroll: what was it that you were doing when you hit the bug | |
| 17:50:07 | figleaf | ? | |
| 17:50:23 | cdent | traceback on the review | |
| 17:50:30 | jroll | figleaf: so I've got devstack. ironic stuff has resource classes. using this code. | |
| 17:50:40 | jroll | fixed traceback as mentioned (leaving a comment on that now) | |
| 17:51:02 | jroll | looks like things go through but I don't see anything in inventory table that references the custom RP | |
| 17:51:07 | jroll | er, s/RP/RC/ | |
| 17:56:55 | figleaf | jroll: ugh - looks like there are more of those all over the place | |
| 18:01:01 | cdent | figleaf, jroll, johnthetubaguy: sorry guys I gotta go. I haven't eaten enough and starting to fade badly. If there's something I can do on sunday or in the UK morning monday, let me know, either by email to os-dev or direct to me. | |
| 18:02:01 | jroll | cdent: no worries man, I'll just leave anything I find on gerrit | |
| 18:02:09 | jroll | looks like it submits inventory correctly | |
| 18:02:10 | johnthetubaguy | cdent: figleaf: I am going to run soon too, similar comments about monday morning | |
| 18:02:28 | cdent | best of luck | |
| 18:02:42 | johnthetubaguy | jroll: well, not sure, once we get bauzas's change in, you will no longer be able to pick iroinc nodes if the current proposal merges | |
| 18:02:59 | johnthetubaguy | figleaf: I attempted to add a summary comment about what I think we should do | |
| 18:03:02 | jroll | johnthetubaguy: ... O_o | |
| 18:03:09 | figleaf | johnthetubaguy: cdent: ok, thanks for opening up this can of worms! (Seriously! It would have really sucked to let this out like this) | |
| 18:03:50 | johnthetubaguy | there may be 40% nonsense in my current suggestion, but lets see if that makes things clearer or not | |
| 18:05:17 | jroll | johnthetubaguy: lgtm, I think | |
| 18:06:22 | johnthetubaguy | figleaf: are you on board with the plan I sketched out in https://review.openstack.org/#/c/404472? | |
| 18:13:21 | openstackgerrit | John Garbutt proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | |
| 18:13:53 | figleaf | johnthetubaguy: sorry, working on a different fix atm. Looking now... | |
| 18:14:22 | johnthetubaguy | figleaf: so as I understood it, this is one of the two remaining FFEs, hence my focus on it | |
| 18:14:58 | jroll | fwiw I'm just getting lost in this code at this point | |
| 18:15:12 | figleaf | johnthetubaguy: I was thinking similarly - getting rid of the cleanup of old-style inventory | |
| 18:15:23 | figleaf | jroll: welcome to the club!! | |
| 18:16:05 | figleaf | johnthetubaguy: and trying to ensure that both old- and new-style can coexist | |
| 18:16:40 | figleaf | johnthetubaguy: and then add back the cleanup in Pike, remove it in Queens | |
| 18:16:54 | johnthetubaguy | figleaf: cool, yeah, I think we just report the new resources, for the first step | |
| 18:17:39 | johnthetubaguy | figleaf: yeah, the clean up can only be added after we start scheduling and claiming using the new resource class, I think, we can probably drop reporting the old ones at that point, but not 100% sure | |
| 18:18:30 | johnthetubaguy | actually the scheduling can happen first, then we can start claiming the new style, then we drop reporting the legacy resource classes | |