| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-18 | |||
| 15:16:52 | johnthetubaguy | so its a race with when the flavor gets updated I guess | |
| 15:17:45 | mriedem | i'm not sure at which point you auto-disable | |
| 15:17:57 | dansmith | mriedem: on init_host(), if there are unmigrated instances | |
| 15:18:15 | dansmith | and then re-enable when you're done with the migration | |
| 15:18:44 | johnthetubaguy | but if the resource class isn't set on the node, we are never done | |
| 15:18:44 | mriedem | so originally when we did the flavor migration it was on init_host and then that moved because of the hash ring stuff | |
| 15:19:05 | dansmith | johnthetubaguy: that had to be set before queens, right? | |
| 15:19:09 | mriedem | yeah i'm trying to think if we re-enable too soon, or disable too long | |
| 15:19:29 | dansmith | so queens starting up can assume it's set no? | |
| 15:19:35 | johnthetubaguy | dansmith: yes, but this is really about making pike work | |
| 15:20:05 | johnthetubaguy | so right know I can't update the flavors in pike, for them to be set in time for queens, because of this bug | |
| 15:20:09 | dansmith | oh, was this migration in pike? I'm misremembering the timing | |
| 15:20:23 | johnthetubaguy | yeah, sadly | |
| 15:20:39 | mriedem | yeah we have to backport the fix | |
| 15:21:05 | dansmith | johnthetubaguy: but you can set the node class before pike, and you can run the migration before you start anything else up right/ | |
| 15:21:07 | mriedem | so i'm still thinking queue and hook callback from RT to driver | |
| 15:21:51 | dansmith | mriedem: explain your queue thing again? | |
| 15:22:18 | mriedem | so before https://github.com/openstack/nova/blob/e11a8687aef527eee9f7c733db6cccd5b902afbb/nova/virt/ironic/driver.py#L523 we're going to have to see if the instance already has an allocation for the normalized_rc, | |
| 15:22:22 | cdent | does anyone have a reference to previous discussion/decisions on why we don’t do regular allocation updates? (something I can read, rather than taking us off track here) | |
| 15:22:28 | mriedem | if it doesn't, we have to put the instance/rc into a queue, | |
| 15:22:52 | mriedem | when the RT periodic runs, it calls into the driver to say, give me any instances that need their allocations updated and we'd dequeue at that point | |
| 15:23:38 | mriedem | then ^ should auto-heal during the periodic as the admin is setting node rcs in ironic | |
| 15:23:40 | mriedem | while things are running | |
| 15:24:27 | mriedem | cdent: https://review.openstack.org/#/c/491012/ | |
| 15:24:43 | cdent | thanks | |
| 15:24:50 | dansmith | mriedem: you mean the first time we run the RT periodic we end up migrating all the instances the driver needs before we update inventory? | |
| 15:25:18 | mriedem | cdent: in a nutshell, in pike we want the scheduler to create allocations, including doubling them up for moves and shared providers (before we stopped trying to make shared providers work in pike) - the problem is the periodic task in ocata computes will overwrite the allocations created by the scheduler | |
| 15:25:39 | mriedem | dansmith: need to see if the RT updates inventory before allocations | |
| 15:26:14 | mriedem | it doesn't | |
| 15:26:15 | mriedem | shit | |
| 15:26:24 | mriedem | the RT would update allocations before inventory | |
| 15:26:35 | dansmith | even still, we run most of that code at other times than just the perioidic | |
| 15:27:47 | mriedem | the allocatoin update does happen during instance_claim | |
| 15:27:55 | dansmith | mriedem: so we still do allocation updates if we have ocata computes.. what if we keep doing them if we have ocata nodes and the driver says we need to? | |
| 15:28:06 | dansmith | s/and/or | |
| 15:28:37 | mriedem | and the driver says we need to only for ironic and only until we remove the flavor migration code | |
| 15:28:39 | sean-k-mooney | stephenfin: gladly. ill review it in the next 20 mins or so. i was pretty happy with the previous version so i dont expect that ill see anything wrong with it | |
| 15:28:59 | dansmith | mriedem: yeah | |
| 15:29:02 | johnthetubaguy | ... now that I like | |
| 15:29:02 | mriedem | dansmith: that would be simpler | |
| 15:29:21 | johnthetubaguy | will code that up now | |
| 15:29:33 | dansmith | the commit message would need to be "Extend existing shitpile with more shit because.. why not" | |
| 15:29:53 | mriedem | you can't migrate ironic instances anyway right | |
| 15:29:54 | mriedem | ? | |
| 15:30:05 | mriedem | so the ocata compute allocation overwrite thing is less of a concern there | |
| 15:30:07 | johnthetubaguy | mriedem: there is a spec on that ;) | |
| 15:30:13 | dansmith | no migrations at all? | |
| 15:30:16 | mriedem | johnthetubaguy: sure, but not in pike | |
| 15:30:16 | dansmith | I thought evac worked at least | |
| 15:30:24 | mriedem | evac is the only one i can think of | |
| 15:30:27 | mriedem | but, | |
| 15:30:28 | johnthetubaguy | oh, rebuild does I guess | |
| 15:30:35 | mriedem | you don't care about the allocations on the dead source host | |
| 15:30:45 | mriedem | although these are nodes, not hosts | |
| 15:31:16 | mriedem | hmm, so how does evac work for ironic - the nodes might not be down, but the nova-compute service is, | |
| 15:31:29 | mriedem | so you evacuate and move all of those instances to other ironic nodes managed by another compute host? | |
| 15:32:07 | mriedem | well regardless, | |
| 15:32:12 | mriedem | because of gibi's fixes, | |
| 15:32:26 | mriedem | when/if the source compute comes back up, we remove the allocations from the old instances that were on it | |
| 15:33:01 | mriedem | so yeah the callback to the driver to ask if it should update allocatoins is probably ok for ironic, | |
| 15:33:03 | mriedem | simpler, | |
| 15:33:06 | mriedem | lesser of all evils, | |
| 15:33:09 | mriedem | and temporary | |
| 15:34:22 | dansmith | not not evil, but less evil | |
| 15:34:42 | mriedem | like mike pence | |
| 15:40:24 | dtantsur | johnthetubaguy, cdent, whas was the question, could you please give a tl;dr? | |
| 15:41:02 | johnthetubaguy | dtantsur: hit this bug: https://bugs.launchpad.net/nova/+bug/1724589 | |
| 15:41:03 | openstack | Launchpad bug 1724589 in OpenStack Compute (nova) "Unable to transition to Ironic Node Resource Classes in Pike" [High,In progress] - Assigned to John Garbutt (johngarbutt) | |
| 15:41:05 | edleafe | dtantsur: if you update an ironic flavor with a custom RC, nova may schedule to an existing node | |
| 15:41:29 | edleafe | because we don't create the allocation for nodes with that custom RC | |
| 15:42:23 | dtantsur | wait, wasn't that periodic task in the ironic virt driver written to update the allocations? | |
| 15:43:11 | dansmith | dtantsur: no, the flavors | |
| 15:43:29 | dansmith | dtantsur: when it was written the compute node would unceremoniously heal allocations | |
| 15:43:32 | edleafe | dtantsur: that was supposed to be part of it, but only the flavor migration was added | |
| 15:43:38 | dansmith | dtantsur: right over top of the scheduler, which is why we made it stop, | |
| 15:43:44 | dtantsur | oooooh | |
| 15:43:51 | dansmith | dtantsur: but that periodic in the ironic driver was depending on that | |
| 15:44:09 | dtantsur | was the update bit removed in Pike or Queens or both? | |
| 15:44:29 | johnthetubaguy | Pike | |
| 15:44:29 | dansmith | dtantsur: pike once all compute nodes have been updated | |
| 15:44:44 | dansmith | dtantsur: it would keep doing it as long as ocata nodes were in the cluster | |
| 15:46:30 | dtantsur | so, is our only option to update the periodic task to fix up allocations as well? | |
| 15:46:38 | dtantsur | I don't know nova well enough to suggest anything else :) | |
| 15:48:08 | dansmith | dtantsur: we're going to let ironic have a vote on the "should we keep healing" decision | |
| 15:50:05 | efried | cdent I wrote that test case, but I can't get it to fail properly. I suspect it may be due to that bug, though. | |
| 15:50:20 | efried | cdent Mebbe I post it and see if you can find holes. | |
| 15:50:34 | cdent | efried: sounds like a grand plan | |
| 15:52:22 | dtantsur | dansmith: I just don't see other options. I don't think we should ask operators to go into Placement and fix up stuff themselves. or should we? | |
| 15:52:39 | dansmith | dtantsur: nobody is suggesting that | |
| 15:52:40 | dtantsur | johnthetubaguy: btw you may also hit https://bugs.launchpad.net/nova/+bug/1723423 in your testing | |
| 15:52:41 | openstack | Launchpad bug 1723423 in OpenStack Compute (nova) "Ironic node cannot be used if it does not report VCPU" [Undecided,In progress] - Assigned to Dmitry Tantsur (divius) | |
| 15:53:48 | johnthetubaguy | dtantsur: so ironic should eventually return 0 for all the resources that are no longer being claimed? | |
| 15:54:46 | dtantsur | johnthetubaguy: sorry, I don't quite get it. Which of the bugs are you referring to? | |
| 15:55:07 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for resource classes https://review.openstack.org/511182 | |
| 15:55:08 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: [WIP] RP list: member_of and resources parameters https://review.openstack.org/511183 | |
| 15:55:20 | johnthetubaguy | dtantsur: the one you just linked too, that might be a bit of the picture I was missing | |
| 15:55:41 | johnthetubaguy | dtantsur: once the resource class is updated, we stop reporting the vcpu mem and disk from ironic? | |
| 15:56:46 | dtantsur | johnthetubaguy: we don't, no. only if you remove 'cpus' key from node.properties | |
| 15:56:51 | efried | cdent Actually the problem is that AllocationCandidates refuses to find anything in the shared RP as long as the compute node also has inventory in the same RC (DISK_GB). | |
| 15:56:52 | dtantsur | which is what I tried doing in my testing | |