Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-18
15:16:41 dansmith with the scheduler
15:16:52 johnthetubaguy so its a race with when the flavor gets updated I guess
15:17:45 mriedem i'm not sure at which point you auto-disable
15:17:57 dansmith mriedem: on init_host(), if there are unmigrated instances
15:18:15 dansmith and then re-enable when you're done with the migration
15:18:44 mriedem so originally when we did the flavor migration it was on init_host and then that moved because of the hash ring stuff
15:18:44 johnthetubaguy but if the resource class isn't set on the node, we are never done
15:19:05 dansmith johnthetubaguy: that had to be set before queens, right?
15:19:09 mriedem yeah i'm trying to think if we re-enable too soon, or disable too long
15:19:29 dansmith so queens starting up can assume it's set no?
15:19:35 johnthetubaguy dansmith: yes, but this is really about making pike work
15:20:05 johnthetubaguy so right know I can't update the flavors in pike, for them to be set in time for queens, because of this bug
15:20:09 dansmith oh, was this migration in pike? I'm misremembering the timing
15:20:23 johnthetubaguy yeah, sadly
15:20:39 mriedem yeah we have to backport the fix
15:21:05 dansmith johnthetubaguy: but you can set the node class before pike, and you can run the migration before you start anything else up right/
15:21:07 mriedem so i'm still thinking queue and hook callback from RT to driver
15:21:51 dansmith mriedem: explain your queue thing again?
15:22:18 mriedem so before https://github.com/openstack/nova/blob/e11a8687aef527eee9f7c733db6cccd5b902afbb/nova/virt/ironic/driver.py#L523 we're going to have to see if the instance already has an allocation for the normalized_rc,
15:22:22 cdent does anyone have a reference to previous discussion/decisions on why we don’t do regular allocation updates? (something I can read, rather than taking us off track here)
15:22:28 mriedem if it doesn't, we have to put the instance/rc into a queue,
15:22:52 mriedem when the RT periodic runs, it calls into the driver to say, give me any instances that need their allocations updated and we'd dequeue at that point
15:23:38 mriedem then ^ should auto-heal during the periodic as the admin is setting node rcs in ironic
15:23:40 mriedem while things are running
15:24:27 mriedem cdent: https://review.openstack.org/#/c/491012/
15:24:43 cdent thanks
15:24:50 dansmith mriedem: you mean the first time we run the RT periodic we end up migrating all the instances the driver needs before we update inventory?
15:25:18 mriedem cdent: in a nutshell, in pike we want the scheduler to create allocations, including doubling them up for moves and shared providers (before we stopped trying to make shared providers work in pike) - the problem is the periodic task in ocata computes will overwrite the allocations created by the scheduler
15:25:39 mriedem dansmith: need to see if the RT updates inventory before allocations
15:26:14 mriedem it doesn't
15:26:15 mriedem shit
15:26:24 mriedem the RT would update allocations before inventory
15:26:35 dansmith even still, we run most of that code at other times than just the perioidic
15:27:47 mriedem the allocatoin update does happen during instance_claim
15:27:55 dansmith mriedem: so we still do allocation updates if we have ocata computes.. what if we keep doing them if we have ocata nodes and the driver says we need to?
15:28:06 dansmith s/and/or
15:28:37 mriedem and the driver says we need to only for ironic and only until we remove the flavor migration code
15:28:39 sean-k-mooney stephenfin: gladly. ill review it in the next 20 mins or so. i was pretty happy with the previous version so i dont expect that ill see anything wrong with it
15:28:59 dansmith mriedem: yeah
15:29:02 mriedem dansmith: that would be simpler
15:29:02 johnthetubaguy ... now that I like
15:29:21 johnthetubaguy will code that up now
15:29:33 dansmith the commit message would need to be "Extend existing shitpile with more shit because.. why not"
15:29:53 mriedem you can't migrate ironic instances anyway right
15:29:54 mriedem ?
15:30:05 mriedem so the ocata compute allocation overwrite thing is less of a concern there
15:30:07 johnthetubaguy mriedem: there is a spec on that ;)
15:30:13 dansmith no migrations at all?
15:30:16 dansmith I thought evac worked at least
15:30:16 mriedem johnthetubaguy: sure, but not in pike
15:30:24 mriedem evac is the only one i can think of
15:30:27 mriedem but,
15:30:28 johnthetubaguy oh, rebuild does I guess
15:30:35 mriedem you don't care about the allocations on the dead source host
15:30:45 mriedem although these are nodes, not hosts
15:31:16 mriedem hmm, so how does evac work for ironic - the nodes might not be down, but the nova-compute service is,
15:31:29 mriedem so you evacuate and move all of those instances to other ironic nodes managed by another compute host?
15:32:07 mriedem well regardless,
15:32:12 mriedem because of gibi's fixes,
15:32:26 mriedem when/if the source compute comes back up, we remove the allocations from the old instances that were on it
15:33:01 mriedem so yeah the callback to the driver to ask if it should update allocatoins is probably ok for ironic,
15:33:03 mriedem simpler,
15:33:06 mriedem lesser of all evils,
15:33:09 mriedem and temporary
15:34:22 dansmith not not evil, but less evil
15:34:42 mriedem like mike pence
15:40:24 dtantsur johnthetubaguy, cdent, whas was the question, could you please give a tl;dr?
15:41:02 johnthetubaguy dtantsur: hit this bug: https://bugs.launchpad.net/nova/+bug/1724589
15:41:03 openstack Launchpad bug 1724589 in OpenStack Compute (nova) "Unable to transition to Ironic Node Resource Classes in Pike" [High,In progress] - Assigned to John Garbutt (johngarbutt)
15:41:05 edleafe dtantsur: if you update an ironic flavor with a custom RC, nova may schedule to an existing node
15:41:29 edleafe because we don't create the allocation for nodes with that custom RC
15:42:23 dtantsur wait, wasn't that periodic task in the ironic virt driver written to update the allocations?
15:43:11 dansmith dtantsur: no, the flavors
15:43:29 dansmith dtantsur: when it was written the compute node would unceremoniously heal allocations
15:43:32 edleafe dtantsur: that was supposed to be part of it, but only the flavor migration was added
15:43:38 dansmith dtantsur: right over top of the scheduler, which is why we made it stop,
15:43:44 dtantsur oooooh
15:43:51 dansmith dtantsur: but that periodic in the ironic driver was depending on that
15:44:09 dtantsur was the update bit removed in Pike or Queens or both?
15:44:29 dansmith dtantsur: pike once all compute nodes have been updated
15:44:29 johnthetubaguy Pike
15:44:44 dansmith dtantsur: it would keep doing it as long as ocata nodes were in the cluster
15:46:30 dtantsur so, is our only option to update the periodic task to fix up allocations as well?
15:46:38 dtantsur I don't know nova well enough to suggest anything else :)
15:48:08 dansmith dtantsur: we're going to let ironic have a vote on the "should we keep healing" decision
15:50:05 efried cdent I wrote that test case, but I can't get it to fail properly. I suspect it may be due to that bug, though.
15:50:20 efried cdent Mebbe I post it and see if you can find holes.
15:50:34 cdent efried: sounds like a grand plan
15:52:22 dtantsur dansmith: I just don't see other options. I don't think we should ask operators to go into Placement and fix up stuff themselves. or should we?
15:52:39 dansmith dtantsur: nobody is suggesting that
15:52:40 dtantsur johnthetubaguy: btw you may also hit https://bugs.launchpad.net/nova/+bug/1723423 in your testing
15:52:41 openstack Launchpad bug 1723423 in OpenStack Compute (nova) "Ironic node cannot be used if it does not report VCPU" [Undecided,In progress] - Assigned to Dmitry Tantsur (divius)
15:53:48 johnthetubaguy dtantsur: so ironic should eventually return 0 for all the resources that are no longer being claimed?
15:54:46 dtantsur johnthetubaguy: sorry, I don't quite get it. Which of the bugs are you referring to?
15:55:07 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: CLI for resource classes https://review.openstack.org/511182
15:55:08 openstackgerrit Andrey Volkov proposed openstack/osc-placement master: [WIP] RP list: member_of and resources parameters https://review.openstack.org/511183
15:55:20 johnthetubaguy dtantsur: the one you just linked too, that might be a bit of the picture I was missing
15:55:41 johnthetubaguy dtantsur: once the resource class is updated, we stop reporting the vcpu mem and disk from ironic?
15:56:46 dtantsur johnthetubaguy: we don't, no. only if you remove 'cpus' key from node.properties
15:56:51 efried cdent Actually the problem is that AllocationCandidates refuses to find anything in the shared RP as long as the compute node also has inventory in the same RC (DISK_GB).

Earlier   Later