| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-18 | |||
| 14:09:17 | gibi | bauzas: hi! I pushed the follow up patch to refactor allocation removal as you suggested https://review.openstack.org/#/c/513041/ | |
| 14:10:18 | gmann | janki: no prob, i got it fro log and its None | |
| 14:10:48 | janki | gmann, but then again floating IP related calls are passing for other tempest tests expect this one. | |
| 14:10:53 | janki | gmann, where did you find it? | |
| 14:11:06 | openstackgerrit | Steve Noyes proposed openstack/nova master: Update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 14:13:10 | johnthetubaguy | I am looking at ironic and resource classes, and hitting some problems with the transition around claims, is that a known issue / known user error? | |
| 14:14:37 | johnthetubaguy | basically we update the resource class in the flavor, but the allocations don't get updated | |
| 14:14:47 | johnthetubaguy | dunno if that is as designed | |
| 14:14:52 | johnthetubaguy | it seems to cause problems | |
| 14:15:24 | cdent | johnthetubaguy: you mean already exisitng allocations? | |
| 14:15:30 | johnthetubaguy | cdent: yes | |
| 14:15:50 | cdent | I think you’d have to do some kind of move/migration/resize/whatever for them to change | |
| 14:15:52 | mriedem | johnthetubaguy: isn't that similar to editing a flavor on an existing instance? which we don't allow outside of resize? | |
| 14:16:35 | mriedem | we probably never considered that, but now that flavor resource allocations are going to be tied to classes in the flavor extra specs, and you can edit extra specs at will, | |
| 14:16:41 | johnthetubaguy | so... I should roll back, this is basically trying to do the Pike resource class transition for ironic | |
| 14:16:44 | mriedem | people might think that will auto-adjust the instance using that flavor somehow | |
| 14:17:14 | johnthetubaguy | so I update my ironic nodes to have resources classes, thats all cool | |
| 14:17:40 | johnthetubaguy | put the existing instances have only allocations for some of the resources now | |
| 14:18:02 | johnthetubaguy | so if I update my existing flavors to request the new resource class, and stop requesting VCPU I have a problem | |
| 14:18:33 | johnthetubaguy | when I do a build instance, obviously I see all the nodes with existing instances as candidate hosts, as they still have the resources I need | |
| 14:18:43 | johnthetubaguy | boom... my transition path is busted | |
| 14:19:06 | johnthetubaguy | now what does work, is keeping claiming VCPU and RAM in the flavor | |
| 14:19:26 | johnthetubaguy | but I think that will cause problems in queens when we stop reporting those resources for ironic | |
| 14:19:47 | johnthetubaguy | ... wondering if I am missing something here | |
| 14:20:52 | cdent | does it make any different if you keep the old flavors and makeentirely new flavors? | |
| 14:21:05 | johnthetubaguy | no, same resource request problem | |
| 14:21:48 | mriedem | johnthetubaguy: are you seeing logs in the compute from here? https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L531 | |
| 14:21:51 | cdent | what about not updating the in use node to have resource classes? | |
| 14:22:08 | johnthetubaguy | all my nodes have a resource class now | |
| 14:22:31 | johnthetubaguy | mriedem: that stuff all works, all my instances have their flavor updated, but the allocations are not refreshed | |
| 14:22:59 | johnthetubaguy | (basically I did update the in use nodes to have a resource class) | |
| 14:23:02 | mriedem | johnthetubaguy: did you change the allocation amounts? | |
| 14:23:04 | efried | mriedem dansmith Can we get https://review.openstack.org/#/c/510957/ pushed through, please? Clean cherry-pick, and got a vendor request for it. | |
| 14:23:19 | johnthetubaguy | mriedem: where would I do that? | |
| 14:23:31 | efried | (cc gibi) | |
| 14:23:40 | mriedem | johnthetubaguy: well i'm confused what you mean by allocations being refreshed | |
| 14:23:52 | mriedem | the allocation amounts shouldn't change just because there is a resource class now | |
| 14:23:55 | mriedem | or, | |
| 14:24:03 | mriedem | do you mean the custom resource class allocatoin isn't showing up for the instance? | |
| 14:24:35 | johnthetubaguy | I mean old instances don't have allocations for the new resource, which breaks the scheduling of new flavors that use the resource class | |
| 14:24:49 | mriedem | so instance A had vcpu/ram/disk allocations before the node.resource_class was set, then you set node.resource_class = baremetal and now you expect to see a 'baremetal' allocation for instance A in placement | |
| 14:25:15 | johnthetubaguy | that is what I expected, yes | |
| 14:25:23 | johnthetubaguy | (clearly incorrectly) | |
| 14:25:30 | mriedem | that's probably because the RT isn't reporting allocations anymore once all of your computes are pike | |
| 14:25:31 | mriedem | sucka! | |
| 14:25:36 | johnthetubaguy | yup | |
| 14:25:48 | cdent | more importantly, with that expectation not met, scheduling breaks, right? | |
| 14:25:55 | johnthetubaguy | cdent: +1 | |
| 14:26:05 | edleafe | johnthetubaguy: this came up last cycle | |
| 14:26:06 | gibi | efried: I'm +1 on pushing that notification backport through | |
| 14:26:08 | mriedem | in the before times, the update_available_resource periodic task would update the allocations for the instances running on that node | |
| 14:26:13 | mriedem | until we squashed that | |
| 14:26:25 | johnthetubaguy | mriedem: yeah, that is what I was thinking | |
| 14:26:30 | mriedem | efried: i can't +W my own backport | |
| 14:26:52 | edleafe | johnthetubaguy: unfortunately, it was in the middle of a bunch of other discussions about the ironic transition to custom RCs | |
| 14:27:15 | cdent | edleafe: yeah, I seem to recall dtantsur|brb had something to say about this. since he is brb, maybe he’ll brb | |
| 14:27:46 | mriedem | johnthetubaguy: ok so the scheduling issue is a new instance request can try to claim the 'baremetal' resource on node A even though instance A is already using it | |
| 14:27:51 | mriedem | which causes a scheduler failure yes? | |
| 14:27:54 | efried | mriedem Right, and I'm led to understand the set of cores on stable is not the same as master, but I don't really understand how it works. Guess I'm asking what needs to be done to get it in? | |
| 14:27:58 | johnthetubaguy | mriedem: yes | |
| 14:28:14 | mriedem | johnthetubaguy: ok yeah i seem to remember this coming up too....maybe dansmith remembers | |
| 14:28:53 | mriedem | efried: yes different core group on stable https://review.openstack.org/#/admin/groups/540,members | |
| 14:28:56 | mriedem | b/c different rules | |
| 14:29:04 | dansmith | um what | |
| 14:29:10 | mriedem | efried: https://docs.openstack.org/project-team-guide/stable-branches.html | |
| 14:29:13 | dansmith | johnthetubaguy: mriedem I'm not sure what you're talking about | |
| 14:29:22 | dansmith | I'm on a call right now so I'm a bit distracted | |
| 14:29:38 | mriedem | efried: we could use more stable cores, so if that's something you're interested in helping with, please dig in, get to know the rules, and do reviews | |
| 14:29:47 | efried | mriedem ack | |
| 14:30:18 | mriedem | basically means don't backport features, or backward incompatible changes, all things start on master and go backward, and there are support phases for what's appropriate to backport based on severity | |
| 14:30:56 | mriedem | johnthetubaguy: ok, got it - want to start by reporting a bug? | |
| 14:31:19 | johnthetubaguy | mriedem: yeah, will do, I was hoping I just miss-read the docs | |
| 14:33:11 | mriedem | johnthetubaguy: https://review.openstack.org/#/c/492964/ is the change i was thinking of, | |
| 14:33:14 | mriedem | slightly different though | |
| 14:33:21 | mriedem | that's about reporting inventory, not allocations | |
| 14:33:29 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extract instance allocation removal code https://review.openstack.org/513041 | |
| 14:33:38 | gibi | mriedem: fixed your nit in ^^ | |
| 14:35:15 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: PCI NUMA Policies https://review.openstack.org/361140 | |
| 14:35:23 | efried | stephenfin ^ | |
| 14:35:36 | efried | I basically just took that nonsensical chunk out, and fixed up those couple of words. | |
| 14:36:58 | openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec for API extensions policy removal https://review.openstack.org/508101 | |
| 14:38:06 | mriedem | johnthetubaguy: ok so normally, at least if you have 1 ocata compute, the update_available_resource periodic would run, that would get available nodes, which would refresh nodes from ironic, and that refresh does the resoure class / flavor migration thing, | |
| 14:38:27 | mriedem | and then as part of the update_available_resource periodic, the RT would update allocations for each instance on the node (if you have at least 1 ocata compute) | |
| 14:39:15 | mriedem | we don't really have a hook between the virt driver and the RT to say if the allocations should be updated, | |
| 14:39:35 | stephenfin | efried: Looks good to me. Thank you :) | |
| 14:39:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Moving more utils to ServerResourceAllocationTestBase https://review.openstack.org/499539 | |
| 14:39:37 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test resource allocation during soft delete https://review.openstack.org/495159 | |
| 14:39:37 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: factor out compute service start in ServerMovingTest https://review.openstack.org/503037 | |
| 14:39:56 | stephenfin | dansmith: Could you stick https://review.openstack.org/361140 on your review backlog? | |
| 14:40:08 | stephenfin | sean-k-mooney, cdent: Ye might like to look at that again too ^ | |
| 14:40:11 | mriedem | johnthetubaguy: random thoughts: we could have the ironic driver set a flag when a flavor was migrated and the RT calls into the driver to see if the flag was set and allocations should be forcefully updated (kind of gross and maybe racy), | |
| 14:40:30 | mriedem | johnthetubaguy: we could have the driver update allocations on it's own...also kind of gross but at least very specific case | |
| 14:40:33 | dansmith | stephenfin: I can, but the queue is long and I'm not sure I'm the best person to review that | |
| 14:40:43 | dansmith | I guess I don't really know who is though | |
| 14:41:01 | johnthetubaguy | mriedem: yeah, being part of the instance flavor migration doesn't seem totally crazy | |
| 14:41:17 | stephenfin | dansmith: That's the problem :( jaypipes would be the best person, but he has his hands full with nested-resource-providers | |
| 14:41:30 | dansmith | stephenfin: he has his hands full with being a slacker | |
| 14:41:35 | stephenfin | dansmith: You could just approve it and assume it's perfect? :P | |
| 14:41:42 | johnthetubaguy | mriedem: I attempted to write it all up here: https://bugs.launchpad.net/nova/+bug/1724589 | |
| 14:41:43 | openstack | Launchpad bug 1724589 in OpenStack Compute (nova) "Unable to transition to Ironic Node Resource Classes in Pike" [High,New] | |