| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-14 | |||
| 14:29:13 | sean-k-mooney | jangutter: anyway got to run | |
| 14:29:18 | cdent | ✔ | |
| 14:29:25 | jangutter | sean-k-mooney: thanks very much again! | |
| 14:31:27 | leakypipes | cdent, figleaf, mriedem, superdan: so... something I didn't quite anticipate (or rather, was ignoring and hoping we wouldn't run into it.) is resize to same host. With the code I added, we're getting NoValidHosts returning because the resize to same host operations are not taking resource claiming into account. | |
| 14:31:48 | leakypipes | cdent, figleaf, mriedem, superdan: I'm wondering if we need to just disable the scheduler claiming thing for move operations. | |
| 14:32:11 | cdent | oh yeah. I _think_ that’s one of the ideas we talked about in the past | |
| 14:32:48 | cdent | Is a bit annoying that yet another small piece of state needs to travel around, but life is hard | |
| 14:32:59 | figleaf | leakypipes: yeah, the scheduler is just used in those cases to ensure that the new host satisfies the original request | |
| 14:33:43 | leakypipes | figleaf: yeah. | |
| 14:33:48 | figleaf | leakypipes: but then again, why would that make things easier? | |
| 14:34:14 | leakypipes | figleaf: not sure it would... just brainfarting right now :) | |
| 14:34:25 | leakypipes | figleaf: investigating these few tempest failures. | |
| 14:34:31 | figleaf | leakypipes: ah | |
| 14:34:52 | figleaf | It's always better to have a single code path if possible | |
| 14:34:58 | openstackgerrit | Andrey Volkov proposed openstack/nova master: PoC: Select PCI devices with distinct tag values https://review.openstack.org/448008 | |
| 14:36:03 | cdent | if we had perfect memory we could recall all the conversations we had about this particular topic because it sure seems like we had a plan (short and long term) but my brain is grasping at nothing | |
| 14:36:29 | figleaf | cdent: I think the problem is that we have had several plans :) | |
| 14:36:41 | figleaf | Hard to keep them distinct | |
| 14:36:57 | cdent | I thinkin back all the way to early 2016 | |
| 14:37:10 | cdent | lost to the sands of time | |
| 14:37:18 | leakypipes | cdent: you mean the Crustacean period of OpenStack? | |
| 14:37:29 | cdent | wasn’t that the precambrian? | |
| 14:37:34 | leakypipes | cdent: lemme get my archeological shovel handy | |
| 14:38:25 | cdent | no, couldn’t have been precambrian, because big tent would have been in the cambrian explosion | |
| 14:38:44 | mriedem | resize to same host shouldn't be double claimed, as we already claimed it once | |
| 14:38:46 | mriedem | same with rebuild | |
| 14:38:57 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Add support for VIFPortProfileOVSRepresentor https://review.openstack.org/483921 | |
| 14:38:59 | mriedem | you have to consider rebuild to same host vs evacuate to another host - where evacuate is going to claim on the other host | |
| 14:39:09 | mriedem | where rebuild to same host is not going to claim again on the same host | |
| 14:39:35 | cdent | mriedem: we can’t double claim can we (unless we explicitily create a single claim with double values) because consumer uuids stay the same? | |
| 14:39:42 | cdent | (or is that not true) | |
| 14:40:05 | cdent | /o\ | |
| 14:40:20 | mriedem | i mean, in the compute manager today, it does a NopClaim if it's on the same host | |
| 14:41:43 | cdent | if anyone has any things they want to remind me of for today’s rp update, send me some mail, I’m going to go hime and write it | |
| 14:41:50 | cdent | home | |
| 14:41:58 | leakypipes | mriedem: well, not even rebuild/evacuate. it's also any move operation. because when claiming for the instance, we will end up overwriting the allocations for that instance_uuid. | |
| 14:42:49 | openstackgerrit | Jan Gutter proposed openstack/nova master: Netronome SmartNIC Enablement https://review.openstack.org/483459 | |
| 14:42:57 | mriedem | leakypipes: aren't the allocations for the instance + some RP? | |
| 14:42:58 | leakypipes | mriedem: which means that the resources will be claimed against the destination host resource provider and no longer on the source host as soon as scheduling happens. | |
| 14:43:25 | leakypipes | mriedem: no, an allocation is a set of resource consumed on one or more resource providers for a specific consumer. | |
| 14:43:36 | mriedem | sure, one or more | |
| 14:43:49 | mriedem | until we move, aren't we consuming on 2 RPs? | |
| 14:44:01 | mriedem | there is the allocation we're consuming on the source RP node, and the claim for the dest RP node | |
| 14:44:25 | leakypipes | mriedem: no. the key is the consumer UUID. so we overwrite all allocations for the consumer when we call PUT /allocations/{consumer_uuid} | |
| 14:44:28 | figleaf | mriedem: a consumer can only consume one set of allocations at a time | |
| 14:44:52 | mriedem | so....if it's still one or more RPs, | |
| 14:44:55 | figleaf | leakypipes: we would have to combine the two sets and PUT that | |
| 14:44:55 | mriedem | i don't see the problem, | |
| 14:45:05 | leakypipes | figleaf: right. so as soon as the scheduler claims resources against the newly-selected destination host for a move operation, the old allocations for the original source host are replaced. | |
| 14:45:14 | mriedem | we have an allocaiton for the instance on source RP node, and then we update that allocation to include the dest RP node | |
| 14:45:18 | leakypipes | figleaf: yeah, that's about all I can think to do. | |
| 14:45:35 | leakypipes | figleaf: which means the scheduler is going to need to know whether this is a move operation. | |
| 14:45:39 | mriedem | right i'm thinking the same, worded differently | |
| 14:45:41 | figleaf | leakypipes: or add a PATCH resource | |
| 14:45:49 | figleaf | :( | |
| 14:46:07 | leakypipes | figleaf: potentially, but the scheduler would still need to know it's a move operation, right? | |
| 14:46:16 | figleaf | yeah | |
| 14:46:38 | mriedem | i was going to send this to cdent https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2779 | |
| 14:46:49 | mriedem | but ^ is what i was talking about with the nopclaim for rebuild on same host | |
| 14:46:58 | leakypipes | figleaf, mriedem: I think the "safest" thing to do is just disable claims in the scheduler for all move operations. so we'd just pass an indicator in the RequestSpec that this is a move operation and don't do any claiming if it is. | |
| 14:47:16 | mriedem | and then smartify it in queens? | |
| 14:47:20 | mriedem | because that's a short term hack right? | |
| 14:47:21 | leakypipes | yeah | |
| 14:47:36 | leakypipes | smartify it == the solution figleaf said above. | |
| 14:47:42 | figleaf | mriedem: I like smartifying stuff | |
| 14:47:57 | mriedem | can't we figure out if it's a move by checking if there are already existing allocations for that consumer? | |
| 14:48:12 | mriedem | or does that tread on retry territory | |
| 14:48:38 | leakypipes | mriedem: yes, but that would mean another set of HTTP communications for each scheduler request. | |
| 14:48:51 | leakypipes | mriedem: and we are already calling GET /allocation_candidates once per request. | |
| 14:48:53 | mriedem | yeah, so let's not do that then | |
| 14:49:11 | mriedem | if reqspec.movify: | |
| 14:49:12 | figleaf | leakypipes: true, but it might not mean dirtying as much code now | |
| 14:49:17 | melwitt | superdan: do you know if instance.old_flavor is a thing I can rely on to be there for an instance that's in VERIFY_RESIZE state? resized but not confirmed nor reverted? | |
| 14:49:27 | leakypipes | mriedem: especially since it's easy enough for the nova-api or conductor to add a field to the RequestSpec saying it's a move operation. | |
| 14:49:32 | mriedem | figleaf: it does mean a perf hit though | |
| 14:49:38 | mriedem | i'd rather pass a flag than take a perf hit | |
| 14:49:42 | mriedem | for ALL requests | |
| 14:49:43 | superdan | melwitt: yep, should be find if it's in any resize-related state I think | |
| 14:49:49 | melwitt | yesss | |
| 14:49:56 | figleaf | mriedem: yes, but small, but it will be smartified later! | |
| 14:50:13 | mriedem | figleaf: how do you know it would be small? | |
| 14:50:28 | leakypipes | mriedem: but do we have all that information during scheduling? I don't think the old flavor stuff is in the RequestSpec is it? | |
| 14:50:35 | figleaf | mriedem: GET /allocations is pretty lightweight | |
| 14:50:43 | mriedem | multicreate request with 100 instances for a new boot, would hit all of these 'do i already have allocations' requests for nothing | |
| 14:51:06 | mriedem | leakypipes: you'd put it on the request spec in conductor i thought | |
| 14:51:09 | mriedem | like you just said | |
| 14:51:17 | mriedem | migrate/resize/rebuild all go through conductor | |
| 14:51:23 | mriedem | with a reqspec | |
| 14:51:29 | leakypipes | mriedem: ah, sorry, I thought you were saying we should query that information for the instance within the scheduler. | |
| 14:51:45 | mriedem | leakypipes: that was an idea to avoid a flag being passed in on the reqsspec | |
| 14:51:51 | mriedem | but as noted it's extra overhead for perf | |
| 14:51:55 | leakypipes | right | |
| 14:51:57 | mriedem | which i don't tink we want long-term | |
| 14:52:20 | leakypipes | mriedem, figleaf: a simple flag "is_move" would be cool with me. I think that's actually pretty useful anyway. | |
| 14:52:32 | mriedem | seems that needs to be there anyway, long-term | |
| 14:52:41 | mriedem | if you don't want to make these GET /allocatoins calls in the shceduler to tell if you're moving | |
| 14:52:53 | figleaf | leakypipes: agreed | |
| 14:53:00 | mriedem | i don't care how lightweight GET /allocations is, we should avoid anything unnecessary within the scheduler as possible | |
| 14:53:11 | leakypipes | mriedem: FYI, since we don't support multiple instances doing resize/move, we'd never do >1 GET /allocations in the scheduler because we'd never call it if num_instances > 1. | |
| 14:53:20 | mriedem | we've already had reports of perf regression in server create times since mitaka | |