| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-01 | |||
| 17:29:48 | dansmith | not to mention that nova on its own won't know whether they're still up and fine or not | |
| 17:29:50 | cfriesen | if the host is "down", then we fence it off and force a reboot. those instances are guaranteed to be toast | |
| 17:29:53 | dansmith | which is why we don't call them "stopped" | |
| 17:30:15 | dansmith | cfriesen: okay, well, that's better in that case but vanilla nova can't do or know that | |
| 17:30:42 | cfriesen | agreed, nova itself can't know the bigger picture | |
| 17:30:47 | dansmith | belmoreira: depends on what it is.. if it's integration issues then probably here | |
| 17:32:03 | belmoreira | is the increase of the number of requests to placement | |
| 17:32:16 | cfriesen | dansmith: although, if an external entity uses the nova API to tell nova that the compute node is "down", it's supposed to have already fenced off the node to prevent instances from (eg) talking to volumes. | |
| 17:32:44 | belmoreira | have a look into: https://docs.google.com/document/d/1d5k1hA3DbGmMyJbXdVcekR12gyrFTaj_tJdFwdQy-8E/edit?usp=sharing | |
| 17:32:49 | cfriesen | otherwise you could evacuate and then have two copies of an instance trying to access the same cinder volume | |
| 17:33:17 | dansmith | cfriesen: yeah, true.. I guess I just prefer something less overloaded like UNKNOWN than saying it's stopped or error | |
| 17:33:18 | belmoreira | this is the number of requests to placement when compute-nodes get upgrade to Rocky | |
| 17:33:28 | dansmith | efried: ^ | |
| 17:33:39 | efried | how far back am I reading? | |
| 17:33:44 | dansmith | efried: one line | |
| 17:33:52 | dansmith | and the url he posted | |
| 17:35:08 | dansmith | belmoreira: is it really increasing, or is that you bringing nodes on over time? | |
| 17:36:00 | belmoreira | the increase of requests shows the compute nodes being upgraded over time (queens -> rocky) | |
| 17:36:10 | dansmith | okay | |
| 17:36:15 | dansmith | (ouch) | |
| 17:36:22 | efried | What's happening at that cat-head bump? | |
| 17:36:31 | efried | or possibly batman | |
| 17:36:34 | dansmith | online migrations? | |
| 17:37:22 | efried | those look like trait requests | |
| 17:37:34 | efried | if I'm reading this right. | |
| 17:37:57 | belmoreira | no, it must be a cell that upgraded and then stopped nova-compute | |
| 17:38:31 | belmoreira | so, in the second graph we can see all the new requests | |
| 17:39:25 | belmoreira | UUID/trais ; ?in_tree; UUID/aggregates; ... | |
| 17:40:54 | efried | right, so it looks to me like, before rocky, we weren't calling ?in_tree, UUID/aggregates, or ?member_of at all. Which makes sense. | |
| 17:41:27 | belmoreira | yes, and this seems to be the reason of the increase of requests | |
| 17:42:10 | efried | but also increased number of requests for inventories. | |
| 17:42:16 | belmoreira | but is a huge increase. Just added another graph with the response time of my placement infrastructure | |
| 17:42:56 | efried | I have to say, this isn't all that surprising. | |
| 17:44:39 | efried | although, hm, I would have expected this jump in queens | |
| 17:44:57 | efried | belmoreira: Was this an upgrade from queens, or from earlier? | |
| 17:45:33 | belmoreira | efried from queens | |
| 17:47:46 | belmoreira | I could handle it creating more placement nodes (x3). But looks too much... | |
| 17:48:15 | efried | belmoreira: Can you give me a sense of what this timeline represents? At what point are all the upgrades done and the cloud in stable state? | |
| 17:50:34 | belmoreira | efried the nova/placement control plane was upgraded between 8:00 and 9:00. ~12:00 the compute nodes started to upgrade (this takes 24h for all of them upgrade) | |
| 17:51:28 | belmoreira | at 12:00 (today) almost all compute nodes are in Rocky. | |
| 17:51:58 | efried | belmoreira: So where it tails off at the end, that's when the upgrades are pretty much done? | |
| 17:52:02 | belmoreira | the load graphs shows when I added more capacity for placement | |
| 17:52:13 | efried | Do you have a graph for what it looks like right now? | |
| 17:52:30 | efried | I'm just wondering if it's a massive spike during upgrade, but then it evens back out afterward. | |
| 17:52:35 | efried | in which case... yeah | |
| 17:54:45 | belmoreira | efried I'm getting a new graph from now | |
| 17:55:21 | efried | though once again, I wouldn't have expected e.g. ?in_tree to be zero at queens. That should be happening every periodic. | |
| 17:57:42 | dansmith | efried: you mean you think it's startup storm? | |
| 17:57:50 | dansmith | so every time they reboot computes they'll get this? | |
| 17:58:10 | efried | dansmith: If you reboot a thousand computes... | |
| 17:58:26 | efried | dansmith: I just wanted to understand *whether* it was startup storm. | |
| 17:58:27 | dansmith | right but presumably they're not rebooting them every second | |
| 17:58:31 | dansmith | ack | |
| 17:58:39 | efried | Whether it goes back to normal once everything stabilizes | |
| 17:58:41 | openstackgerrit | Merged openstack/nova stable/rocky: De-dupe subnet IDs when calling neutron /subnets API https://review.openstack.org/608336 | |
| 17:58:43 | dansmith | they also know what upgrades look like | |
| 17:58:52 | efried | (I don't) | |
| 17:58:56 | dansmith | so the fact that they're concerned probably means something | |
| 17:59:15 | efried | Heh, I'm not tryng to weasel out of anything. Just trying to grok the problem domain. | |
| 17:59:24 | dansmith | no, I know | |
| 17:59:28 | dansmith | just sain' | |
| 18:00:06 | dansmith | even if we just made the reboot storm a lot worse, that's something we probably need to look at | |
| 18:00:12 | belmoreira | efried a new graph from now | |
| 18:00:43 | belmoreira | it is flat at the end. That is the total number of requests that we handle now | |
| 18:01:18 | efried | dansmith: Can you sanity-check me on this, though - the _refresh_associations code is in queens, including _ensure_resource_provider invoking _get_provider_in_tree, which is what invokes the ?in_tree URI. | |
| 18:01:40 | efried | the mystery being, why would they be seeing zero ?in_tree calls right before the upgrade? | |
| 18:01:42 | dansmith | I just headed into a meeting I have to pay attention to | |
| 18:04:34 | belmoreira | humm. tssurya just point out the "resource_provider_association_refresh" configuration that we had in queens we don't have it in rocky | |
| 18:05:15 | efried | mm, that'd explain a lot. Y'all added that to compensate for this kind of spike in placement traffic iirc | |
| 18:05:40 | belmoreira | efried that explains " I wouldn't have expected e.g. ?in_tree to be zero at queens" | |
| 18:05:55 | efried | yup | |
| 18:06:23 | mriedem | i thought you totally nuked resource_provider_association_refresh rather than just set it to a large value? | |
| 18:06:36 | efried | but also why all those things are zero before the upgrade and nonzero after. Like I was saying, I expect all this stuff to happen at the queens boundary, not rocky. | |
| 18:07:58 | belmoreira | in queens we patch it and set it to a very large number (to not run again). And I miss it now. My fault! | |
| 18:08:09 | efried | IOW I suspect that turning that you would have seen the same graphs simply by turning that switch off and leaving your nodes at queens | |
| 18:08:52 | belmoreira | but the number of requests we really impressive! meaning that is very difficult to keep this option in a large infrastructure | |
| 18:09:20 | efried | belmoreira: I don't disagree with that. | |
| 18:10:07 | mriedem | so by default, every compute (70K?) is refreshing inventory every 1 minute, and every 5 minutes it's also refreshing in_tree, aggregates and traits? | |
| 18:10:08 | efried | I would think moving it to a fairly generous interval and hoping your computes don't all hit that interval at the same time :) | |
| 18:10:20 | tssurya | mriedem: yea | |
| 18:10:35 | mriedem | and we do'nt use the aggregates stuff in compute yet at all from what i can tell | |
| 18:10:45 | mriedem | it was there for sharing providers which we don't support yet | |
| 18:10:51 | efried | well, didn't we start cloning host azs ? | |
| 18:10:58 | mriedem | that's in the API | |
| 18:11:18 | efried | but we're not using that in the scheduler yet? | |
| 18:11:43 | mriedem | the mirrored aggregates stuff? yes there are pre-request placement filters that rely on it (or something external doing the mirroring) | |
| 18:11:56 | mriedem | i'm not sure what that has to do with the cache / refresh for aggregates in all the computes | |
| 18:12:39 | mriedem | iow, i'm not sure what the cache in the compute buys us | |
| 18:12:45 | efried | yeah, I'm actually trying to think what we actually use the cache for at all... right. | |
| 18:13:50 | efried | cdent has been grousing for a while that we should just be able to make placement calls when we need 'em. | |
| 18:14:14 | sean-k-mooney | if we really wanted to make the storm less likely we could use a random prime ofset for the update | |
| 18:14:43 | efried | I was thinking it, but then you said it. | |
| 18:15:10 | mriedem | oslo already does something like that for periodics | |
| 18:15:28 | efried | Trying to think what it would take to rip out the cache completely. | |
| 18:15:38 | belmoreira | I'm changing this option, it will take ~2h to propagate. Will let you know the result | |
| 18:15:44 | efried | ack | |
| 18:16:00 | belmoreira | I have to leave now for some minutes. Thanks for all your help | |
| 18:16:22 | efried | o/ | |
| 18:17:34 | efried | mriedem: We use the cache data so the virt driver has the opportunity every periodic to update the provider tree. | |
| 18:18:19 | efried | mriedem: assuming stable placement, no _refresh'ing, we would be doing a helluva lot fewer calls | |