| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-01 | |||
| 16:56:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810 | |
| 16:56:57 | mriedem | tssurya: ^ | |
| 16:57:10 | tssurya | looking, thanks | |
| 17:04:43 | tssurya | mriedem: okay I am going to write the tests here https://review.openstack.org/#/c/591657/12/nova/tests/functional/api_sample_tests/test_servers.py based on your fixture | |
| 17:13:08 | openstackgerrit | Merged openstack/nova master: Make ResourceTracker.tracked_instances a set https://review.openstack.org/608781 | |
| 17:18:31 | dansmith | mriedem: tssurya I'm explaining the host-status concept to someone right now, and why an instance state doesn't go to STOPPED just because the compute node is down | |
| 17:18:47 | dansmith | mriedem: I wonder if it would make sense to integrate the use of the UNKNOWN state we're adding here with that feature, | |
| 17:18:59 | dansmith | so that in the same microversion, instances with a down host show up as UNKNOWN as well | |
| 17:25:59 | tssurya | dansmith: you mean you want to add a new "UNKNOWN" vm_state ? | |
| 17:26:20 | dansmith | tssurya: you're already doing that from the external view right now | |
| 17:26:25 | tssurya | yea | |
| 17:26:39 | dansmith | we would do a similar thing for real instances we can look up just fine, but which have down hosts | |
| 17:27:05 | dansmith | the only problem would be that right now UNKNOWN means "the rest of the instance details aren't there" which would be slightly more ambiguous in this case | |
| 17:28:27 | tssurya | hmm, makes sense to make the instance state UNKNOWN since we don't know the host state, I mean I guess "UNKNOWN" could mean unknown details/state right ? | |
| 17:28:54 | belmoreira | dansmith mriedem should placement/nova issues be discussed here or in placement channel | |
| 17:28:57 | cfriesen | dansmith: for what it's worth, in our environment if a compute node goes down an external entity sets all of the instances to the "error" state, until they were automatically recovered. | |
| 17:29:30 | dansmith | that seems like an improper use of the error state to me | |
| 17:29:48 | dansmith | not to mention that nova on its own won't know whether they're still up and fine or not | |
| 17:29:50 | cfriesen | if the host is "down", then we fence it off and force a reboot. those instances are guaranteed to be toast | |
| 17:29:53 | dansmith | which is why we don't call them "stopped" | |
| 17:30:15 | dansmith | cfriesen: okay, well, that's better in that case but vanilla nova can't do or know that | |
| 17:30:42 | cfriesen | agreed, nova itself can't know the bigger picture | |
| 17:30:47 | dansmith | belmoreira: depends on what it is.. if it's integration issues then probably here | |
| 17:32:03 | belmoreira | is the increase of the number of requests to placement | |
| 17:32:16 | cfriesen | dansmith: although, if an external entity uses the nova API to tell nova that the compute node is "down", it's supposed to have already fenced off the node to prevent instances from (eg) talking to volumes. | |
| 17:32:44 | belmoreira | have a look into: https://docs.google.com/document/d/1d5k1hA3DbGmMyJbXdVcekR12gyrFTaj_tJdFwdQy-8E/edit?usp=sharing | |
| 17:32:49 | cfriesen | otherwise you could evacuate and then have two copies of an instance trying to access the same cinder volume | |
| 17:33:17 | dansmith | cfriesen: yeah, true.. I guess I just prefer something less overloaded like UNKNOWN than saying it's stopped or error | |
| 17:33:18 | belmoreira | this is the number of requests to placement when compute-nodes get upgrade to Rocky | |
| 17:33:28 | dansmith | efried: ^ | |
| 17:33:39 | efried | how far back am I reading? | |
| 17:33:44 | dansmith | efried: one line | |
| 17:33:52 | dansmith | and the url he posted | |
| 17:35:08 | dansmith | belmoreira: is it really increasing, or is that you bringing nodes on over time? | |
| 17:36:00 | belmoreira | the increase of requests shows the compute nodes being upgraded over time (queens -> rocky) | |
| 17:36:10 | dansmith | okay | |
| 17:36:15 | dansmith | (ouch) | |
| 17:36:22 | efried | What's happening at that cat-head bump? | |
| 17:36:31 | efried | or possibly batman | |
| 17:36:34 | dansmith | online migrations? | |
| 17:37:22 | efried | those look like trait requests | |
| 17:37:34 | efried | if I'm reading this right. | |
| 17:37:57 | belmoreira | no, it must be a cell that upgraded and then stopped nova-compute | |
| 17:38:31 | belmoreira | so, in the second graph we can see all the new requests | |
| 17:39:25 | belmoreira | UUID/trais ; ?in_tree; UUID/aggregates; ... | |
| 17:40:54 | efried | right, so it looks to me like, before rocky, we weren't calling ?in_tree, UUID/aggregates, or ?member_of at all. Which makes sense. | |
| 17:41:27 | belmoreira | yes, and this seems to be the reason of the increase of requests | |
| 17:42:10 | efried | but also increased number of requests for inventories. | |
| 17:42:16 | belmoreira | but is a huge increase. Just added another graph with the response time of my placement infrastructure | |
| 17:42:56 | efried | I have to say, this isn't all that surprising. | |
| 17:44:39 | efried | although, hm, I would have expected this jump in queens | |
| 17:44:57 | efried | belmoreira: Was this an upgrade from queens, or from earlier? | |
| 17:45:33 | belmoreira | efried from queens | |
| 17:47:46 | belmoreira | I could handle it creating more placement nodes (x3). But looks too much... | |
| 17:48:15 | efried | belmoreira: Can you give me a sense of what this timeline represents? At what point are all the upgrades done and the cloud in stable state? | |
| 17:50:34 | belmoreira | efried the nova/placement control plane was upgraded between 8:00 and 9:00. ~12:00 the compute nodes started to upgrade (this takes 24h for all of them upgrade) | |
| 17:51:28 | belmoreira | at 12:00 (today) almost all compute nodes are in Rocky. | |
| 17:51:58 | efried | belmoreira: So where it tails off at the end, that's when the upgrades are pretty much done? | |
| 17:52:02 | belmoreira | the load graphs shows when I added more capacity for placement | |
| 17:52:13 | efried | Do you have a graph for what it looks like right now? | |
| 17:52:30 | efried | I'm just wondering if it's a massive spike during upgrade, but then it evens back out afterward. | |
| 17:52:35 | efried | in which case... yeah | |
| 17:54:45 | belmoreira | efried I'm getting a new graph from now | |
| 17:55:21 | efried | though once again, I wouldn't have expected e.g. ?in_tree to be zero at queens. That should be happening every periodic. | |
| 17:57:42 | dansmith | efried: you mean you think it's startup storm? | |
| 17:57:50 | dansmith | so every time they reboot computes they'll get this? | |
| 17:58:10 | efried | dansmith: If you reboot a thousand computes... | |
| 17:58:26 | efried | dansmith: I just wanted to understand *whether* it was startup storm. | |
| 17:58:27 | dansmith | right but presumably they're not rebooting them every second | |
| 17:58:31 | dansmith | ack | |
| 17:58:39 | efried | Whether it goes back to normal once everything stabilizes | |
| 17:58:41 | openstackgerrit | Merged openstack/nova stable/rocky: De-dupe subnet IDs when calling neutron /subnets API https://review.openstack.org/608336 | |
| 17:58:43 | dansmith | they also know what upgrades look like | |
| 17:58:52 | efried | (I don't) | |
| 17:58:56 | dansmith | so the fact that they're concerned probably means something | |
| 17:59:15 | efried | Heh, I'm not tryng to weasel out of anything. Just trying to grok the problem domain. | |
| 17:59:24 | dansmith | no, I know | |
| 17:59:28 | dansmith | just sain' | |
| 18:00:06 | dansmith | even if we just made the reboot storm a lot worse, that's something we probably need to look at | |
| 18:00:12 | belmoreira | efried a new graph from now | |
| 18:00:43 | belmoreira | it is flat at the end. That is the total number of requests that we handle now | |
| 18:01:18 | efried | dansmith: Can you sanity-check me on this, though - the _refresh_associations code is in queens, including _ensure_resource_provider invoking _get_provider_in_tree, which is what invokes the ?in_tree URI. | |
| 18:01:40 | efried | the mystery being, why would they be seeing zero ?in_tree calls right before the upgrade? | |
| 18:01:42 | dansmith | I just headed into a meeting I have to pay attention to | |
| 18:04:34 | belmoreira | humm. tssurya just point out the "resource_provider_association_refresh" configuration that we had in queens we don't have it in rocky | |
| 18:05:15 | efried | mm, that'd explain a lot. Y'all added that to compensate for this kind of spike in placement traffic iirc | |
| 18:05:40 | belmoreira | efried that explains " I wouldn't have expected e.g. ?in_tree to be zero at queens" | |
| 18:05:55 | efried | yup | |
| 18:06:23 | mriedem | i thought you totally nuked resource_provider_association_refresh rather than just set it to a large value? | |
| 18:06:36 | efried | but also why all those things are zero before the upgrade and nonzero after. Like I was saying, I expect all this stuff to happen at the queens boundary, not rocky. | |
| 18:07:58 | belmoreira | in queens we patch it and set it to a very large number (to not run again). And I miss it now. My fault! | |
| 18:08:09 | efried | IOW I suspect that turning that you would have seen the same graphs simply by turning that switch off and leaving your nodes at queens | |
| 18:08:52 | belmoreira | but the number of requests we really impressive! meaning that is very difficult to keep this option in a large infrastructure | |
| 18:09:20 | efried | belmoreira: I don't disagree with that. | |
| 18:10:07 | mriedem | so by default, every compute (70K?) is refreshing inventory every 1 minute, and every 5 minutes it's also refreshing in_tree, aggregates and traits? | |
| 18:10:08 | efried | I would think moving it to a fairly generous interval and hoping your computes don't all hit that interval at the same time :) | |
| 18:10:20 | tssurya | mriedem: yea | |
| 18:10:35 | mriedem | and we do'nt use the aggregates stuff in compute yet at all from what i can tell | |
| 18:10:45 | mriedem | it was there for sharing providers which we don't support yet | |
| 18:10:51 | efried | well, didn't we start cloning host azs ? | |