| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-01 | |||
| 16:22:10 | sean-k-mooney | mriedem: i was just debating if we shoudl triage them as incomplete or wontfix unless a different way to reporduce can be provided | |
| 16:23:48 | mriedem | i marked one of them as opinion | |
| 16:25:01 | johnthetubaguy | FWIW, I always wanted to be able to "timeout" tasks to try and catch that pending forever case. They caused me endless pain at Rackspace (I think mostly in the migrate/resize code path). The difference was they were more expected / user triggered errors. | |
| 16:25:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add --before to nova-manage db archive_deleted_rows https://review.openstack.org/556751 | |
| 16:26:07 | mriedem | johnthetubaguy: how much of that was resolved with service user tokens though? | |
| 16:26:14 | mriedem | or the long_rpc_timeout we have since rocky | |
| 16:26:23 | mriedem | which we're using now in the live migration flows that do rpc calls | |
| 16:27:17 | johnthetubaguy | mriedem: yeah, I saw that go in. Although most of those cases it went to Error (eventually) when it didn't have to. | |
| 16:27:34 | mriedem | that's a different bug then | |
| 16:29:05 | sean-k-mooney | johnthetubaguy: i have seen this happen with rabbitmq restarte in the past where when perstiency was disabled on instance build and a few other cases. | |
| 16:29:40 | sean-k-mooney | i never really considerd that a nova bug however because i cased the issue by restarting rabbit | |
| 16:31:04 | sean-k-mooney | johnthetubaguy: but ya its proably more complcated then jsut set to error after x time as some request could still be in flight | |
| 16:36:00 | melwitt | mriedem: yes, it completely slipped my mind :( and I'm not done going through the entire list of the schedule yet | |
| 16:40:12 | mriedem | melwitt: i added several sessions in there based on my schedule | |
| 16:41:09 | melwitt | ok, thank you. that's helpful | |
| 16:47:07 | johnthetubaguy | sean-k-mooney: yeah, its hard to get right | |
| 16:56:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: API microversion bump for handling-down-cell https://review.openstack.org/591657 | |
| 16:56:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810 | |
| 16:56:57 | mriedem | tssurya: ^ | |
| 16:57:10 | tssurya | looking, thanks | |
| 17:04:43 | tssurya | mriedem: okay I am going to write the tests here https://review.openstack.org/#/c/591657/12/nova/tests/functional/api_sample_tests/test_servers.py based on your fixture | |
| 17:13:08 | openstackgerrit | Merged openstack/nova master: Make ResourceTracker.tracked_instances a set https://review.openstack.org/608781 | |
| 17:18:31 | dansmith | mriedem: tssurya I'm explaining the host-status concept to someone right now, and why an instance state doesn't go to STOPPED just because the compute node is down | |
| 17:18:47 | dansmith | mriedem: I wonder if it would make sense to integrate the use of the UNKNOWN state we're adding here with that feature, | |
| 17:18:59 | dansmith | so that in the same microversion, instances with a down host show up as UNKNOWN as well | |
| 17:25:59 | tssurya | dansmith: you mean you want to add a new "UNKNOWN" vm_state ? | |
| 17:26:20 | dansmith | tssurya: you're already doing that from the external view right now | |
| 17:26:25 | tssurya | yea | |
| 17:26:39 | dansmith | we would do a similar thing for real instances we can look up just fine, but which have down hosts | |
| 17:27:05 | dansmith | the only problem would be that right now UNKNOWN means "the rest of the instance details aren't there" which would be slightly more ambiguous in this case | |
| 17:28:27 | tssurya | hmm, makes sense to make the instance state UNKNOWN since we don't know the host state, I mean I guess "UNKNOWN" could mean unknown details/state right ? | |
| 17:28:54 | belmoreira | dansmith mriedem should placement/nova issues be discussed here or in placement channel | |
| 17:28:57 | cfriesen | dansmith: for what it's worth, in our environment if a compute node goes down an external entity sets all of the instances to the "error" state, until they were automatically recovered. | |
| 17:29:30 | dansmith | that seems like an improper use of the error state to me | |
| 17:29:48 | dansmith | not to mention that nova on its own won't know whether they're still up and fine or not | |
| 17:29:50 | cfriesen | if the host is "down", then we fence it off and force a reboot. those instances are guaranteed to be toast | |
| 17:29:53 | dansmith | which is why we don't call them "stopped" | |
| 17:30:15 | dansmith | cfriesen: okay, well, that's better in that case but vanilla nova can't do or know that | |
| 17:30:42 | cfriesen | agreed, nova itself can't know the bigger picture | |
| 17:30:47 | dansmith | belmoreira: depends on what it is.. if it's integration issues then probably here | |
| 17:32:03 | belmoreira | is the increase of the number of requests to placement | |
| 17:32:16 | cfriesen | dansmith: although, if an external entity uses the nova API to tell nova that the compute node is "down", it's supposed to have already fenced off the node to prevent instances from (eg) talking to volumes. | |
| 17:32:44 | belmoreira | have a look into: https://docs.google.com/document/d/1d5k1hA3DbGmMyJbXdVcekR12gyrFTaj_tJdFwdQy-8E/edit?usp=sharing | |
| 17:32:49 | cfriesen | otherwise you could evacuate and then have two copies of an instance trying to access the same cinder volume | |
| 17:33:17 | dansmith | cfriesen: yeah, true.. I guess I just prefer something less overloaded like UNKNOWN than saying it's stopped or error | |
| 17:33:18 | belmoreira | this is the number of requests to placement when compute-nodes get upgrade to Rocky | |
| 17:33:28 | dansmith | efried: ^ | |
| 17:33:39 | efried | how far back am I reading? | |
| 17:33:44 | dansmith | efried: one line | |
| 17:33:52 | dansmith | and the url he posted | |
| 17:35:08 | dansmith | belmoreira: is it really increasing, or is that you bringing nodes on over time? | |
| 17:36:00 | belmoreira | the increase of requests shows the compute nodes being upgraded over time (queens -> rocky) | |
| 17:36:10 | dansmith | okay | |
| 17:36:15 | dansmith | (ouch) | |
| 17:36:22 | efried | What's happening at that cat-head bump? | |
| 17:36:31 | efried | or possibly batman | |
| 17:36:34 | dansmith | online migrations? | |
| 17:37:22 | efried | those look like trait requests | |
| 17:37:34 | efried | if I'm reading this right. | |
| 17:37:57 | belmoreira | no, it must be a cell that upgraded and then stopped nova-compute | |
| 17:38:31 | belmoreira | so, in the second graph we can see all the new requests | |
| 17:39:25 | belmoreira | UUID/trais ; ?in_tree; UUID/aggregates; ... | |
| 17:40:54 | efried | right, so it looks to me like, before rocky, we weren't calling ?in_tree, UUID/aggregates, or ?member_of at all. Which makes sense. | |
| 17:41:27 | belmoreira | yes, and this seems to be the reason of the increase of requests | |
| 17:42:10 | efried | but also increased number of requests for inventories. | |
| 17:42:16 | belmoreira | but is a huge increase. Just added another graph with the response time of my placement infrastructure | |
| 17:42:56 | efried | I have to say, this isn't all that surprising. | |
| 17:44:39 | efried | although, hm, I would have expected this jump in queens | |
| 17:44:57 | efried | belmoreira: Was this an upgrade from queens, or from earlier? | |
| 17:45:33 | belmoreira | efried from queens | |
| 17:47:46 | belmoreira | I could handle it creating more placement nodes (x3). But looks too much... | |
| 17:48:15 | efried | belmoreira: Can you give me a sense of what this timeline represents? At what point are all the upgrades done and the cloud in stable state? | |
| 17:50:34 | belmoreira | efried the nova/placement control plane was upgraded between 8:00 and 9:00. ~12:00 the compute nodes started to upgrade (this takes 24h for all of them upgrade) | |
| 17:51:28 | belmoreira | at 12:00 (today) almost all compute nodes are in Rocky. | |
| 17:51:58 | efried | belmoreira: So where it tails off at the end, that's when the upgrades are pretty much done? | |
| 17:52:02 | belmoreira | the load graphs shows when I added more capacity for placement | |
| 17:52:13 | efried | Do you have a graph for what it looks like right now? | |
| 17:52:30 | efried | I'm just wondering if it's a massive spike during upgrade, but then it evens back out afterward. | |
| 17:52:35 | efried | in which case... yeah | |
| 17:54:45 | belmoreira | efried I'm getting a new graph from now | |
| 17:55:21 | efried | though once again, I wouldn't have expected e.g. ?in_tree to be zero at queens. That should be happening every periodic. | |
| 17:57:42 | dansmith | efried: you mean you think it's startup storm? | |
| 17:57:50 | dansmith | so every time they reboot computes they'll get this? | |
| 17:58:10 | efried | dansmith: If you reboot a thousand computes... | |
| 17:58:26 | efried | dansmith: I just wanted to understand *whether* it was startup storm. | |
| 17:58:27 | dansmith | right but presumably they're not rebooting them every second | |
| 17:58:31 | dansmith | ack | |
| 17:58:39 | efried | Whether it goes back to normal once everything stabilizes | |
| 17:58:41 | openstackgerrit | Merged openstack/nova stable/rocky: De-dupe subnet IDs when calling neutron /subnets API https://review.openstack.org/608336 | |
| 17:58:43 | dansmith | they also know what upgrades look like | |
| 17:58:52 | efried | (I don't) | |
| 17:58:56 | dansmith | so the fact that they're concerned probably means something | |
| 17:59:15 | efried | Heh, I'm not tryng to weasel out of anything. Just trying to grok the problem domain. | |
| 17:59:24 | dansmith | no, I know | |
| 17:59:28 | dansmith | just sain' | |
| 18:00:06 | dansmith | even if we just made the reboot storm a lot worse, that's something we probably need to look at | |
| 18:00:12 | belmoreira | efried a new graph from now | |
| 18:00:43 | belmoreira | it is flat at the end. That is the total number of requests that we handle now | |
| 18:01:18 | efried | dansmith: Can you sanity-check me on this, though - the _refresh_associations code is in queens, including _ensure_resource_provider invoking _get_provider_in_tree, which is what invokes the ?in_tree URI. | |
| 18:01:40 | efried | the mystery being, why would they be seeing zero ?in_tree calls right before the upgrade? | |