| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-19 | |||
| 18:30:00 | mriedem | not that you're setting up networking | |
| 18:31:41 | penick_ | The ocata api hosts running the conductor service are all idle | |
| 18:31:54 | dansmith | penick_: during a delete storm? | |
| 18:32:26 | penick_ | That I don't know yet, i'm going to set one up. I'll let you know what I come up with. | |
| 18:32:38 | dansmith | at idle I'd expect them to be idle ;) | |
| 18:33:15 | penick_ | But first our -juno- conductors are all exploding as they're being deployed. Gotta help the team tune those first | |
| 18:34:14 | penick_ | I set myself up for that :| | |
| 18:35:06 | melwitt | exploding meaning they won't run? or they are overloaded and you need to adjust workers? | |
| 18:38:16 | penick_ | Overloaded, we're only running 26 workers per node, on 3 nodes. Going to try adding more | |
| 18:40:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't update RT in _allocate_network https://review.openstack.org/513473 | |
| 18:51:26 | openstackgerrit | sean mooney proposed openstack/nova-specs master: Use neutron's new port binding API for live migration https://review.openstack.org/375580 | |
| 18:53:13 | melwitt | penick_: that seems like a lot to me but would have to consult dansmith. I don't remember what the sane starting point for number of workers per node is | |
| 18:54:25 | dansmith | workers per node is not a useful metric, it's workers per compute | |
| 18:54:57 | dansmith | 26*3 is a fair number of conductor workers, but if you have a thousand computes, then ... probably not insane | |
| 18:55:04 | dansmith | it also depends on what is happening | |
| 18:55:17 | dansmith | if everything is idle, that's probably more than enough | |
| 18:55:20 | melwitt | okay, sorry I think I was thinking of earlier when penick_ said 1-5 computes but if he's talking about juno then that would be 100s of computes | |
| 18:55:33 | melwitt | the 1-5 computes is the ocata testing | |
| 18:55:53 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: integrate ProviderTree to report client https://review.openstack.org/415921 | |
| 18:55:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: add nested resource providers https://review.openstack.org/377138 | |
| 18:55:54 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: allow filter providers in tree https://review.openstack.org/377215 | |
| 18:55:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: adds REST API for nested providers https://review.openstack.org/384807 | |
| 18:55:55 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: update client to set parent provider https://review.openstack.org/385693 | |
| 18:55:59 | jaypipes | dansmith: ^ | |
| 18:57:36 | dansmith | melwitt: right | |
| 19:13:52 | sahid | jaypipes: ci is green https://review.openstack.org/#/c/485522/ | |
| 19:14:31 | sahid | btw mriedem, did you take descison for this: https://review.openstack.org/#/c/511188/ ? | |
| 19:21:42 | mriedem | sahid: no | |
| 19:21:47 | mriedem | it's not on my list | |
| 19:24:02 | sahid | mriedem: thanks | |
| 19:31:15 | penick_ | melwitt: this particular cluster is about 1500 nodes. But I think I found the issue. They had the hypervisors set to use the conductor service, but the API was still configured to use local conductor. Which i'm pretty sure will never work. I had them fix it and so far it appears things are working. They'll do some more tests and get back to me | |
| 19:32:11 | melwitt | aha, cool | |
| 19:32:40 | dansmith | penick_: APIs should use always go straight to the db for object access | |
| 19:32:59 | dansmith | and in juno I'm not sure there was anything in conductor that apis needed to access, but local conductor for them should be fine | |
| 19:33:16 | dansmith | penick_: what was "fixed" .. the load on the conductors? | |
| 19:34:48 | efried | jaypipes This is old news, but I definitely saw a way to get None there. The placement call fails, but report client uses raise_exc=False so the response just comes back, and the caller is doing soft gets everywhere, so populates the inventory with {} and no generation. That percolates down to the code in question. | |
| 19:34:50 | melwitt | oh, oops. local conductor on API | |
| 19:39:08 | dansmith | yeah: https://github.com/openstack/nova/blob/juno-eol/nova/cmd/compute.py#L65 | |
| 19:39:15 | dansmith | no such line there for api, which means it ignores the local conductor flag | |
| 19:39:44 | dansmith | also, github groaned at me when I selected the juno-eol tag | |
| 19:39:50 | dansmith | so penick_ owes me a beverage | |
| 19:39:58 | penick_ | melwitt: this particular cluster is about 1500 nodes. But I think I found the issue. They had the hypervisors set to use the conductor service, but the API was still configured to use local conductor. Which i'm pretty sure will never work. I had them fix it and so far it appears things are working. They'll do some more tests and get back to me | |
| 19:40:02 | penick_ | dansmith sorry about that | |
| 19:40:34 | dansmith | heh | |
| 19:41:05 | melwitt | penick_: yeah, as dansmith said, that shouldn't matter. the nova/cmd/api.py doesn't consider the local conductor conf setting | |
| 19:41:08 | dansmith | actually I'm pretty sure penick_ has bought me beverages before | |
| 19:41:17 | dansmith | so I'm probably in beverage debt | |
| 19:41:23 | melwitt | penick_: and that API is supposed to access the database directly anyhow | |
| 19:41:33 | melwitt | (it does so automatically) | |
| 19:41:34 | dansmith | yeah, | |
| 19:41:52 | penick_ | that makes sense | |
| 19:41:54 | dansmith | so maybe after updating the config a bunch of computes were storming the castle and it settled down near the time you flipped that flag, | |
| 19:42:01 | dansmith | thinking it was causal? | |
| 19:42:54 | efried | What is "cold migration" aka? E.g. in the support matrix https://docs.openstack.org/nova/latest/user/support-matrix.html | |
| 19:43:08 | dansmith | efried: resize | |
| 19:43:09 | dansmith | efried: or nova migrate | |
| 19:43:35 | efried | dansmith thx | |
| 19:43:55 | mriedem | ildikov: easy fixes in https://review.openstack.org/#/c/499777/ - you want to update quick? otherwise i can | |
| 19:44:18 | efried | how do you tell it to resize to another host? | |
| 19:44:44 | melwitt | resize automatically goes to another host | |
| 19:45:00 | dansmith | well | |
| 19:45:05 | mriedem | efried: only way it can go back is the config option | |
| 19:45:07 | dansmith | unless configured | |
| 19:45:09 | dansmith | yeah | |
| 19:45:09 | mriedem | allow_resize_to_same_host | |
| 19:45:14 | melwitt | unless you have configured "allow_resize_to_same_host" it won't consider the same host | |
| 19:45:23 | efried | whoah, okay. | |
| 19:45:34 | melwitt | note that means it CAN go to the same host if you config that, not that it will ONLY go there | |
| 19:45:50 | efried | And if you only have one host, presumably, you don't need to switch the setting on? | |
| 19:45:52 | melwitt | some ppl get confused by it | |
| 19:45:53 | mriedem | efried: remember that cold migrate is called 'resize' in the code, | |
| 19:45:57 | mriedem | just like evacuate is called rebuild | |
| 19:46:02 | mriedem | totally not confusing | |
| 19:46:08 | cdent | efried: force resize to same has been discussed at various times and I’m sure patched in by vairous people | |
| 19:46:10 | mriedem | and migrate != live migration | |
| 19:46:11 | efried | "remember", sure. | |
| 19:46:14 | mriedem | because if it's not live, it's old | |
| 19:46:15 | mriedem | *cold | |
| 19:46:34 | mriedem | takashi has a spec for specifying a host for cold migratoin | |
| 19:46:41 | mriedem | it was only approved after i got him to remove the 'force' flag option | |
| 19:46:47 | mriedem | specifying a host is fine as long as the scheduler validates it first, | |
| 19:46:56 | efried | Oh, not the same spec as 'abort cold', the reviewing of which led me to this series of questions? | |
| 19:46:57 | mriedem | force flag bypasses the scheduler which gets all sorts of crazy | |
| 19:47:05 | mriedem | abort cold is different | |
| 19:47:19 | dansmith | efried: you still need the config option even if you only have one host | |
| 19:47:34 | mriedem | devstack defaults to set allow_resize_to_same_host=True | |
| 19:47:37 | mriedem | b/c it's single node | |
| 19:47:41 | efried | dansmith otherwise it'll just fail? | |
| 19:47:44 | mriedem | efried: yes | |
| 19:47:50 | efried | whee | |
| 19:48:49 | mriedem | because cold migrating to yourself doesn't make sense, | |
| 19:48:57 | mriedem | unless you're resizing the flavor | |
| 19:49:13 | mriedem | and if you're resizing to the same host, we quintuple your cpu allocations during scheduling just to make you pay for it | |
| 19:49:31 | efried | Well, to me "resize" means "change the size of". Not "move to another host". But what do I know. | |
| 19:49:50 | mriedem | hence the 'remember cold migrate is called resize internally | |
| 19:50:04 | efried | perfect. | |
| 19:50:07 | efried | Well, also in the docs. | |
| 19:50:08 | mriedem | and evacuate == rebuild | |
| 19:50:12 | efried | Which is why /me was confused. | |
| 19:50:17 | penick_ | dansmith You're probably right about the thundering herd being the problem. I think the SE team missed that these machines have CPU and memory utilization spiking. The memory spiked enough to hit swap. Also I see at least one panic in the log. | |