Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-19
18:55:59 jaypipes dansmith: ^
18:57:36 dansmith melwitt: right
19:13:52 sahid jaypipes: ci is green https://review.openstack.org/#/c/485522/
19:14:31 sahid btw mriedem, did you take descison for this: https://review.openstack.org/#/c/511188/ ?
19:21:42 mriedem sahid: no
19:21:47 mriedem it's not on my list
19:24:02 sahid mriedem: thanks
19:31:15 penick_ melwitt: this particular cluster is about 1500 nodes. But I think I found the issue. They had the hypervisors set to use the conductor service, but the API was still configured to use local conductor. Which i'm pretty sure will never work. I had them fix it and so far it appears things are working. They'll do some more tests and get back to me
19:32:11 melwitt aha, cool
19:32:40 dansmith penick_: APIs should use always go straight to the db for object access
19:32:59 dansmith and in juno I'm not sure there was anything in conductor that apis needed to access, but local conductor for them should be fine
19:33:16 dansmith penick_: what was "fixed" .. the load on the conductors?
19:34:48 efried jaypipes This is old news, but I definitely saw a way to get None there. The placement call fails, but report client uses raise_exc=False so the response just comes back, and the caller is doing soft gets everywhere, so populates the inventory with {} and no generation. That percolates down to the code in question.
19:34:50 melwitt oh, oops. local conductor on API
19:39:08 dansmith yeah: https://github.com/openstack/nova/blob/juno-eol/nova/cmd/compute.py#L65
19:39:15 dansmith no such line there for api, which means it ignores the local conductor flag
19:39:44 dansmith also, github groaned at me when I selected the juno-eol tag
19:39:50 dansmith so penick_ owes me a beverage
19:39:58 penick_ melwitt: this particular cluster is about 1500 nodes. But I think I found the issue. They had the hypervisors set to use the conductor service, but the API was still configured to use local conductor. Which i'm pretty sure will never work. I had them fix it and so far it appears things are working. They'll do some more tests and get back to me
19:40:02 penick_ dansmith sorry about that
19:40:34 dansmith heh
19:41:05 melwitt penick_: yeah, as dansmith said, that shouldn't matter. the nova/cmd/api.py doesn't consider the local conductor conf setting
19:41:08 dansmith actually I'm pretty sure penick_ has bought me beverages before
19:41:17 dansmith so I'm probably in beverage debt
19:41:23 melwitt penick_: and that API is supposed to access the database directly anyhow
19:41:33 melwitt (it does so automatically)
19:41:34 dansmith yeah,
19:41:52 penick_ that makes sense
19:41:54 dansmith so maybe after updating the config a bunch of computes were storming the castle and it settled down near the time you flipped that flag,
19:42:01 dansmith thinking it was causal?
19:42:54 efried What is "cold migration" aka? E.g. in the support matrix https://docs.openstack.org/nova/latest/user/support-matrix.html
19:43:08 dansmith efried: resize
19:43:09 dansmith efried: or nova migrate
19:43:35 efried dansmith thx
19:43:55 mriedem ildikov: easy fixes in https://review.openstack.org/#/c/499777/ - you want to update quick? otherwise i can
19:44:18 efried how do you tell it to resize to another host?
19:44:44 melwitt resize automatically goes to another host
19:45:00 dansmith well
19:45:05 mriedem efried: only way it can go back is the config option
19:45:07 dansmith unless configured
19:45:09 dansmith yeah
19:45:09 mriedem allow_resize_to_same_host
19:45:14 melwitt unless you have configured "allow_resize_to_same_host" it won't consider the same host
19:45:23 efried whoah, okay.
19:45:34 melwitt note that means it CAN go to the same host if you config that, not that it will ONLY go there
19:45:50 efried And if you only have one host, presumably, you don't need to switch the setting on?
19:45:52 melwitt some ppl get confused by it
19:45:53 mriedem efried: remember that cold migrate is called 'resize' in the code,
19:45:57 mriedem just like evacuate is called rebuild
19:46:02 mriedem totally not confusing
19:46:08 cdent efried: force resize to same has been discussed at various times and I’m sure patched in by vairous people
19:46:10 mriedem and migrate != live migration
19:46:11 efried "remember", sure.
19:46:14 mriedem because if it's not live, it's old
19:46:15 mriedem *cold
19:46:34 mriedem takashi has a spec for specifying a host for cold migratoin
19:46:41 mriedem it was only approved after i got him to remove the 'force' flag option
19:46:47 mriedem specifying a host is fine as long as the scheduler validates it first,
19:46:56 efried Oh, not the same spec as 'abort cold', the reviewing of which led me to this series of questions?
19:46:57 mriedem force flag bypasses the scheduler which gets all sorts of crazy
19:47:05 mriedem abort cold is different
19:47:19 dansmith efried: you still need the config option even if you only have one host
19:47:34 mriedem devstack defaults to set allow_resize_to_same_host=True
19:47:37 mriedem b/c it's single node
19:47:41 efried dansmith otherwise it'll just fail?
19:47:44 mriedem efried: yes
19:47:50 efried whee
19:48:49 mriedem because cold migrating to yourself doesn't make sense,
19:48:57 mriedem unless you're resizing the flavor
19:49:13 mriedem and if you're resizing to the same host, we quintuple your cpu allocations during scheduling just to make you pay for it
19:49:31 efried Well, to me "resize" means "change the size of". Not "move to another host". But what do I know.
19:49:50 mriedem hence the 'remember cold migrate is called resize internally
19:50:04 efried perfect.
19:50:07 efried Well, also in the docs.
19:50:08 mriedem and evacuate == rebuild
19:50:12 efried Which is why /me was confused.
19:50:17 penick_ dansmith You're probably right about the thundering herd being the problem. I think the SE team missed that these machines have CPU and memory utilization spiking. The memory spiked enough to hit swap. Also I see at least one panic in the log.
19:50:43 dansmith efried: have you read my blog post?
19:50:56 efried dansmith Which?
19:51:08 dansmith penick_: ack
19:51:09 dansmith efried: the secret decoder ring for all these move operations
19:51:14 efried dansmith I hope it's not too long.
19:51:32 efried dansmith But send me the link and I'll see if I can plough my way through it.
19:51:57 efried (In all seriousness, dansmith, I would love a decoder ring for the move ops)
19:52:07 dansmith oh it's long, real long..
19:52:14 dansmith er, the post.. yeah, not too long
19:52:23 dansmith efried: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/
19:52:30 efried IBM forbids me even responding to that.
19:53:05 dansmith ack
19:53:22 dansmith my corporate compliance training requirement is coming up soon
19:53:24 dansmith I better re-read
19:54:15 openstackgerrit Merged openstack/nova master: Add 'delete_host' command in 'nova-manage cell_v2' https://review.openstack.org/510324
19:56:34 ildikov mriedem: not at the laptop now
19:56:54 mriedem ildikov: i'll update it
19:57:44 ildikov mriedem: thank you! I owe you, I guess way more than one!!!
19:58:08 mriedem get me out of work meetings and customer dinners in sydney and the debt i spaid
19:58:42 efried dansmith Super helpful article, thank you.
20:01:05 penick_ dansmith and melwitt thanks for the help, btw
20:01:32 dansmith penick_: no problem. Now I have to go shower the juno off me
20:02:10 penick_ Sorry about that. On the upside you don't need to bathe in the juno-ness all day every day.

Earlier   Later