Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-27
15:43:29 mnaser -1 or 1 to pack vs distribute
15:43:36 mriedem host_subset_size?
15:43:42 mnaser yeah it is gr
15:44:03 mnaser https://github.com/openstack/nova/blob/master/nova/scheduler/weights/ram.py i guess no way of having that per host or per aggregate or anything
15:44:30 mgagne mnaser: we have a private implementation of the RAMWeigher per aggregate
15:44:39 melwitt mgagne: while you're here, don't forget to re-propose https://review.openstack.org/312626 for stein. I was +2 on the implementation but we were at feature freeze at the time
15:45:06 mgagne melwitt: thanks for the follow up
15:45:39 mnaser mgagne: that'd be nice to have upstream i guess, assuming that's a possiblity
15:46:00 mgagne mnaser: sure, unfortunately, I'm not sure it's gonna play well with placement API :D
15:46:09 mgagne or maybe it's not related?
15:46:12 openstackgerrit Balazs Gibizer proposed openstack/nova master: Ignore forcing of live migration for nested instance https://review.openstack.org/605785
15:46:12 mnaser weighers are after placement
15:46:13 mnaser so it doesnt matter
15:46:18 mgagne mnaser: awesome
15:46:19 mnaser its just "which machine do i prefer"
15:46:39 mnaser but im not a nova dev but thats as far as i understand it
15:46:46 mnaser weighers run after placement allocations AND filters have ran
15:46:50 mgagne so you just need a high host_subset_size for weigher to get some hosts to choose from
15:47:37 mnaser yeah we have that bumped up. the silly idea is right now by default nova spreads vms which is ok but with large vms it becomes problematic
15:47:47 mnaser because we have enough capacity for them but just not in aggregate if that makes sense
15:47:54 mgagne hehe, I know =)
15:48:12 bauzas mriedem: you're my rebuild specialist, so lemme bug you about some silly question
15:48:19 mnaser but we had some really bad decision making with our first flavors so running it with -1 ram multiplier for * .. bad things happen
15:48:51 bauzas mriedem: as of today, do we rebuild by calling the scheduler or have we stopped this ? /me is confused by the number of bugs we had about
15:49:04 bauzas my brain sucks
15:50:50 openstackgerrit Elod Illes proposed openstack/nova master: Reject networks with QoS policy https://review.openstack.org/570079
15:51:40 mgagne mnaser: based on Newton (sorry): https://gist.github.com/mgagne/142e20e32049abd0cdf5d2da7e048608
15:51:42 melwitt mnaser: hm, supposed to pack by default, I thought
15:52:07 mnaser mnaser: by default the weighers are set to positive values so distribute
15:52:59 melwitt ok. in the past the scheduler used to pack by default, so I'm not sure when/how that changed
15:53:07 mgagne doesn't make much sense to pack since if you have lets say the openstack infra team spawning 120 VMs at the same time, you will overload the same hosts with requests.
15:53:35 melwitt not since claims in the scheduler, but in the past yeah
15:53:46 mgagne true
15:54:40 mgagne but still, maybe you don't want 6 images being downloaded for the first time on the same host at the same time. or other similar expansive operations.
15:54:48 mnaser mgagne: thanks for that, ill have a look
15:55:24 melwitt generally speaking, I think pack is the more desired behavior for efficient usage of compute hosts. the only reason people increased subset size, as I understand it, was to avoid the racing of parallel requests trying to claim the same nodes with the old way of claiming
15:55:46 mnaser yeah that was what we had to do for a while
15:55:47 mriedem bauzas: we call the scheduler if the server is being rebuilt with a new image
15:55:56 mriedem b/c we need to validate the new image for the host that the instance is on
15:56:10 bauzas ok, I didn't remember all the conditionals
15:56:12 melwitt yeah, if it's a first time ever image. but when I worked at yahoo we used to warm the cache for images on the compute hosts before letting users at it
15:56:25 bauzas mriedem: thanks
15:56:29 mnaser with ceph it's not even an issue because cow
15:56:38 melwitt true. we didn't use ceph
15:57:28 mriedem mnaser: you're asking about having the RamWeigher applied to an aggregate?
15:57:37 bauzas mriedem: that's from Queens, right?
15:57:42 bauzas I remember the CVE
15:57:48 mriedem bauzas: i think so, but it was backported so ...
15:57:56 bauzas mriedem: okay
15:58:16 mriedem mnaser: reminds me of Kevin_Zheng's spec https://review.openstack.org/#/c/599308/
15:58:43 mriedem trying to make the weight configuration not global
16:00:30 mgagne mriedem: yes and I commented that I had a similar solution per aggregate, not per flavor. code posted above in a gist =)
16:03:15 mriedem mgagne: ok i remember reading your comment but totally missed the part about having the weight configuration per aggregate
16:03:50 mriedem so Kevin_Zheng's spec is maybe way too extreme on the granular side, being per-flavor,
16:04:02 mriedem but global weight configs is also pretty extreme,
16:04:12 mriedem it seems per-aggregate weight configuration would be a nice compromise
16:05:15 bauzas mriedem: mgagne: I think there was a consensus on that approach, even at the PTG
16:05:42 bauzas putting on a flavor, I nacked, but I'm okay with it per aggregate
16:05:59 bauzas (in the spec, I meant)
16:06:41 mriedem yeah i totally didn't connect the dots on what the alternative was (weights per aggregate)
16:06:54 mriedem i must have been thinking about just pinning flavors to aggregates or something, idk
16:07:03 bauzas you can mix both indeed
16:07:03 openstackgerrit Matthew Booth proposed openstack/nova master: Fix a race evacuating instances in an anti-affinity group https://review.openstack.org/605436
16:07:10 bauzas if that helps your case
16:07:33 bauzas stick flavors to aggregates, the latter having specific weight policies
16:07:47 bauzas that would fit Kevin_Zheng's concern
16:08:09 bauzas anyway, I need to disappear for a meetup, \o
16:14:01 nicolasbock mriedem: Hi. I had asked you about "lost" servers a while back, i.e. servers that were migrated but nova's database was not updated. You had mentioned that resource provider allocation show will tell me about where placement thinks the server is running.
16:14:19 cfriesen gmann: sorry, I didn't notice your question earlier. the validation of flavor extra-specs and image properties would be done on instance creation, instance resize, and instance rebuild.
16:14:42 nicolasbock Unfortunately, in our deployment none of the hypervisors shows anything using this command
16:15:19 nicolasbock Is there an issue with placement? Or are we missing some configuration? Sorry if I sound confused, but I am ;)
16:19:05 openstackgerrit Chen proposed openstack/nova master: remove commented-out code https://review.openstack.org/605635
16:21:21 melwitt nicolasbock: for that command, you need to pass the instance uuid "the consumer". did you pass that or something else? https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-show
16:22:34 nicolasbock melwitt: I ran 'resource provider list' first and took the UUIDs as argument for 'resource provider allocation show'
16:23:27 melwitt nicolasbock: ok, those would be the compute host uuids, which is not what you need to pass. you need to pass the uuid of the instance/server, that is "lost"
16:24:09 melwitt and then it will show you information about that instance's allocations and where they are, which resource provider aka which compute host
16:25:07 nicolasbock Oh sorry, I totally misunderstood the command :(
16:25:09 nicolasbock It's working much better now
16:25:41 nicolasbock Thanks!
16:25:44 melwitt no worries, I had thought the same thing the first time I learned about the command
16:43:39 cfriesen mdbooth: I think I found a flaw in your fail-fast algorithm for https://review.openstack.org/605436
16:46:42 cfriesen We're proposing kind of a "big hammer" fix for a missing marker during online data migration. (https://review.openstack.org/#/c/605164/) Does anyone have a more elegant solution?
16:56:45 mriedem melwitt: nicolasbock: https://docs.openstack.org/osc-placement/latest/cli/index.html#cmdoption-openstack-resource-provider-allocation-show-arg-uuid describes the uuid but we could rename that metavar to be consumer_uuid so it's more obvious from the beginning
16:57:19 mriedem consumers aren't a top-level resource in placement so that's probably why it's confusing
16:57:26 mriedem unlike openstack resource provider show https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-show
16:57:41 melwitt yeah. it does say "consumer". I think the confusion comes from the fact that it's in the resource provider command family
16:58:29 melwitt the first time I read it, I thought the documentation was a mistake. but edleafe confirmed that it is indeed supposed to be "consumer" uuid
16:58:49 mriedem probably should have been "openstack resource allocation list <consumer_uuid>"
16:59:36 mriedem could still add that and deprecate the old command
17:02:08 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930
17:15:29 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix stacktraces with redis caching backend https://review.openstack.org/605748
17:22:54 mnaser openstackclient doesnt let you do a live migration unless you specify a host?
17:23:29 mnaser https://github.com/openstack/python-openstackclient/blob/c0567806916995698e94734d2b2c422a4bf5a1db/openstackclient/compute/v2/server.py#L1333-L1337
17:23:43 nicolasbock Thanks mriedem and melwitt . Yes, the wording could be clearer, but then again I probabaly could have read the help a little bit more carefully :)
17:23:58 nicolasbock I think that's true mnaser
17:24:16 mnaser nova can do live migrations without specifying a host
17:24:19 mnaser the novaclient lets you do it
17:24:32 mnaser and i think there's been voices of "forcing a host in migrations is a bad idea™"
17:25:41 nicolasbock But in 'nova live-migration' you also need to specify a host
17:25:49 nicolasbock Same as in 'openstack server migrate'

Earlier   Later