Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-27
14:58:58 mnaser mriedem: bauzas ^
14:59:07 mnaser took me longer to come up with a decent commit message jeez
14:59:39 melwitt .
15:00:10 mriedem bauzas: i'll give myself a todo to write a regression test for this
15:00:23 bauzas mriedem: ack
15:01:06 mriedem mnaser: +2
15:01:54 openstackgerrit Christoph Manns proposed openstack/nova master: Fix stacktraces with redis caching backend https://review.openstack.org/605748
15:01:59 bauzas mnaser: +Wipped
15:02:15 bauzas mnaser: please make a cherry-pick for rocky
15:06:58 openstackgerrit Mohammed Naser proposed openstack/nova stable/rocky: Use INFO for logging no allocation candidates https://review.openstack.org/605771
15:07:09 mnaser bauzas: done
15:26:53 mnaser AggregateRamFilter is still relevant and working? i remember there was an aggregate filter that had a long ml discussion about how it wasnt really working?
15:28:14 mriedem mnaser: ask jaypipes re https://review.openstack.org/#/c/544683/ and https://review.openstack.org/#/c/552105/
15:38:06 melwitt mnaser: this is the situation http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html and it's still the case now. those two specs ^ are what's needed to restore the ability to set allocation ratios per aggregate
15:41:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: Ignore forcing of live migration for nested instance https://review.openstack.org/605785
15:41:33 gibi mriedem: my first stab for ignoring the force flag ^^
15:42:24 mgagne @mriedem: what's up with caching scheduler?
15:42:31 mriedem mgagne: are we ok to remove it now?
15:42:32 mriedem in stein
15:43:01 mnaser melwitt: i see, i think i might be looking at the wrong filter then
15:43:03 mriedem i.e. is heal_allocation sufficient for you right now to get upgraded to a FilterScheduler world
15:43:03 mgagne mriedem: I think we figured out it was ok after you wrote the allocation healing tool.
15:43:16 mgagne ++
15:43:17 mriedem mgagne: ok
15:43:18 mriedem thanks
15:43:24 mnaser maybe it was a weigher that had the flip of a switch
15:43:29 mnaser -1 or 1 to pack vs distribute
15:43:36 mriedem host_subset_size?
15:43:42 mnaser yeah it is gr
15:44:03 mnaser https://github.com/openstack/nova/blob/master/nova/scheduler/weights/ram.py i guess no way of having that per host or per aggregate or anything
15:44:30 mgagne mnaser: we have a private implementation of the RAMWeigher per aggregate
15:44:39 melwitt mgagne: while you're here, don't forget to re-propose https://review.openstack.org/312626 for stein. I was +2 on the implementation but we were at feature freeze at the time
15:45:06 mgagne melwitt: thanks for the follow up
15:45:39 mnaser mgagne: that'd be nice to have upstream i guess, assuming that's a possiblity
15:46:00 mgagne mnaser: sure, unfortunately, I'm not sure it's gonna play well with placement API :D
15:46:09 mgagne or maybe it's not related?
15:46:12 mnaser weighers are after placement
15:46:12 openstackgerrit Balazs Gibizer proposed openstack/nova master: Ignore forcing of live migration for nested instance https://review.openstack.org/605785
15:46:13 mnaser so it doesnt matter
15:46:18 mgagne mnaser: awesome
15:46:19 mnaser its just "which machine do i prefer"
15:46:39 mnaser but im not a nova dev but thats as far as i understand it
15:46:46 mnaser weighers run after placement allocations AND filters have ran
15:46:50 mgagne so you just need a high host_subset_size for weigher to get some hosts to choose from
15:47:37 mnaser yeah we have that bumped up. the silly idea is right now by default nova spreads vms which is ok but with large vms it becomes problematic
15:47:47 mnaser because we have enough capacity for them but just not in aggregate if that makes sense
15:47:54 mgagne hehe, I know =)
15:48:12 bauzas mriedem: you're my rebuild specialist, so lemme bug you about some silly question
15:48:19 mnaser but we had some really bad decision making with our first flavors so running it with -1 ram multiplier for * .. bad things happen
15:48:51 bauzas mriedem: as of today, do we rebuild by calling the scheduler or have we stopped this ? /me is confused by the number of bugs we had about
15:49:04 bauzas my brain sucks
15:50:50 openstackgerrit Elod Illes proposed openstack/nova master: Reject networks with QoS policy https://review.openstack.org/570079
15:51:40 mgagne mnaser: based on Newton (sorry): https://gist.github.com/mgagne/142e20e32049abd0cdf5d2da7e048608
15:51:42 melwitt mnaser: hm, supposed to pack by default, I thought
15:52:07 mnaser mnaser: by default the weighers are set to positive values so distribute
15:52:59 melwitt ok. in the past the scheduler used to pack by default, so I'm not sure when/how that changed
15:53:07 mgagne doesn't make much sense to pack since if you have lets say the openstack infra team spawning 120 VMs at the same time, you will overload the same hosts with requests.
15:53:35 melwitt not since claims in the scheduler, but in the past yeah
15:53:46 mgagne true
15:54:40 mgagne but still, maybe you don't want 6 images being downloaded for the first time on the same host at the same time. or other similar expansive operations.
15:54:48 mnaser mgagne: thanks for that, ill have a look
15:55:24 melwitt generally speaking, I think pack is the more desired behavior for efficient usage of compute hosts. the only reason people increased subset size, as I understand it, was to avoid the racing of parallel requests trying to claim the same nodes with the old way of claiming
15:55:46 mnaser yeah that was what we had to do for a while
15:55:47 mriedem bauzas: we call the scheduler if the server is being rebuilt with a new image
15:55:56 mriedem b/c we need to validate the new image for the host that the instance is on
15:56:10 bauzas ok, I didn't remember all the conditionals
15:56:12 melwitt yeah, if it's a first time ever image. but when I worked at yahoo we used to warm the cache for images on the compute hosts before letting users at it
15:56:25 bauzas mriedem: thanks
15:56:29 mnaser with ceph it's not even an issue because cow
15:56:38 melwitt true. we didn't use ceph
15:57:28 mriedem mnaser: you're asking about having the RamWeigher applied to an aggregate?
15:57:37 bauzas mriedem: that's from Queens, right?
15:57:42 bauzas I remember the CVE
15:57:48 mriedem bauzas: i think so, but it was backported so ...
15:57:56 bauzas mriedem: okay
15:58:16 mriedem mnaser: reminds me of Kevin_Zheng's spec https://review.openstack.org/#/c/599308/
15:58:43 mriedem trying to make the weight configuration not global
16:00:30 mgagne mriedem: yes and I commented that I had a similar solution per aggregate, not per flavor. code posted above in a gist =)
16:03:15 mriedem mgagne: ok i remember reading your comment but totally missed the part about having the weight configuration per aggregate
16:03:50 mriedem so Kevin_Zheng's spec is maybe way too extreme on the granular side, being per-flavor,
16:04:02 mriedem but global weight configs is also pretty extreme,
16:04:12 mriedem it seems per-aggregate weight configuration would be a nice compromise
16:05:15 bauzas mriedem: mgagne: I think there was a consensus on that approach, even at the PTG
16:05:42 bauzas putting on a flavor, I nacked, but I'm okay with it per aggregate
16:05:59 bauzas (in the spec, I meant)
16:06:41 mriedem yeah i totally didn't connect the dots on what the alternative was (weights per aggregate)
16:06:54 mriedem i must have been thinking about just pinning flavors to aggregates or something, idk
16:07:03 openstackgerrit Matthew Booth proposed openstack/nova master: Fix a race evacuating instances in an anti-affinity group https://review.openstack.org/605436
16:07:03 bauzas you can mix both indeed
16:07:10 bauzas if that helps your case
16:07:33 bauzas stick flavors to aggregates, the latter having specific weight policies
16:07:47 bauzas that would fit Kevin_Zheng's concern
16:08:09 bauzas anyway, I need to disappear for a meetup, \o
16:14:01 nicolasbock mriedem: Hi. I had asked you about "lost" servers a while back, i.e. servers that were migrated but nova's database was not updated. You had mentioned that resource provider allocation show will tell me about where placement thinks the server is running.
16:14:19 cfriesen gmann: sorry, I didn't notice your question earlier. the validation of flavor extra-specs and image properties would be done on instance creation, instance resize, and instance rebuild.
16:14:42 nicolasbock Unfortunately, in our deployment none of the hypervisors shows anything using this command
16:15:19 nicolasbock Is there an issue with placement? Or are we missing some configuration? Sorry if I sound confused, but I am ;)
16:19:05 openstackgerrit Chen proposed openstack/nova master: remove commented-out code https://review.openstack.org/605635
16:21:21 melwitt nicolasbock: for that command, you need to pass the instance uuid "the consumer". did you pass that or something else? https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-show
16:22:34 nicolasbock melwitt: I ran 'resource provider list' first and took the UUIDs as argument for 'resource provider allocation show'

Earlier   Later