Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-23
16:04:06 mriedem s10: if you can test out https://review.openstack.org/#/c/569247/ and report back that would be very helpful in moving this forward
16:04:38 sean-k-mooney mriedem: sure that makes sense.
16:04:51 cdent mriedem, s10: whatever you work out here, it would still be great to see some kind of summary to the mailing list
16:05:08 s10 mriedem: Ok, I will test it
16:05:31 s10 mriedem: could this change be cherry-picked without rework to Pike?
16:06:52 sean-k-mooney s10 it looks pretty trivial in terms of code change. this would not qualify for backport however as it is not fixing a bug. it is a performance enhancement
16:08:45 mriedem s10: probably, there was another change merged in the series before it that you'll need https://review.openstack.org/#/q/topic:bug/1737465+(status:open+OR+status:merged)
16:08:55 mriedem https://review.openstack.org/#/q/Iccefbfdfa578515a004ef6ac718bac1a49d5c5fd
16:09:00 mriedem you'll need that in pike first
16:09:18 mriedem and then https://review.openstack.org/#/c/569247/
16:15:15 s10 Our out-of-tree filter rely on host_state.instances.values(), we need values of the instance.memory_mb. Will we be hit with a lazy-load penalty in this case, with https://review.openstack.org/#/c/570083/?
16:16:53 dansmith yup
16:18:40 sean-k-mooney s10: what does your real ram filter actully check vs what placement will do?
16:19:12 sean-k-mooney s10: are you trying to disable oversubscription?
16:20:19 s10 sean-k-mooney: actual usage of the ram on compute nodes. Every QEMU process produces some memory overhead (up to 500-1000mb in some cases, depends on disk usage and rbd cached, I believe). Placement doesn't know anything about it and can schedule instance to the host, where real size of free ram is less than reserved_memory_mb.
16:20:51 mriedem yeah we don't track overhead in placement, that's a known thing
16:21:27 mriedem and we've said for now, buffer that with reserved_host_memory_mb
16:21:40 sean-k-mooney s10: how are you getting the real host memory usage?
16:23:02 mriedem s10: also fyi http://lists.openstack.org/pipermail/openstack-dev/2018-May/130624.html
16:23:04 s10 We patched nova/scheduler/host_manager.py, added self.memory_mb_used = compute.memory_mb_used
16:23:21 mriedem s10: then why would you need host_state.instances.values()?
16:23:54 s10 Because without it we can't compute reserved_memory_mb, it's in config...
16:24:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Heal RequestSpec.is_bfv for legacy instances during moves https://review.openstack.org/583715
16:24:07 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix wonky reqspec handling in conductor.unshelve_instance https://review.openstack.org/583739
16:24:08 openstackgerrit Matt Riedemann proposed openstack/nova master: Add shelve/unshelve wrinkle to volume-backed disk func test https://review.openstack.org/584931
16:24:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Cache is_bfv check in ResourceTracker https://review.openstack.org/584962
16:24:26 s10 Maybe we should pass reserved_memory_mb same way as we pass memory_mb_used...
16:25:01 sean-k-mooney s10: it would make sense to me to just embed the reserved memory in the host state object
16:25:24 mriedem to be clear, CONF.reserved_host_memory_mb is used to reserve capacity for MEMORY_MB in the compute node resource provider's inventory in placement
16:25:35 mriedem which is used to filter hosts before they even get to your filter
16:25:58 mriedem if you need to reserve space on the host for qemu/rbd, you should increase reserved_host_memory_mb per host
16:26:32 s10 But it is still calculated per compute, based on the config in nova-compute, right?
16:26:51 mriedem the config is read in compute yes
16:27:17 mriedem compute node memory_mb_used is later adjusted in the resource tracker based on the instance flavor
16:27:20 mriedem per instance on that host
16:27:43 sean-k-mooney mriedem: ture but s10's filter would catch the case where they did not reserve enough and the free ram is actully less then the reserved
16:27:52 mriedem but that information is also available via placement and is used to calculate the allocation candidates that the scheduler gets before we even hit the filters
16:28:28 mriedem sean-k-mooney: you might need to be conservative then in the reserved memory_mb
16:28:31 sean-k-mooney mriedem: placement only has allocations based on the flavor request. not the actul memory used by the instance
16:28:32 mriedem if this is a persistent problem
16:29:24 mriedem Kevin_Zheng: fyi https://review.openstack.org/#/c/584962/
16:29:32 s10 We used reserved_host_memory_mb for that purpose, but it doesn't work well for us, some instances produces memory overhead over time, and in this case we might end up reserving 50% of host memory
16:29:55 sean-k-mooney mriedem: im just wondering could we symplify this by comparing free_ram_mb > reseved in the fileter if we added reserved to the hostsate object
16:30:21 mriedem s10: hmm, ok, it would be helpful to write something up about this problem in the openstack-dev mailing list, tag the subject line with [nova] and describe the issue and what your scheduler filter does to resolve it
16:30:36 mriedem s10: upstream changes only affect out of tree filters while those filters are out of tree,
16:30:47 mriedem if your filter were in tree, we would have considered this when making that change
16:31:18 sean-k-mooney s10: well one of the main issue with reserved_host_memory_mb is that it does not take account of numa topology of the host so your instance can be killed by oom if you are not useing hugepages and a numa node is exausted but non local memory is free
16:31:26 mriedem sean-k-mooney: maybe, but solutions should be discussed in the ML
16:31:41 s10 Thank you. I will test performance with fixes for the host_state, and ask our developers to change our filter not to use host_state.
16:31:51 sean-k-mooney mriedem: sure :) it was just a taught
16:32:28 mriedem mostly just because this is a bad week with FF on thursday
16:32:35 mriedem don't really have time to design solutions for things that can go into the ML
16:35:00 sean-k-mooney s10: is there a bug open for your usecase? its 2 late to adress in rocky but we could likely adrss in early stien if we captured the request.
16:38:03 s10 sean-k-mooney: I didn't see bugs, related to real memory overhead problem. Only remember, that there was a thread in openstack-operators (?)
16:40:47 s10 https://bugs.launchpad.net/nova/+bug/1683858
16:40:48 openstack Launchpad bug 1683858 in OpenStack Compute (nova) "Allocation records do not contain overhead information" [Medium,Won't fix]
16:41:22 sean-k-mooney s10: based on the wording i assume that was suggesting this in placement hence the wont fix.
16:41:59 sean-k-mooney s10: i dont think placement is the right place to adress this but an in tree filter would be potentially reasonable.
16:44:57 mriedem stephenfin: we've got a runway slot opening so i'm going to throw the numa vswitch series back in ther
16:44:58 mriedem *there
16:44:59 mriedem just FYI
16:45:16 stephenfin Sweet. Should have that move thing done in a short bit
16:49:58 sean-k-mooney s10: i wont promise anything but i have added this usecase to my whiteboard of random TODOs. i might see if i can come up with somthing for stien if there is still interest.
16:50:56 AJaeger nova cores, could you review https://review.openstack.org/#/c/583023/ to move the job in-tree, please? that'S a followup to work for os-vif. stephenfin and sean-k-mooney, want to look at it again, please?
16:52:50 stephenfin AJaeger: ack
16:53:21 stephenfin lyarwood: I didn't get to https://review.openstack.org/#/c/562072/ yet :( It's first thing I'm doing tomorrow morning though, I promise
16:53:38 mriedem dansmith should probably look at those
16:53:45 mriedem given evacness
16:53:59 stephenfin mriedem: Suits me down to the ground
16:54:57 sean-k-mooney AJaeger: will nova beable to import the job from os-vif? i just kicked off an experimental job to make sure it all works
16:57:45 openstackgerrit Merged openstack/nova master: perform reshaper operations in single transaction https://review.openstack.org/582383
16:57:53 openstackgerrit Merged openstack/nova master: Refactor _heal_instances_in_cell https://review.openstack.org/577896
16:58:34 AJaeger sean-k-mooney: that'S how zuul was designed ;)
16:58:42 AJaeger sean-k-mooney: but yeah, I just have done that - thanks
16:59:12 AJaeger thanks, stephenfin
17:00:16 AJaeger sean-k-mooney: it's "check experimental"
17:00:18 sean-k-mooney AJaeger: os-vif has to be whitelisted in the tenant config to suppy job definitons... that said it is in project-config already is so ya i guess this should work.
17:00:51 AJaeger sean-k-mooney: all active repos are in that list ;)
17:01:36 sean-k-mooney AJaeger: ya i was just getting a little confused with how i was configuring my thridparty ci when i was playying with zuulv3
17:02:19 AJaeger ;)
17:02:39 AJaeger sean-k-mooney: the job is scheduled to run according to zuul.openstack.org, so it's fine.
17:04:52 sean-k-mooney AJaeger: yep i was just checking that myself
17:05:04 openstackgerrit Elod Illes proposed openstack/nova stable/queens: Call generate_image_url only for legacy notification https://review.openstack.org/584969
17:06:54 AJaeger ok, hope another core will review the job change in 583023 then - so that we can cleanup the infra files.
17:10:01 sean-k-mooney AJaeger: two quick questions by the way. i plan on converting it to a non legacy job. are there good examples of a vannily devstack job i can base it off. and 2 have ye updated the zuul from scratch to cover creating your own base job yet?
17:11:13 sean-k-mooney AJaeger: perhaps those are better asked on the zuul/infra channels instead.
17:12:25 AJaeger sean-k-mooney: for your second question: better ask on #zuul - I'm not involved there. For the devstack job, best ask andreaf on #openstack-qa. I know he converted some but don't know which one is a good example. You might want to search using codesearch, e.g. http://codesearch.openstack.org/?q=parent%3A%20devstack&i=nope&files=&repos=
17:14:23 sean-k-mooney AJaeger: thanks. its too late for rocky but i hope to convert the os-vif legacy job to non legacy and add a linux bridge version in the next month or so
17:17:29 kashyap cfriesen_: Thanks for the quick review
17:18:14 AJaeger sean-k-mooney: IMHO never too late to convert it ;)
17:22:12 sean-k-mooney AJaeger: well what i ment is we are past non-client ff so it wont be on the stable/rocky branch unless we backport it
17:39:35 mriedem stephenfin: heh, sorry, just meant dan has special context on that code
17:39:41 mriedem so he should (also) review it
17:52:54 sean-k-mooney do you ever read code and have to fight the urge to delete it all.
18:02:52 AJaeger sean-k-mooney: ah, yes
18:05:37 mriedem zzzeek: is there any major difference between doing a select query filter with == None vs == null() https://review.openstack.org/#/c/584504/3/nova/objects/instance_mapping.py@156 ?
18:06:28 zzzeek mriedem: shouldn't make a difference
18:06:32 mriedem ok thanks
18:07:31 zzzeek mriedem: null() is helpful in other cases where None might mean something different, like you want to force a SQL expression in some cases. but here == None will do "IS NULL" anyway
18:10:10 openstackgerrit Matt Riedemann proposed openstack/nova master: Use consumer generation in _heal_allocations_for_instance https://review.openstack.org/577905

Earlier   Later