| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 16:24:26 | s10 | Maybe we should pass reserved_memory_mb same way as we pass memory_mb_used... | |
| 16:25:01 | sean-k-mooney | s10: it would make sense to me to just embed the reserved memory in the host state object | |
| 16:25:24 | mriedem | to be clear, CONF.reserved_host_memory_mb is used to reserve capacity for MEMORY_MB in the compute node resource provider's inventory in placement | |
| 16:25:35 | mriedem | which is used to filter hosts before they even get to your filter | |
| 16:25:58 | mriedem | if you need to reserve space on the host for qemu/rbd, you should increase reserved_host_memory_mb per host | |
| 16:26:32 | s10 | But it is still calculated per compute, based on the config in nova-compute, right? | |
| 16:26:51 | mriedem | the config is read in compute yes | |
| 16:27:17 | mriedem | compute node memory_mb_used is later adjusted in the resource tracker based on the instance flavor | |
| 16:27:20 | mriedem | per instance on that host | |
| 16:27:43 | sean-k-mooney | mriedem: ture but s10's filter would catch the case where they did not reserve enough and the free ram is actully less then the reserved | |
| 16:27:52 | mriedem | but that information is also available via placement and is used to calculate the allocation candidates that the scheduler gets before we even hit the filters | |
| 16:28:28 | mriedem | sean-k-mooney: you might need to be conservative then in the reserved memory_mb | |
| 16:28:31 | sean-k-mooney | mriedem: placement only has allocations based on the flavor request. not the actul memory used by the instance | |
| 16:28:32 | mriedem | if this is a persistent problem | |
| 16:29:24 | mriedem | Kevin_Zheng: fyi https://review.openstack.org/#/c/584962/ | |
| 16:29:32 | s10 | We used reserved_host_memory_mb for that purpose, but it doesn't work well for us, some instances produces memory overhead over time, and in this case we might end up reserving 50% of host memory | |
| 16:29:55 | sean-k-mooney | mriedem: im just wondering could we symplify this by comparing free_ram_mb > reseved in the fileter if we added reserved to the hostsate object | |
| 16:30:21 | mriedem | s10: hmm, ok, it would be helpful to write something up about this problem in the openstack-dev mailing list, tag the subject line with [nova] and describe the issue and what your scheduler filter does to resolve it | |
| 16:30:36 | mriedem | s10: upstream changes only affect out of tree filters while those filters are out of tree, | |
| 16:30:47 | mriedem | if your filter were in tree, we would have considered this when making that change | |
| 16:31:18 | sean-k-mooney | s10: well one of the main issue with reserved_host_memory_mb is that it does not take account of numa topology of the host so your instance can be killed by oom if you are not useing hugepages and a numa node is exausted but non local memory is free | |
| 16:31:26 | mriedem | sean-k-mooney: maybe, but solutions should be discussed in the ML | |
| 16:31:41 | s10 | Thank you. I will test performance with fixes for the host_state, and ask our developers to change our filter not to use host_state. | |
| 16:31:51 | sean-k-mooney | mriedem: sure :) it was just a taught | |
| 16:32:28 | mriedem | mostly just because this is a bad week with FF on thursday | |
| 16:32:35 | mriedem | don't really have time to design solutions for things that can go into the ML | |
| 16:35:00 | sean-k-mooney | s10: is there a bug open for your usecase? its 2 late to adress in rocky but we could likely adrss in early stien if we captured the request. | |
| 16:38:03 | s10 | sean-k-mooney: I didn't see bugs, related to real memory overhead problem. Only remember, that there was a thread in openstack-operators (?) | |
| 16:40:47 | s10 | https://bugs.launchpad.net/nova/+bug/1683858 | |
| 16:40:48 | openstack | Launchpad bug 1683858 in OpenStack Compute (nova) "Allocation records do not contain overhead information" [Medium,Won't fix] | |
| 16:41:22 | sean-k-mooney | s10: based on the wording i assume that was suggesting this in placement hence the wont fix. | |
| 16:41:59 | sean-k-mooney | s10: i dont think placement is the right place to adress this but an in tree filter would be potentially reasonable. | |
| 16:44:57 | mriedem | stephenfin: we've got a runway slot opening so i'm going to throw the numa vswitch series back in ther | |
| 16:44:58 | mriedem | *there | |
| 16:44:59 | mriedem | just FYI | |
| 16:45:16 | stephenfin | Sweet. Should have that move thing done in a short bit | |
| 16:49:58 | sean-k-mooney | s10: i wont promise anything but i have added this usecase to my whiteboard of random TODOs. i might see if i can come up with somthing for stien if there is still interest. | |
| 16:50:56 | AJaeger | nova cores, could you review https://review.openstack.org/#/c/583023/ to move the job in-tree, please? that'S a followup to work for os-vif. stephenfin and sean-k-mooney, want to look at it again, please? | |
| 16:52:50 | stephenfin | AJaeger: ack | |
| 16:53:21 | stephenfin | lyarwood: I didn't get to https://review.openstack.org/#/c/562072/ yet :( It's first thing I'm doing tomorrow morning though, I promise | |
| 16:53:38 | mriedem | dansmith should probably look at those | |
| 16:53:45 | mriedem | given evacness | |
| 16:53:59 | stephenfin | mriedem: Suits me down to the ground | |
| 16:54:57 | sean-k-mooney | AJaeger: will nova beable to import the job from os-vif? i just kicked off an experimental job to make sure it all works | |
| 16:57:45 | openstackgerrit | Merged openstack/nova master: perform reshaper operations in single transaction https://review.openstack.org/582383 | |
| 16:57:53 | openstackgerrit | Merged openstack/nova master: Refactor _heal_instances_in_cell https://review.openstack.org/577896 | |
| 16:58:34 | AJaeger | sean-k-mooney: that'S how zuul was designed ;) | |
| 16:58:42 | AJaeger | sean-k-mooney: but yeah, I just have done that - thanks | |
| 16:59:12 | AJaeger | thanks, stephenfin | |
| 17:00:16 | AJaeger | sean-k-mooney: it's "check experimental" | |
| 17:00:18 | sean-k-mooney | AJaeger: os-vif has to be whitelisted in the tenant config to suppy job definitons... that said it is in project-config already is so ya i guess this should work. | |
| 17:00:51 | AJaeger | sean-k-mooney: all active repos are in that list ;) | |
| 17:01:36 | sean-k-mooney | AJaeger: ya i was just getting a little confused with how i was configuring my thridparty ci when i was playying with zuulv3 | |
| 17:02:19 | AJaeger | ;) | |
| 17:02:39 | AJaeger | sean-k-mooney: the job is scheduled to run according to zuul.openstack.org, so it's fine. | |
| 17:04:52 | sean-k-mooney | AJaeger: yep i was just checking that myself | |
| 17:05:04 | openstackgerrit | Elod Illes proposed openstack/nova stable/queens: Call generate_image_url only for legacy notification https://review.openstack.org/584969 | |
| 17:06:54 | AJaeger | ok, hope another core will review the job change in 583023 then - so that we can cleanup the infra files. | |
| 17:10:01 | sean-k-mooney | AJaeger: two quick questions by the way. i plan on converting it to a non legacy job. are there good examples of a vannily devstack job i can base it off. and 2 have ye updated the zuul from scratch to cover creating your own base job yet? | |
| 17:11:13 | sean-k-mooney | AJaeger: perhaps those are better asked on the zuul/infra channels instead. | |
| 17:12:25 | AJaeger | sean-k-mooney: for your second question: better ask on #zuul - I'm not involved there. For the devstack job, best ask andreaf on #openstack-qa. I know he converted some but don't know which one is a good example. You might want to search using codesearch, e.g. http://codesearch.openstack.org/?q=parent%3A%20devstack&i=nope&files=&repos= | |
| 17:14:23 | sean-k-mooney | AJaeger: thanks. its too late for rocky but i hope to convert the os-vif legacy job to non legacy and add a linux bridge version in the next month or so | |
| 17:17:29 | kashyap | cfriesen_: Thanks for the quick review | |
| 17:18:14 | AJaeger | sean-k-mooney: IMHO never too late to convert it ;) | |
| 17:22:12 | sean-k-mooney | AJaeger: well what i ment is we are past non-client ff so it wont be on the stable/rocky branch unless we backport it | |
| 17:39:35 | mriedem | stephenfin: heh, sorry, just meant dan has special context on that code | |
| 17:39:41 | mriedem | so he should (also) review it | |
| 17:52:54 | sean-k-mooney | do you ever read code and have to fight the urge to delete it all. | |
| 18:02:52 | AJaeger | sean-k-mooney: ah, yes | |
| 18:05:37 | mriedem | zzzeek: is there any major difference between doing a select query filter with == None vs == null() https://review.openstack.org/#/c/584504/3/nova/objects/instance_mapping.py@156 ? | |
| 18:06:28 | zzzeek | mriedem: shouldn't make a difference | |
| 18:06:32 | mriedem | ok thanks | |
| 18:07:31 | zzzeek | mriedem: null() is helpful in other cases where None might mean something different, like you want to force a SQL expression in some cases. but here == None will do "IS NULL" anyway | |
| 18:10:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use consumer generation in _heal_allocations_for_instance https://review.openstack.org/577905 | |
| 18:13:49 | mriedem | dansmith: i'm +2 on your queued_for_delete migration, i guess we can rebase the series to swap those | |
| 18:13:56 | dansmith | cool | |
| 18:15:16 | mriedem | looks like powervm might have some ci issues http://184.172.12.213/17/560317/30/check/nova-in-tree-pvm/bf5d5bc/logs/stack.sh.txt.gz | |
| 18:15:23 | mriedem | error: command 'powerpc64le-linux-gnu-gcc' failed with exit status 1 | |
| 18:15:37 | mriedem | error: PIC register clobbered by ‘r30’ in ‘asm’ | |
| 18:16:14 | mriedem | powerkvm ci failing on the same https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/17/560317/30/check/tempest-dsvm-full-xenial/ac71b2d/devstacklog.txt.gz | |
| 18:16:19 | mriedem | edmondsw: ^ | |
| 18:17:07 | edmondsw | mriedem yep, we're aware, have a fix proposed to requirements to block greenlet 0.4.14 | |
| 18:17:07 | mriedem | https://pypi.org/project/greenlet/0.4.14/ | |
| 18:17:11 | mriedem | ah cool | |
| 18:17:24 | edmondsw | also broken the KVM CI from what I heard | |
| 18:17:29 | edmondsw | broke* | |
| 18:17:35 | mriedem | it did | |
| 18:17:54 | edmondsw | getting a little pushback from the requirements guys, so we'll see how this goes | |
| 18:18:05 | edmondsw | https://review.openstack.org/#/c/584881/ | |
| 18:18:11 | mriedem | yeah i'm looking at it | |
| 18:18:49 | edmondsw | just posted a comment in response | |
| 18:19:48 | mriedem | hmm, so in the before times we would have also blacklisted greenlet 0.4.14 in global-requirements.txt, | |
| 18:19:55 | mriedem | but that's per-project now, | |
| 18:20:09 | mriedem | so i'm wondering if nova can just blacklist 0.4.14 in nova's requirements.txt? | |
| 18:20:16 | mriedem | smcginnis: do you know ^ ? | |
| 18:21:15 | smcginnis | mriedem: Yeah, I believe Nova can just block that now. | |
| 18:27:18 | smcginnis | FWIW, until the gcc issue gets clarified I think we should block it in requirements too. | |
| 18:27:38 | openstackgerrit | Merged openstack/nova master: Rename auth_uri to www_authenticate_uri https://review.openstack.org/576820 | |
| 18:43:52 | openstackgerrit | sean mooney proposed openstack/nova master: fix disk_bus handeling https://review.openstack.org/584999 | |
| 18:55:04 | openstackgerrit | Merged openstack/nova master: Func test for improper cn local DISK_GB reporting https://review.openstack.org/583646 | |