| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-04 | |||
| 17:14:08 | dansmith | mriedem: I'm super confused about this | |
| 17:14:17 | dansmith | this spec seems to confuse disabledness and down-ness | |
| 17:14:20 | dansmith | I'm commenting | |
| 17:17:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Recalculate 'RequestSpec.numa_topology' on resize https://review.opendev.org/662522 | |
| 17:18:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add '_setup_compute_services' helper https://review.opendev.org/663102 | |
| 17:21:20 | spatel | Folks, i am getting this error qemu-kvm: -object memory-backend-ram,id=ram-node0,size=12884901888,host-nodes=0,policy=bind: cannot set up guest memory 'ram-node0': Cannot allocate memory | |
| 17:21:56 | spatel | I have 32G memory and hugemem is 25G | |
| 17:22:11 | spatel | i am trying to build VM with 24G and its throwing this error | |
| 17:23:18 | sean-k-mooney | spatel: do you have 24G of hugepages on a single numa node | |
| 17:23:47 | spatel | no | |
| 17:23:48 | sean-k-mooney | by default the kernel will split it across all numa nodes if you allcoate the hugepages on teh kernel commandline | |
| 17:24:06 | spatel | How do i tell my flavor to use both side of NUMA ? | |
| 17:24:33 | sean-k-mooney | am you have to create a guest with multiple numa nodes which is done by setting hw:numa_nodes=2 | |
| 17:24:49 | sean-k-mooney | in the flavor | |
| 17:25:01 | spatel | I have that option already in flavor | |
| 17:25:02 | sean-k-mooney | or hw_numa_nodes=2 in the image metadat | |
| 17:25:27 | sean-k-mooney | well yes it requesting 12 Gof hugepages on numa 0 | |
| 17:25:34 | sean-k-mooney | but you may not have 12G free | |
| 17:26:34 | spatel | This is i have currently hw:cpu_policy='dedicated', hw:mem_page_sizee='large', hw:numa_nodes='2' | |
| 17:27:34 | sean-k-mooney | yes so that will use hugepage,cpu pinning and create a vm with 2 numa nodes | |
| 17:27:46 | spatel | I have 32G memory total ( 16G per numa) | |
| 17:27:47 | sean-k-mooney | if you check cat /sys/devices/system/node/node*/meminfo | |
| 17:28:11 | sean-k-mooney | do you have 12G of free hugepages per node | |
| 17:29:10 | stephenfin | sean-k-mooney: Now with functional tests https://review.opendev.org/#/c/662522/ | |
| 17:29:25 | spatel | This is i have in my grub hugepagesz=2M hugepages=12288 | |
| 17:29:39 | stephenfin | Still need to figure out how I can rollback the changes to the field in the event that the resize fails but that's tomorrow's problem | |
| 17:29:53 | sean-k-mooney | stephenfin: cool i take it we never save the modifed request_spec? | |
| 17:30:04 | sean-k-mooney | stephenfin: well first confirm ^ | |
| 17:30:14 | sean-k-mooney | if we dont save it then your good | |
| 17:30:15 | stephenfin | not sure yet. I'll check that out first, yup | |
| 17:30:31 | spatel | sean-k-mooney: 25723 MB total huge page | |
| 17:31:55 | spatel | should i assume its divided between two numa node (25/2 = 12.4G) | |
| 17:32:07 | sean-k-mooney | spatel: ya that should give you ~ 12.5G per numa node but what does "cat /sys/devices/system/node/node*/meminfo | grep HugePages" show | |
| 17:34:20 | spatel | oh!!! | |
| 17:34:26 | spatel | 6G free | |
| 17:34:50 | sean-k-mooney | no | |
| 17:34:58 | sean-k-mooney | you default hugepage size it 2m | |
| 17:35:07 | sean-k-mooney | that is reported in pages not MBs | |
| 17:35:16 | spatel | oh, ok | |
| 17:35:22 | sean-k-mooney | so you have exactly 12G of hugepage per numanode | |
| 17:35:36 | spatel | should i create flavor with 23G? | |
| 17:35:45 | sean-k-mooney | i susspect that that would work yes | |
| 17:35:57 | spatel | let me try.. hold tight | |
| 17:36:19 | sean-k-mooney | i think we have an off by 1 issue when we woudl use it exactly | |
| 17:36:49 | sean-k-mooney | i know i have had issue wiht that in the past so i usally round up my allocation on the kernel slightly | |
| 17:37:23 | sean-k-mooney | so hugepagesz=2M hugepages=12300 in your case instead of hugepagesz=2M hugepages=12288 | |
| 17:42:33 | sean-k-mooney | spatel: if 23G works then then you are likely hitting the isssue were we cant consume every hugepage page which has been a thing forever. i thought that was fixed litrally years ago but we could have regressed. | |
| 17:45:44 | spatel | sean-k-mooney: it works!! instance is Up and running, I am in :) | |
| 17:46:32 | sean-k-mooney | spatel: as a workaround i would just increase your group settings 12300 instead of 12288 | |
| 17:46:46 | sean-k-mooney | 12288 is exactly 24G of hugepages | |
| 17:47:04 | spatel | hmm! | |
| 17:47:23 | sean-k-mooney | can you file a bug so we have something to track fixing the case where we use every single page in 1 vm | |
| 17:48:19 | spatel | can you explain me what went wrong in my case so it would be easy for me to open bug | |
| 17:49:02 | sean-k-mooney | so you had allocated 12288 2MB hugepages which is 24G exactly | |
| 17:49:22 | sean-k-mooney | and then you created a flavor that requested all 24G | |
| 17:49:31 | sean-k-mooney | e.g. 12288 hugpages | |
| 17:49:54 | sean-k-mooney | in this case i belive we have an off by 1 error where we say it can't fit when it can fit exactly | |
| 17:50:17 | sean-k-mooney | basically i think we are doign a < so where where it shoudl be <= | |
| 17:50:50 | spatel | opening bug now | |
| 17:51:34 | sean-k-mooney | i know this used to be a bug when we first intoduced hugepages back in like 2014 but its totally posibel that that bug has been reinotduce or that we just never got aroud to fixing it | |
| 17:55:14 | sean-k-mooney | this should be correct https://github.com/openstack/nova/blob/master/nova/objects/numa.py#L140-L157 but that said the error came from qemu so perhaps this is a qemu issue | |
| 18:02:46 | openstackgerrit | Merged openstack/nova master: db: Remove cell APIs https://review.opendev.org/651309 | |
| 18:04:46 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Workaround missing RequestSpec.instance_group.uuid https://review.opendev.org/663110 | |
| 18:06:12 | spatel | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1831652 | |
| 18:06:13 | openstack | Launchpad bug 1831652 in OpenStack Compute (nova) "fixing the case where we use every single page in 1 vm" [Undecided,New] | |
| 18:06:24 | spatel | sean-k-mooney: Thank you for help :) | |
| 18:08:17 | sean-k-mooney | spatel: no worries as i said the error seams to be coming form qemu so its likely that we fixed the nova issue already and qemu also has a bug but at least we have something to track figuring that out | |
| 18:30:18 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using q35 machine type https://review.opendev.org/663011 | |
| 18:30:19 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887 | |
| 18:40:39 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using q35 machine type https://review.opendev.org/663011 | |
| 18:40:40 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887 | |
| 18:50:56 | mriedem | dansmith: thanks replied on that pre-filter for disabled computes spec | |
| 18:51:48 | mriedem | tl;dr it sounds like don't worry about down services | |
| 19:05:41 | dansmith | mriedem: ack, replied, but yeah, that's MHO | |
| 19:06:57 | mriedem | regarding old computes that won't be reporting the trait but they get disabled in the api, what are your thoughts on that? the api manages the trait until the compute is upgraded, or just ignore it until upgraded, or add a sync CLI? | |
| 19:07:30 | mriedem | if we do'nt add a sync CLI for old computes that are already disabled, then i'd just ignore those requests in the API | |
| 19:10:11 | dansmith | neither | |
| 19:10:17 | dansmith | ignore until upgrade | |
| 19:10:42 | dansmith | we set disabled=true on the compute record, let the computefilter continue to exclude those at great expense | |
| 19:10:58 | dansmith | when the compute is upgraded and restarted, the u-p-t sync will update placement | |
| 19:11:11 | mriedem | upt won't set this trait | |
| 19:11:17 | dansmith | why? | |
| 19:11:32 | mriedem | well, i hadn't planned on adding that - it was an alternative but not one i'd baked into the proposed change | |
| 19:11:43 | dansmith | but why not? it solves a lot right? | |
| 19:12:02 | mriedem | solves as in upgrades, dropped calls, out of sync, etc? | |
| 19:12:05 | dansmith | it's kindof our "heal placement" loop as it is, so I don't see why we wouldn't | |
| 19:12:10 | dansmith | yeah | |
| 19:13:01 | mriedem | ok. in my reply i said we could build that in, so i might as well i guess, i'll move that out of alternatives. | |
| 19:31:57 | efried | melwitt: Is there a reason you're holding back from approving https://review.opendev.org/#/c/579897/ ? | |
| 19:34:06 | melwitt | efried: yeah, I hadn't reviewed deep into the change previously and don't know much about the context (the original feature). but since I had expressed a reno would be helpful, I reviewed the change and acked on that basis | |
| 19:34:44 | efried | okay, thanks. | |
| 19:34:47 | mriedem | efried: dustinc: i'm marking https://review.opendev.org/#/c/662881/ as a WIP since it's clearly a WIP | |
| 19:34:49 | efried | Not sure who we're looking for to send it | |
| 19:34:59 | efried | ack mriedem | |
| 19:35:31 | mriedem | efried: just spin the roulette wheel of nova cores | |
| 19:36:04 | efried | mriedem: Tempted to send it myself, adding my almost+2 to the testament of people who tested it live | |
| 19:36:25 | mriedem | probably shouldn't, | |
| 19:36:33 | mriedem | just wait for stephenfin or johnthetubaguy to take a look i'd say | |
| 19:36:39 | efried | I didn't touch much of it. But yeah. | |
| 19:37:09 | mriedem | https://www.youtube.com/watch?v=JGftIcp2SC0 ? | |