Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-04
17:23:48 sean-k-mooney by default the kernel will split it across all numa nodes if you allcoate the hugepages on teh kernel commandline
17:24:06 spatel How do i tell my flavor to use both side of NUMA ?
17:24:33 sean-k-mooney am you have to create a guest with multiple numa nodes which is done by setting hw:numa_nodes=2
17:24:49 sean-k-mooney in the flavor
17:25:01 spatel I have that option already in flavor
17:25:02 sean-k-mooney or hw_numa_nodes=2 in the image metadat
17:25:27 sean-k-mooney well yes it requesting 12 Gof hugepages on numa 0
17:25:34 sean-k-mooney but you may not have 12G free
17:26:34 spatel This is i have currently hw:cpu_policy='dedicated', hw:mem_page_sizee='large', hw:numa_nodes='2'
17:27:34 sean-k-mooney yes so that will use hugepage,cpu pinning and create a vm with 2 numa nodes
17:27:46 spatel I have 32G memory total ( 16G per numa)
17:27:47 sean-k-mooney if you check cat /sys/devices/system/node/node*/meminfo
17:28:11 sean-k-mooney do you have 12G of free hugepages per node
17:29:10 stephenfin sean-k-mooney: Now with functional tests https://review.opendev.org/#/c/662522/
17:29:25 spatel This is i have in my grub hugepagesz=2M hugepages=12288
17:29:39 stephenfin Still need to figure out how I can rollback the changes to the field in the event that the resize fails but that's tomorrow's problem
17:29:53 sean-k-mooney stephenfin: cool i take it we never save the modifed request_spec?
17:30:04 sean-k-mooney stephenfin: well first confirm ^
17:30:14 sean-k-mooney if we dont save it then your good
17:30:15 stephenfin not sure yet. I'll check that out first, yup
17:30:31 spatel sean-k-mooney: 25723 MB total huge page
17:31:55 spatel should i assume its divided between two numa node (25/2 = 12.4G)
17:32:07 sean-k-mooney spatel: ya that should give you ~ 12.5G per numa node but what does "cat /sys/devices/system/node/node*/meminfo | grep HugePages" show
17:34:20 spatel oh!!!
17:34:26 spatel 6G free
17:34:50 sean-k-mooney no
17:34:58 sean-k-mooney you default hugepage size it 2m
17:35:07 sean-k-mooney that is reported in pages not MBs
17:35:16 spatel oh, ok
17:35:22 sean-k-mooney so you have exactly 12G of hugepage per numanode
17:35:36 spatel should i create flavor with 23G?
17:35:45 sean-k-mooney i susspect that that would work yes
17:35:57 spatel let me try.. hold tight
17:36:19 sean-k-mooney i think we have an off by 1 issue when we woudl use it exactly
17:36:49 sean-k-mooney i know i have had issue wiht that in the past so i usally round up my allocation on the kernel slightly
17:37:23 sean-k-mooney so hugepagesz=2M hugepages=12300 in your case instead of hugepagesz=2M hugepages=12288
17:42:33 sean-k-mooney spatel: if 23G works then then you are likely hitting the isssue were we cant consume every hugepage page which has been a thing forever. i thought that was fixed litrally years ago but we could have regressed.
17:45:44 spatel sean-k-mooney: it works!! instance is Up and running, I am in :)
17:46:32 sean-k-mooney spatel: as a workaround i would just increase your group settings 12300 instead of 12288
17:46:46 sean-k-mooney 12288 is exactly 24G of hugepages
17:47:04 spatel hmm!
17:47:23 sean-k-mooney can you file a bug so we have something to track fixing the case where we use every single page in 1 vm
17:48:19 spatel can you explain me what went wrong in my case so it would be easy for me to open bug
17:49:02 sean-k-mooney so you had allocated 12288 2MB hugepages which is 24G exactly
17:49:22 sean-k-mooney and then you created a flavor that requested all 24G
17:49:31 sean-k-mooney e.g. 12288 hugpages
17:49:54 sean-k-mooney in this case i belive we have an off by 1 error where we say it can't fit when it can fit exactly
17:50:17 sean-k-mooney basically i think we are doign a < so where where it shoudl be <=
17:50:50 spatel opening bug now
17:51:34 sean-k-mooney i know this used to be a bug when we first intoduced hugepages back in like 2014 but its totally posibel that that bug has been reinotduce or that we just never got aroud to fixing it
17:55:14 sean-k-mooney this should be correct https://github.com/openstack/nova/blob/master/nova/objects/numa.py#L140-L157 but that said the error came from qemu so perhaps this is a qemu issue
18:02:46 openstackgerrit Merged openstack/nova master: db: Remove cell APIs https://review.opendev.org/651309
18:04:46 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Workaround missing RequestSpec.instance_group.uuid https://review.opendev.org/663110
18:06:12 spatel sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1831652
18:06:13 openstack Launchpad bug 1831652 in OpenStack Compute (nova) "fixing the case where we use every single page in 1 vm" [Undecided,New]
18:06:24 spatel sean-k-mooney: Thank you for help :)
18:08:17 sean-k-mooney spatel: no worries as i said the error seams to be coming form qemu so its likely that we fixed the nova issue already and qemu also has a bug but at least we have something to track figuring that out
18:30:18 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using q35 machine type https://review.opendev.org/663011
18:30:19 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887
18:40:39 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using q35 machine type https://review.opendev.org/663011
18:40:40 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887
18:50:56 mriedem dansmith: thanks replied on that pre-filter for disabled computes spec
18:51:48 mriedem tl;dr it sounds like don't worry about down services
19:05:41 dansmith mriedem: ack, replied, but yeah, that's MHO
19:06:57 mriedem regarding old computes that won't be reporting the trait but they get disabled in the api, what are your thoughts on that? the api manages the trait until the compute is upgraded, or just ignore it until upgraded, or add a sync CLI?
19:07:30 mriedem if we do'nt add a sync CLI for old computes that are already disabled, then i'd just ignore those requests in the API
19:10:11 dansmith neither
19:10:17 dansmith ignore until upgrade
19:10:42 dansmith we set disabled=true on the compute record, let the computefilter continue to exclude those at great expense
19:10:58 dansmith when the compute is upgraded and restarted, the u-p-t sync will update placement
19:11:11 mriedem upt won't set this trait
19:11:17 dansmith why?
19:11:32 mriedem well, i hadn't planned on adding that - it was an alternative but not one i'd baked into the proposed change
19:11:43 dansmith but why not? it solves a lot right?
19:12:02 mriedem solves as in upgrades, dropped calls, out of sync, etc?
19:12:05 dansmith it's kindof our "heal placement" loop as it is, so I don't see why we wouldn't
19:12:10 dansmith yeah
19:13:01 mriedem ok. in my reply i said we could build that in, so i might as well i guess, i'll move that out of alternatives.
19:31:57 efried melwitt: Is there a reason you're holding back from approving https://review.opendev.org/#/c/579897/ ?
19:34:06 melwitt efried: yeah, I hadn't reviewed deep into the change previously and don't know much about the context (the original feature). but since I had expressed a reno would be helpful, I reviewed the change and acked on that basis
19:34:44 efried okay, thanks.
19:34:47 mriedem efried: dustinc: i'm marking https://review.opendev.org/#/c/662881/ as a WIP since it's clearly a WIP
19:34:49 efried Not sure who we're looking for to send it
19:34:59 efried ack mriedem
19:35:31 mriedem efried: just spin the roulette wheel of nova cores
19:36:04 efried mriedem: Tempted to send it myself, adding my almost+2 to the testament of people who tested it live
19:36:25 mriedem probably shouldn't,
19:36:33 mriedem just wait for stephenfin or johnthetubaguy to take a look i'd say
19:36:39 efried I didn't touch much of it. But yeah.
19:37:09 mriedem https://www.youtube.com/watch?v=JGftIcp2SC0 ?
19:37:26 dansmith rock on
19:37:56 melwitt lol, a song for every occasion
19:38:36 mriedem bon was ahead of his time when it came to collaborative software development
19:39:42 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Add regression recreate test for bug 1830747 https://review.opendev.org/663124
19:39:43 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Workaround missing RequestSpec.instance_group.uuid https://review.opendev.org/663125
19:39:44 openstack bug 1830747 in OpenStack Compute (nova) rocky "Error 500 trying to migrate an instance after wrong request_spec" [High,In progress] https://launchpad.net/bugs/1830747 - Assigned to Matt Riedemann (mriedem)
19:40:08 sean-k-mooney efried: for what its worth i think the hypervior hiding chance makes sense the only reservation i have is nvida or other may work around this in the future so i tond want us to have to keep addign these workaround but for this specific case i think it makes sense
19:44:13 dustinc mriedem: thanks, I meant to put WIP in the title
20:13:45 efried dustinc: In case it's not clear, if it looks like I'm answering questions in the sdk spec, me (or anyone) answering in comments isn't the end of the road - the salient points should be included in the document itself.
20:27:03 dustinc efried: very clear, thanks

Earlier   Later