Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-04
16:34:49 sean-k-mooney i then binds that arq the same way it bind the prot
16:35:08 mriedem sean-k-mooney: right that's what i'm getting at,
16:35:35 mriedem and i worry that will get more complicated down the road (like SR-IOV ports) which requires the compute API to grow complexity to handle new types of devices
16:35:56 dansmith I think the benefit of not letting them pass an ARQ in first is that it delays our commitment to the api user until after this kinda actually works
16:35:58 sean-k-mooney mriedem: so in a way yes we are proxing the creation of the arq but we chose to do that to reduce the change in the api.
16:36:00 mriedem whereas for sriov ports we just said, nope, use the neutron api first if you want those
16:36:12 openstackgerrit John Garbutt proposed openstack/nova master: Add functional test for admin_actions https://review.opendev.org/657698
16:36:13 openstackgerrit John Garbutt proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095
16:36:13 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add scope check, see tests catch the change https://review.opendev.org/657823
16:36:53 sean-k-mooney dansmith: ya i think that was the basis of your original argument for this approch in the ptg session
16:37:06 mriedem sean-k-mooney: dansmith: ok, ack, i'll yield on that, and had said in my review that i'm sure this has been debated and discussed before and i'm just catching up on that, so don't want to block on it
16:37:10 sean-k-mooney e.g. to defer the api change until this actully works
16:37:18 dansmith sean-k-mooney: yep
16:37:39 mriedem based on all that the spec is probably mostly ready to go after Sundar cleans it up a bit
16:38:28 sean-k-mooney long term i think have a --device option on the server create commandline and the appropriate api change makes the most sense and allow the precreation of ports
16:40:43 mdbooth lyarwood: Done.
16:40:58 mriedem ok thanks, linked to this discussion for Sundar and once he cleans up i'll probably +2 the spec
16:49:19 mriedem dansmith: if there is one spec you review on this, the 3rd formal spec review sprint of the train release, it should be *my* spec for pre-filtering disabled computes https://review.opendev.org/#/c/657884/ :)
16:49:39 dansmith orly?
16:50:12 mriedem mostly b/c i have a quandary in there
16:53:44 openstackgerrit Merged openstack/nova stable/stein: Workaround missing RequestSpec.instance_group.uuid https://review.opendev.org/662894
16:54:08 sean-k-mooney mriedem: oh i thought your disabled nodes prefilter spec was already merged
17:11:43 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add scope check, see tests catch the change https://review.opendev.org/657823
17:14:08 dansmith mriedem: I'm super confused about this
17:14:17 dansmith this spec seems to confuse disabledness and down-ness
17:14:20 dansmith I'm commenting
17:17:59 openstackgerrit Stephen Finucane proposed openstack/nova master: Recalculate 'RequestSpec.numa_topology' on resize https://review.opendev.org/662522
17:18:00 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Add '_setup_compute_services' helper https://review.opendev.org/663102
17:21:20 spatel Folks, i am getting this error qemu-kvm: -object memory-backend-ram,id=ram-node0,size=12884901888,host-nodes=0,policy=bind: cannot set up guest memory 'ram-node0': Cannot allocate memory
17:21:56 spatel I have 32G memory and hugemem is 25G
17:22:11 spatel i am trying to build VM with 24G and its throwing this error
17:23:18 sean-k-mooney spatel: do you have 24G of hugepages on a single numa node
17:23:47 spatel no
17:23:48 sean-k-mooney by default the kernel will split it across all numa nodes if you allcoate the hugepages on teh kernel commandline
17:24:06 spatel How do i tell my flavor to use both side of NUMA ?
17:24:33 sean-k-mooney am you have to create a guest with multiple numa nodes which is done by setting hw:numa_nodes=2
17:24:49 sean-k-mooney in the flavor
17:25:01 spatel I have that option already in flavor
17:25:02 sean-k-mooney or hw_numa_nodes=2 in the image metadat
17:25:27 sean-k-mooney well yes it requesting 12 Gof hugepages on numa 0
17:25:34 sean-k-mooney but you may not have 12G free
17:26:34 spatel This is i have currently hw:cpu_policy='dedicated', hw:mem_page_sizee='large', hw:numa_nodes='2'
17:27:34 sean-k-mooney yes so that will use hugepage,cpu pinning and create a vm with 2 numa nodes
17:27:46 spatel I have 32G memory total ( 16G per numa)
17:27:47 sean-k-mooney if you check cat /sys/devices/system/node/node*/meminfo
17:28:11 sean-k-mooney do you have 12G of free hugepages per node
17:29:10 stephenfin sean-k-mooney: Now with functional tests https://review.opendev.org/#/c/662522/
17:29:25 spatel This is i have in my grub hugepagesz=2M hugepages=12288
17:29:39 stephenfin Still need to figure out how I can rollback the changes to the field in the event that the resize fails but that's tomorrow's problem
17:29:53 sean-k-mooney stephenfin: cool i take it we never save the modifed request_spec?
17:30:04 sean-k-mooney stephenfin: well first confirm ^
17:30:14 sean-k-mooney if we dont save it then your good
17:30:15 stephenfin not sure yet. I'll check that out first, yup
17:30:31 spatel sean-k-mooney: 25723 MB total huge page
17:31:55 spatel should i assume its divided between two numa node (25/2 = 12.4G)
17:32:07 sean-k-mooney spatel: ya that should give you ~ 12.5G per numa node but what does "cat /sys/devices/system/node/node*/meminfo | grep HugePages" show
17:34:20 spatel oh!!!
17:34:26 spatel 6G free
17:34:50 sean-k-mooney no
17:34:58 sean-k-mooney you default hugepage size it 2m
17:35:07 sean-k-mooney that is reported in pages not MBs
17:35:16 spatel oh, ok
17:35:22 sean-k-mooney so you have exactly 12G of hugepage per numanode
17:35:36 spatel should i create flavor with 23G?
17:35:45 sean-k-mooney i susspect that that would work yes
17:35:57 spatel let me try.. hold tight
17:36:19 sean-k-mooney i think we have an off by 1 issue when we woudl use it exactly
17:36:49 sean-k-mooney i know i have had issue wiht that in the past so i usally round up my allocation on the kernel slightly
17:37:23 sean-k-mooney so hugepagesz=2M hugepages=12300 in your case instead of hugepagesz=2M hugepages=12288
17:42:33 sean-k-mooney spatel: if 23G works then then you are likely hitting the isssue were we cant consume every hugepage page which has been a thing forever. i thought that was fixed litrally years ago but we could have regressed.
17:45:44 spatel sean-k-mooney: it works!! instance is Up and running, I am in :)
17:46:32 sean-k-mooney spatel: as a workaround i would just increase your group settings 12300 instead of 12288
17:46:46 sean-k-mooney 12288 is exactly 24G of hugepages
17:47:04 spatel hmm!
17:47:23 sean-k-mooney can you file a bug so we have something to track fixing the case where we use every single page in 1 vm
17:48:19 spatel can you explain me what went wrong in my case so it would be easy for me to open bug
17:49:02 sean-k-mooney so you had allocated 12288 2MB hugepages which is 24G exactly
17:49:22 sean-k-mooney and then you created a flavor that requested all 24G
17:49:31 sean-k-mooney e.g. 12288 hugpages
17:49:54 sean-k-mooney in this case i belive we have an off by 1 error where we say it can't fit when it can fit exactly
17:50:17 sean-k-mooney basically i think we are doign a < so where where it shoudl be <=
17:50:50 spatel opening bug now
17:51:34 sean-k-mooney i know this used to be a bug when we first intoduced hugepages back in like 2014 but its totally posibel that that bug has been reinotduce or that we just never got aroud to fixing it
17:55:14 sean-k-mooney this should be correct https://github.com/openstack/nova/blob/master/nova/objects/numa.py#L140-L157 but that said the error came from qemu so perhaps this is a qemu issue
18:02:46 openstackgerrit Merged openstack/nova master: db: Remove cell APIs https://review.opendev.org/651309
18:04:46 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Workaround missing RequestSpec.instance_group.uuid https://review.opendev.org/663110
18:06:12 spatel sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1831652
18:06:13 openstack Launchpad bug 1831652 in OpenStack Compute (nova) "fixing the case where we use every single page in 1 vm" [Undecided,New]
18:06:24 spatel sean-k-mooney: Thank you for help :)
18:08:17 sean-k-mooney spatel: no worries as i said the error seams to be coming form qemu so its likely that we fixed the nova issue already and qemu also has a bug but at least we have something to track figuring that out
18:30:18 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using q35 machine type https://review.opendev.org/663011
18:30:19 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887
18:40:39 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Use SATA bus for cdrom devices when using q35 machine type https://review.opendev.org/663011
18:40:40 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM: Run tempest-full-py3 with q35 machine type https://review.opendev.org/662887
18:50:56 mriedem dansmith: thanks replied on that pre-filter for disabled computes spec
18:51:48 mriedem tl;dr it sounds like don't worry about down services
19:05:41 dansmith mriedem: ack, replied, but yeah, that's MHO
19:06:57 mriedem regarding old computes that won't be reporting the trait but they get disabled in the api, what are your thoughts on that? the api manages the trait until the compute is upgraded, or just ignore it until upgraded, or add a sync CLI?
19:07:30 mriedem if we do'nt add a sync CLI for old computes that are already disabled, then i'd just ignore those requests in the API
19:10:11 dansmith neither

Earlier   Later