Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-04
16:29:35 dansmith mriedem: yeah reading your comments, I think what's being described _is_ what we discussed at the previous denver ptg
16:29:53 dansmith which *was* to have nova do the creation, but just based on some static profile listed in the flavor,
16:30:15 dansmith taking the nova api interaction part out
16:30:30 dansmith but that's akin to an attachment for cinder, which I think makes sense here
16:30:41 dansmith or a binding for neutron
16:30:56 mriedem well, you mean nova creating a volume or a port right?
16:31:02 mriedem on behalf of the user
16:31:09 dansmith no
16:31:34 dansmith because the complex configuration bits are wrapped up in the profile, right?
16:31:47 sean-k-mooney kind of
16:32:15 dansmith it's not a direct correlation, but nova currently does its own attaching of ports and volumes to a host once it knows where it's going, even if the complex volume or port was created, configured, etc by the user
16:32:20 dansmith that's the analog I think I'm making here
16:32:30 sean-k-mooney the profile is just a string in the nova flavor extra spec and wehn we ask cyborg for the detail of the profile it gives us back a set of resource requests and traits
16:32:44 dansmith right
16:32:59 sean-k-mooney and later when we bind the arq to a specifc host cyborge say "heres a pci device"
16:33:10 sean-k-mooney and we generagte the correct xml to pass it to the guest
16:33:36 dansmith right, which to me is equivalent to a host binding or cinder attachment
16:34:03 mriedem sure i get that,
16:34:08 mriedem it's not really what i'm talking about though
16:34:08 dansmith I really haven't read the spec in a while so I should probably keep my mouth shut,
16:34:19 mriedem nova has to do the binding either way,
16:34:36 sean-k-mooney yes but the profile is more like a neutron network in that we told you i want an arq of this type "or a port on this network" and nova is creating an new arq instance like it create a neutron port
16:34:42 openstackgerrit Sylvain Bauza proposed openstack/nova master: Pass allocations to virt drivers when resizing https://review.opendev.org/589085
16:34:45 mriedem i was just concerned about nova being responsible for creating the arq resource always rather than passing a pre-created arq id and just using that (which is already linked to a device profile)
16:34:49 sean-k-mooney i then binds that arq the same way it bind the prot
16:35:08 mriedem sean-k-mooney: right that's what i'm getting at,
16:35:35 mriedem and i worry that will get more complicated down the road (like SR-IOV ports) which requires the compute API to grow complexity to handle new types of devices
16:35:56 dansmith I think the benefit of not letting them pass an ARQ in first is that it delays our commitment to the api user until after this kinda actually works
16:35:58 sean-k-mooney mriedem: so in a way yes we are proxing the creation of the arq but we chose to do that to reduce the change in the api.
16:36:00 mriedem whereas for sriov ports we just said, nope, use the neutron api first if you want those
16:36:12 openstackgerrit John Garbutt proposed openstack/nova master: Add functional test for admin_actions https://review.opendev.org/657698
16:36:13 openstackgerrit John Garbutt proposed openstack/nova master: Ensure we pass a target in admin actions https://review.opendev.org/663095
16:36:13 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add scope check, see tests catch the change https://review.opendev.org/657823
16:36:53 sean-k-mooney dansmith: ya i think that was the basis of your original argument for this approch in the ptg session
16:37:06 mriedem sean-k-mooney: dansmith: ok, ack, i'll yield on that, and had said in my review that i'm sure this has been debated and discussed before and i'm just catching up on that, so don't want to block on it
16:37:10 sean-k-mooney e.g. to defer the api change until this actully works
16:37:18 dansmith sean-k-mooney: yep
16:37:39 mriedem based on all that the spec is probably mostly ready to go after Sundar cleans it up a bit
16:38:28 sean-k-mooney long term i think have a --device option on the server create commandline and the appropriate api change makes the most sense and allow the precreation of ports
16:40:43 mdbooth lyarwood: Done.
16:40:58 mriedem ok thanks, linked to this discussion for Sundar and once he cleans up i'll probably +2 the spec
16:49:19 mriedem dansmith: if there is one spec you review on this, the 3rd formal spec review sprint of the train release, it should be *my* spec for pre-filtering disabled computes https://review.opendev.org/#/c/657884/ :)
16:49:39 dansmith orly?
16:50:12 mriedem mostly b/c i have a quandary in there
16:53:44 openstackgerrit Merged openstack/nova stable/stein: Workaround missing RequestSpec.instance_group.uuid https://review.opendev.org/662894
16:54:08 sean-k-mooney mriedem: oh i thought your disabled nodes prefilter spec was already merged
17:11:43 openstackgerrit John Garbutt proposed openstack/nova master: WIP: add scope check, see tests catch the change https://review.opendev.org/657823
17:14:08 dansmith mriedem: I'm super confused about this
17:14:17 dansmith this spec seems to confuse disabledness and down-ness
17:14:20 dansmith I'm commenting
17:17:59 openstackgerrit Stephen Finucane proposed openstack/nova master: Recalculate 'RequestSpec.numa_topology' on resize https://review.opendev.org/662522
17:18:00 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Add '_setup_compute_services' helper https://review.opendev.org/663102
17:21:20 spatel Folks, i am getting this error qemu-kvm: -object memory-backend-ram,id=ram-node0,size=12884901888,host-nodes=0,policy=bind: cannot set up guest memory 'ram-node0': Cannot allocate memory
17:21:56 spatel I have 32G memory and hugemem is 25G
17:22:11 spatel i am trying to build VM with 24G and its throwing this error
17:23:18 sean-k-mooney spatel: do you have 24G of hugepages on a single numa node
17:23:47 spatel no
17:23:48 sean-k-mooney by default the kernel will split it across all numa nodes if you allcoate the hugepages on teh kernel commandline
17:24:06 spatel How do i tell my flavor to use both side of NUMA ?
17:24:33 sean-k-mooney am you have to create a guest with multiple numa nodes which is done by setting hw:numa_nodes=2
17:24:49 sean-k-mooney in the flavor
17:25:01 spatel I have that option already in flavor
17:25:02 sean-k-mooney or hw_numa_nodes=2 in the image metadat
17:25:27 sean-k-mooney well yes it requesting 12 Gof hugepages on numa 0
17:25:34 sean-k-mooney but you may not have 12G free
17:26:34 spatel This is i have currently hw:cpu_policy='dedicated', hw:mem_page_sizee='large', hw:numa_nodes='2'
17:27:34 sean-k-mooney yes so that will use hugepage,cpu pinning and create a vm with 2 numa nodes
17:27:46 spatel I have 32G memory total ( 16G per numa)
17:27:47 sean-k-mooney if you check cat /sys/devices/system/node/node*/meminfo
17:28:11 sean-k-mooney do you have 12G of free hugepages per node
17:29:10 stephenfin sean-k-mooney: Now with functional tests https://review.opendev.org/#/c/662522/
17:29:25 spatel This is i have in my grub hugepagesz=2M hugepages=12288
17:29:39 stephenfin Still need to figure out how I can rollback the changes to the field in the event that the resize fails but that's tomorrow's problem
17:29:53 sean-k-mooney stephenfin: cool i take it we never save the modifed request_spec?
17:30:04 sean-k-mooney stephenfin: well first confirm ^
17:30:14 sean-k-mooney if we dont save it then your good
17:30:15 stephenfin not sure yet. I'll check that out first, yup
17:30:31 spatel sean-k-mooney: 25723 MB total huge page
17:31:55 spatel should i assume its divided between two numa node (25/2 = 12.4G)
17:32:07 sean-k-mooney spatel: ya that should give you ~ 12.5G per numa node but what does "cat /sys/devices/system/node/node*/meminfo | grep HugePages" show
17:34:20 spatel oh!!!
17:34:26 spatel 6G free
17:34:50 sean-k-mooney no
17:34:58 sean-k-mooney you default hugepage size it 2m
17:35:07 sean-k-mooney that is reported in pages not MBs
17:35:16 spatel oh, ok
17:35:22 sean-k-mooney so you have exactly 12G of hugepage per numanode
17:35:36 spatel should i create flavor with 23G?
17:35:45 sean-k-mooney i susspect that that would work yes
17:35:57 spatel let me try.. hold tight
17:36:19 sean-k-mooney i think we have an off by 1 issue when we woudl use it exactly
17:36:49 sean-k-mooney i know i have had issue wiht that in the past so i usally round up my allocation on the kernel slightly
17:37:23 sean-k-mooney so hugepagesz=2M hugepages=12300 in your case instead of hugepagesz=2M hugepages=12288
17:42:33 sean-k-mooney spatel: if 23G works then then you are likely hitting the isssue were we cant consume every hugepage page which has been a thing forever. i thought that was fixed litrally years ago but we could have regressed.
17:45:44 spatel sean-k-mooney: it works!! instance is Up and running, I am in :)
17:46:32 sean-k-mooney spatel: as a workaround i would just increase your group settings 12300 instead of 12288
17:46:46 sean-k-mooney 12288 is exactly 24G of hugepages
17:47:04 spatel hmm!
17:47:23 sean-k-mooney can you file a bug so we have something to track fixing the case where we use every single page in 1 vm
17:48:19 spatel can you explain me what went wrong in my case so it would be easy for me to open bug

Earlier   Later