| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-31 | |||
| 17:49:12 | Sundar | But I agree it is tough to predict when it'll land. | |
| 17:49:44 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider config file schema and loader https://review.opendev.org/673341 | |
| 17:52:03 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider config file schema and loader https://review.opendev.org/673341 | |
| 18:07:44 | mriedem | melwitt: looks like https://review.opendev.org/#/c/668123/3 is ready to go in queens | |
| 18:07:50 | mriedem | lee was wanting that one | |
| 18:09:20 | melwitt | a-ha, cool. will review | |
| 18:10:21 | openstackgerrit | Merged openstack/nova master: Get rid of args to RBDDriver.__init__() https://review.opendev.org/668564 | |
| 18:15:21 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Handle alternative UEFI firmware binary paths https://review.opendev.org/348394 | |
| 18:15:21 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/673790 | |
| 18:23:53 | nafiux | Team, what should I do to properly configure the overcommit of resources? I’m getting this error: “Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.” | |
| 18:30:04 | mriedem | efried_rollin: since i'm not sure how this whole series works, i didn't vote on https://review.opendev.org/#/c/671072/2 but not sure if that should be using a set or list for forbidden_aggregates | |
| 18:30:33 | mriedem | nafiux: you said you listed some resource providers, did you check their inventories to see that they have the allocation_ratio values you configured in nova? you also didn't say what release you were using. | |
| 18:30:47 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list | |
| 18:31:55 | nafiux | Thanks @mriedem. I’m using stein. Let me take a look on that documentation. | |
| 18:33:06 | nafiux | Here what I have: https://pastebin.com/MdXsJYs3 | |
| 18:33:13 | mriedem | nafiux: and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios if you didn't read that yet | |
| 18:33:34 | spatel | sean-k-mooney: are you there? | |
| 18:33:45 | mriedem | yeah so your compute nodes configured ram_allocation_ratio is being reported in the MEMORY_MB inventory for the provider | |
| 18:33:46 | mriedem | | MEMORY_MB | 5.0 | 49139 | 512 | 1 | 1 | 49139 | | |
| 18:33:49 | mriedem | that looks fine | |
| 18:33:56 | sean-k-mooney | spatel: kind of yes | |
| 18:34:08 | mriedem | nafiux: is this a fresh deploy or do you already have some servers on that node? | |
| 18:35:02 | nafiux | It’s a fresh deployment, I don’t have any instance running at this point. I used kolla-ansible. I checked the /etc/nova/nova.conf inside the nova_scheduler container and I was able to see the allocation values: | |
| 18:35:11 | nafiux | docker exec -it nova_scheduler bash | |
| 18:35:20 | spatel | Good, I have machine with 64G and with this setting in grub hugepagesz=2M hugepages=27500 | |
| 18:35:49 | spatel | I have this flavor setting - properties | hw:cpu_policy='dedicated', hw:mem_page_size='large', hw:numa_nodes='2' | |
| 18:35:51 | nafiux | vi /etc/nova/nova.conf, and I can see inside [DEFAULT] the cpu_allocation_ratio = 16.0 and ram_allocation_ratio = 5.0 | |
| 18:36:13 | nafiux | I’m able to launch a server with 16GB of Ram and 8 VPCU, but I’m not able to launch a second one. | |
| 18:36:21 | nafiux | Let me launch one, and provide you the report again. | |
| 18:37:14 | spatel | flavor has 8vcpu / 12G memory | |
| 18:37:38 | spatel | trying to create 2 VM but getting error, but i am able to create 1 vm ( not able to create 2) | |
| 18:38:39 | sean-k-mooney | you should have enough hugepages although the kerenl may not alwasys blance it perfectly | |
| 18:38:58 | sean-k-mooney | you have enough clearance however that that should not be the issue | |
| 18:39:09 | sean-k-mooney | how many cpu to you have | |
| 18:39:29 | spatel | 32 core | |
| 18:39:53 | nafiux | @mriedem: I was able to launch the first one, but whe I try to launch the second one I got the “Got no allocation candidates from the Placement API.” error | |
| 18:40:39 | sean-k-mooney | you have ~20G free per numa node and you need 12 so it should not be hugepage related | |
| 18:41:15 | sean-k-mooney | you have 32 cores is htat 2 16 core cpus or 2 8 core cpus with hyperthread or 1 32 core cpus | |
| 18:41:25 | mriedem | nafiux: how much disk in the flavor? or are these volume-backed servers? | |
| 18:41:50 | mriedem | nafiux: the placement-api also logs the allocation candidate filtering it does so you should trace the requestid through the placement logs - might need to enable debug | |
| 18:41:58 | spatel | sean-k-mooney: that is what i am trying to find.. what is wrong here :( | |
| 18:42:00 | sean-k-mooney | spatel: it would be useful to know what the vcpu_pin_set is set to in your nova.conf | |
| 18:42:02 | mriedem | that should tell you why it kicked out the provider | |
| 18:42:43 | nafiux | Ouch, I think that is it! Is the hard disk. Which actually doesn’t overcommit. | |
| 18:42:54 | nafiux | I have assigned a lot in the flavor, let me reduce it and test it again. | |
| 18:43:13 | mriedem | yeah your provider only has ~300 GB of disk | |
| 18:43:23 | spatel | grep vcpu_pin_set /etc/nova/nova.conf <--- empty | |
| 18:43:42 | spatel | I have queen cluster and same config works there... | |
| 18:43:48 | mriedem | nafiux: if you were doing volume-backed servers then you're probably fine, lots of vcpu and ram on that node | |
| 18:43:48 | spatel | i am running new cluster in stein | |
| 18:43:50 | mriedem | but very little disk | |
| 18:43:55 | spatel | which causing issue | |
| 18:44:45 | nafiux | @mriedem yes, that is true, only 300 GB free, and the flavor has 160 GB, so only 1 instance with that capacity will be allowed. | |
| 18:45:04 | mriedem | \o/ | |
| 18:46:36 | nafiux | Does "volume-backed" means the actuall physical hard disk of the hypervisor/nova node? | |
| 18:46:48 | mriedem | no | |
| 18:46:55 | mriedem | means the root disk is a cinder volume | |
| 18:47:11 | mriedem | https://docs.openstack.org/nova/latest/user/launch-instance-from-volume.html | |
| 18:48:09 | nafiux | Oh, I don’t have installed cinder. That will help a lot, if I understand correctly? | |
| 18:48:31 | mriedem | i don't know how to answer that question :) | |
| 18:48:46 | mriedem | if you have a compute node with 300 GB of disk and flavors with 160 GB of disk, you're not going to get far | |
| 18:48:54 | mriedem | so you'd have to carve down your flavors to have smaller disk, | |
| 18:49:00 | mriedem | or add more storage to your compute nodes, | |
| 18:49:07 | mriedem | or use cinder to back the root disk | |
| 18:50:02 | nafiux | Lol, yeah. How servers backed with cinder volumes behave? Do they use as needed and not the actual size specified at creation time? | |
| 18:53:13 | mriedem | for volume-backed servers the root gb on the flavor doesn't really apply | |
| 18:53:34 | mriedem | the size of the volume is whatever it is when it's created | |
| 18:53:56 | sean-k-mooney | mriedem: im not sure i did a good job of explaing but it tried to explain the issue again im my reponce to your comments | |
| 18:54:00 | sean-k-mooney | https://review.opendev.org/#/c/669738/2 | |
| 18:54:01 | mriedem | the user can pre-create a volume and provide that to nova when creating the server, or the user can give some parameters to nova and nova will create the volume and attach it to the server | |
| 18:54:16 | sean-k-mooney | but yes i can factor the functional test our into there own regression module | |
| 18:54:27 | mriedem | so many words....but i started that | |
| 18:54:29 | sean-k-mooney | and add teh rebuild/shelve tests | |
| 18:55:31 | mriedem | nafiux: some people use cinder exclusively for root volumes of their servers, some people use shared storage for their compute nodes, some mix both - it takes all kinds | |
| 18:57:56 | sean-k-mooney | nafiux: cinder backed volume behvior depend on teh dirver. some cinder backends support thin provisioning allowing over commit some dont | |
| 18:58:49 | sean-k-mooney | and some time the same dirver change behavior based on your config setting which makes it all the more fun to debug | |
| 18:58:59 | nafiux | Thanks @mridem & @sean-k-mooney. I will read more about those topics, so, in general which are the different options for volumens to the instances? | |
| 19:01:11 | sean-k-mooney | there are two sets of volumes, volumes that have independent lifetime form the vm usually called data volumes | |
| 19:01:20 | sean-k-mooney | thos are create via teh cinder api. | |
| 19:01:40 | sean-k-mooney | and then there are volumes that have ther life time cupled to a vm | |
| 19:02:07 | nafiux | Ok, which is the equivalent to the root volumes in AWS? | |
| 19:02:16 | nafiux | Can I have a root volume with cinder? | |
| 19:02:20 | sean-k-mooney | yes | |
| 19:02:31 | sean-k-mooney | we call that boot form volume or bfv for short | |
| 19:03:01 | sean-k-mooney | in that case you tell nova that the root disk should not be create using the compute nodes default driver and instad should be provide by cinder | |
| 19:03:16 | nafiux | Oh, so, due my current setup for learning, I will leverage cinder for both bfv and additional volumes as needed | |
| 19:03:17 | sean-k-mooney | you can optionl specifythat that volume should be deleted when the vm is or not | |
| 19:04:05 | sean-k-mooney | i wasnt following the full conversation but you can use cinder for both | |
| 19:04:22 | nafiux | Ok, yea, is just because I have a lot of CPU & Memory but few hard disk | |
| 19:04:40 | sean-k-mooney | the default backend for non bfv guest is to create a local file on the compute node for the root disk of the instance | |
| 19:05:12 | sean-k-mooney | you can also tell the compute node to use a differnt default image backend such as ceph | |
| 19:05:46 | sean-k-mooney | in which case nova will create a remote ceph volume out of band of cinder | |
| 19:06:16 | nafiux | Ok, I need to read a lot :D cinder, ceph, and will back later with more specific questions, this is really amazing. | |
| 19:06:18 | sean-k-mooney | other peopl in a similar situation somethime mount the default location where libvirt stores its disk image on nfs | |
| 19:07:32 | sean-k-mooney | so as mriedem said there are lots of way to do it and people do many different things but its better to keep it simple when learning | |
| 19:08:13 | nafiux | Agree. Thanks for the support. | |
| 19:11:28 | mriedem | nafiux: more info on cinder in their docs https://docs.openstack.org/cinder/latest/ | |
| 19:11:46 | mriedem | there is also the ops guide which is older but relevant https://docs.openstack.org/operations-guide/ | |
| 19:12:41 | sean-k-mooney | or if you like video content there are way more videos then you will have time to watch on teh offical youtube channel | |
| 19:12:43 | sean-k-mooney | https://www.youtube.com/user/OpenStackFoundation | |