| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-31 | |||
| 17:27:16 | melwitt | if there's a chance for request_spec to be None in reality, then that would work. if not, it's not really ok to do something in the impl just to get around unit tests that should be fixed. I'm not as well-versed about request_spec as mriedem, he probably would know most whether request_spec could possibly be None in this case | |
| 17:31:08 | Sundar | melwitt: Kinda expected that response, but tried my luck anyway. ;) Thank you. | |
| 17:40:47 | mriedem | Sundar: for tests that hit this code https://review.opendev.org/#/c/631244/25/nova/compute/manager.py@1865 you should be passing a real RequestSpec object. for *existing* tests hitting that code, if they are passing {} or None or something, then they need to be updated to pass either a RequestSpec object or a Mock | |
| 17:41:00 | mriedem | or, to simply mock out _create_and_bind_arqs | |
| 17:41:07 | mriedem | since those existing tests don't care about the details of that method | |
| 17:41:15 | mriedem | and your *new* tests would call _create_and_bind_arqs directly | |
| 17:41:17 | mriedem | with a RequestSpec object | |
| 17:41:32 | mriedem | - Failed: 159 | |
| 17:41:34 | mriedem | yikes | |
| 17:41:51 | mriedem | i'm guessing that's a lot of cumulative failures since it's later in the series | |
| 17:42:06 | mriedem | i.e. https://review.opendev.org/#/c/631243/27 | |
| 17:45:07 | Sundar | mriedem: Thanks. I counted 272 errors of that type over all. Mocking _create_and_bind_arqs is certainly easier. | |
| 17:45:27 | melwitt | code review | |
| 17:45:27 | melwitt | Sundar: a bit of advice, what I would do in this situation is write the impl and get the tempest tests passing (I assume something has been or needs to be set up that runs cyborg in it) first, get some review feedback once tempest coverage is passing, then fix the unit test and func test failures after you get some code review. that way you don't have to go through the work to fix hundreds of tests when the impl might change during | |
| 17:46:43 | mriedem | yeah, or just mock out the thing semi-globally to start, | |
| 17:47:08 | mriedem | e.g. in the patch that is failing tests on _create_and_bind_arqs, just stub that out in the test class setUp | |
| 17:47:41 | mriedem | melwitt: the tricky thing with tempest and this series is there is no tempest coverage with cyborg yet, | |
| 17:47:47 | mriedem | when there is, it'll be a fake cyborg driver, | |
| 17:47:53 | mriedem | but cyborg isn't hooked up with all of that yet, not even placement | |
| 17:47:59 | mriedem | so tempest is a loooong ways off | |
| 17:48:03 | melwitt | oh :\ ok | |
| 17:48:27 | Sundar | melwitt: Makes sense. mriedem: We are setting up tempest: https://review.opendev.org/#/c/670999/ | |
| 17:49:12 | Sundar | But I agree it is tough to predict when it'll land. | |
| 17:49:44 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider config file schema and loader https://review.opendev.org/673341 | |
| 17:52:03 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider config file schema and loader https://review.opendev.org/673341 | |
| 18:07:44 | mriedem | melwitt: looks like https://review.opendev.org/#/c/668123/3 is ready to go in queens | |
| 18:07:50 | mriedem | lee was wanting that one | |
| 18:09:20 | melwitt | a-ha, cool. will review | |
| 18:10:21 | openstackgerrit | Merged openstack/nova master: Get rid of args to RBDDriver.__init__() https://review.opendev.org/668564 | |
| 18:15:21 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/673790 | |
| 18:15:21 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Handle alternative UEFI firmware binary paths https://review.opendev.org/348394 | |
| 18:23:53 | nafiux | Team, what should I do to properly configure the overcommit of resources? I’m getting this error: “Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.” | |
| 18:30:04 | mriedem | efried_rollin: since i'm not sure how this whole series works, i didn't vote on https://review.opendev.org/#/c/671072/2 but not sure if that should be using a set or list for forbidden_aggregates | |
| 18:30:33 | mriedem | nafiux: you said you listed some resource providers, did you check their inventories to see that they have the allocation_ratio values you configured in nova? you also didn't say what release you were using. | |
| 18:30:47 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list | |
| 18:31:55 | nafiux | Thanks @mriedem. I’m using stein. Let me take a look on that documentation. | |
| 18:33:06 | nafiux | Here what I have: https://pastebin.com/MdXsJYs3 | |
| 18:33:13 | mriedem | nafiux: and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios if you didn't read that yet | |
| 18:33:34 | spatel | sean-k-mooney: are you there? | |
| 18:33:45 | mriedem | yeah so your compute nodes configured ram_allocation_ratio is being reported in the MEMORY_MB inventory for the provider | |
| 18:33:46 | mriedem | | MEMORY_MB | 5.0 | 49139 | 512 | 1 | 1 | 49139 | | |
| 18:33:49 | mriedem | that looks fine | |
| 18:33:56 | sean-k-mooney | spatel: kind of yes | |
| 18:34:08 | mriedem | nafiux: is this a fresh deploy or do you already have some servers on that node? | |
| 18:35:02 | nafiux | It’s a fresh deployment, I don’t have any instance running at this point. I used kolla-ansible. I checked the /etc/nova/nova.conf inside the nova_scheduler container and I was able to see the allocation values: | |
| 18:35:11 | nafiux | docker exec -it nova_scheduler bash | |
| 18:35:20 | spatel | Good, I have machine with 64G and with this setting in grub hugepagesz=2M hugepages=27500 | |
| 18:35:49 | spatel | I have this flavor setting - properties | hw:cpu_policy='dedicated', hw:mem_page_size='large', hw:numa_nodes='2' | |
| 18:35:51 | nafiux | vi /etc/nova/nova.conf, and I can see inside [DEFAULT] the cpu_allocation_ratio = 16.0 and ram_allocation_ratio = 5.0 | |
| 18:36:13 | nafiux | I’m able to launch a server with 16GB of Ram and 8 VPCU, but I’m not able to launch a second one. | |
| 18:36:21 | nafiux | Let me launch one, and provide you the report again. | |
| 18:37:14 | spatel | flavor has 8vcpu / 12G memory | |
| 18:37:38 | spatel | trying to create 2 VM but getting error, but i am able to create 1 vm ( not able to create 2) | |
| 18:38:39 | sean-k-mooney | you should have enough hugepages although the kerenl may not alwasys blance it perfectly | |
| 18:38:58 | sean-k-mooney | you have enough clearance however that that should not be the issue | |
| 18:39:09 | sean-k-mooney | how many cpu to you have | |
| 18:39:29 | spatel | 32 core | |
| 18:39:53 | nafiux | @mriedem: I was able to launch the first one, but whe I try to launch the second one I got the “Got no allocation candidates from the Placement API.” error | |
| 18:40:39 | sean-k-mooney | you have ~20G free per numa node and you need 12 so it should not be hugepage related | |
| 18:41:15 | sean-k-mooney | you have 32 cores is htat 2 16 core cpus or 2 8 core cpus with hyperthread or 1 32 core cpus | |
| 18:41:25 | mriedem | nafiux: how much disk in the flavor? or are these volume-backed servers? | |
| 18:41:50 | mriedem | nafiux: the placement-api also logs the allocation candidate filtering it does so you should trace the requestid through the placement logs - might need to enable debug | |
| 18:41:58 | spatel | sean-k-mooney: that is what i am trying to find.. what is wrong here :( | |
| 18:42:00 | sean-k-mooney | spatel: it would be useful to know what the vcpu_pin_set is set to in your nova.conf | |
| 18:42:02 | mriedem | that should tell you why it kicked out the provider | |
| 18:42:43 | nafiux | Ouch, I think that is it! Is the hard disk. Which actually doesn’t overcommit. | |
| 18:42:54 | nafiux | I have assigned a lot in the flavor, let me reduce it and test it again. | |
| 18:43:13 | mriedem | yeah your provider only has ~300 GB of disk | |
| 18:43:23 | spatel | grep vcpu_pin_set /etc/nova/nova.conf <--- empty | |
| 18:43:42 | spatel | I have queen cluster and same config works there... | |
| 18:43:48 | spatel | i am running new cluster in stein | |
| 18:43:48 | mriedem | nafiux: if you were doing volume-backed servers then you're probably fine, lots of vcpu and ram on that node | |
| 18:43:50 | mriedem | but very little disk | |
| 18:43:55 | spatel | which causing issue | |
| 18:44:45 | nafiux | @mriedem yes, that is true, only 300 GB free, and the flavor has 160 GB, so only 1 instance with that capacity will be allowed. | |
| 18:45:04 | mriedem | \o/ | |
| 18:46:36 | nafiux | Does "volume-backed" means the actuall physical hard disk of the hypervisor/nova node? | |
| 18:46:48 | mriedem | no | |
| 18:46:55 | mriedem | means the root disk is a cinder volume | |
| 18:47:11 | mriedem | https://docs.openstack.org/nova/latest/user/launch-instance-from-volume.html | |
| 18:48:09 | nafiux | Oh, I don’t have installed cinder. That will help a lot, if I understand correctly? | |
| 18:48:31 | mriedem | i don't know how to answer that question :) | |
| 18:48:46 | mriedem | if you have a compute node with 300 GB of disk and flavors with 160 GB of disk, you're not going to get far | |
| 18:48:54 | mriedem | so you'd have to carve down your flavors to have smaller disk, | |
| 18:49:00 | mriedem | or add more storage to your compute nodes, | |
| 18:49:07 | mriedem | or use cinder to back the root disk | |
| 18:50:02 | nafiux | Lol, yeah. How servers backed with cinder volumes behave? Do they use as needed and not the actual size specified at creation time? | |
| 18:53:13 | mriedem | for volume-backed servers the root gb on the flavor doesn't really apply | |
| 18:53:34 | mriedem | the size of the volume is whatever it is when it's created | |
| 18:53:56 | sean-k-mooney | mriedem: im not sure i did a good job of explaing but it tried to explain the issue again im my reponce to your comments | |
| 18:54:00 | sean-k-mooney | https://review.opendev.org/#/c/669738/2 | |
| 18:54:01 | mriedem | the user can pre-create a volume and provide that to nova when creating the server, or the user can give some parameters to nova and nova will create the volume and attach it to the server | |
| 18:54:16 | sean-k-mooney | but yes i can factor the functional test our into there own regression module | |
| 18:54:27 | mriedem | so many words....but i started that | |
| 18:54:29 | sean-k-mooney | and add teh rebuild/shelve tests | |
| 18:55:31 | mriedem | nafiux: some people use cinder exclusively for root volumes of their servers, some people use shared storage for their compute nodes, some mix both - it takes all kinds | |
| 18:57:56 | sean-k-mooney | nafiux: cinder backed volume behvior depend on teh dirver. some cinder backends support thin provisioning allowing over commit some dont | |
| 18:58:49 | sean-k-mooney | and some time the same dirver change behavior based on your config setting which makes it all the more fun to debug | |
| 18:58:59 | nafiux | Thanks @mridem & @sean-k-mooney. I will read more about those topics, so, in general which are the different options for volumens to the instances? | |
| 19:01:11 | sean-k-mooney | there are two sets of volumes, volumes that have independent lifetime form the vm usually called data volumes | |