| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-31 | |||
| 16:48:10 | mriedem | like mel said, if your test doesn't care about the param, pass a Mock, else pass an actual nova.objects.RequestSpec object, not a dict | |
| 16:48:17 | mriedem | request specs as dicts are super legacy | |
| 17:09:27 | Sundar | mriedem, melwitt: Sorry, was in a call. I am referring to existing Nova unit tests e.g. https://github.com/openstack/nova/blob/master/nova/tests/unit/compute/test_compute.py#L9305. My patches do not anything with flavors etc. but are failing due to this issue: http://logs.openstack.org/44/631244/25/check/openstack-tox-py27/f76bf93/job-output.txt.gz | |
| 17:09:28 | Sundar | #_2019-07-31_10_21_40_684847 | |
| 17:09:38 | Sundar | mriedem, melwitt: Sorry, was in a call. I am referring to existing Nova unit tests e.g. https://github.com/openstack/nova/blob/master/nova/tests/unit/compute/test_compute.py#L9305. My patches do not anything with flavors etc. but are failing due to this issue: http://logs.openstack.org/44/631244/25/check/openstack-tox-py27/f76bf93/job-output.txt.gz | |
| 17:09:39 | Sundar | #_2019-07-31_10_21_40_684847 | |
| 17:10:29 | Sundar | [Yikes, message got duplicated] | |
| 17:10:53 | Sundar | IOW, these are not new test cases I am introducing. | |
| 17:11:55 | melwitt | yeah, so that makes sense considering the code you added. you'll just need to adjust the unit tests to accommodate the new code, like making the test pass Mock objects instead of empty dicts like it is doing right now | |
| 17:14:09 | melwitt | this is a common thing, when adding new code, often existing unit tests will need to be adjusted along with the change. in this particular case, you uncovered a legacy thing (passing {} for objects) that should not be that way | |
| 17:19:33 | Sundar | melwitt: Yes, I know. But there are a lot of unit tests like that. :( Was hoping to get some alternatives. | |
| 17:22:54 | Sundar | melwitt, mriedem: I could adjust my code to check if request_spec is None. Hope that is also acceptable. | |
| 17:27:16 | melwitt | if there's a chance for request_spec to be None in reality, then that would work. if not, it's not really ok to do something in the impl just to get around unit tests that should be fixed. I'm not as well-versed about request_spec as mriedem, he probably would know most whether request_spec could possibly be None in this case | |
| 17:31:08 | Sundar | melwitt: Kinda expected that response, but tried my luck anyway. ;) Thank you. | |
| 17:40:47 | mriedem | Sundar: for tests that hit this code https://review.opendev.org/#/c/631244/25/nova/compute/manager.py@1865 you should be passing a real RequestSpec object. for *existing* tests hitting that code, if they are passing {} or None or something, then they need to be updated to pass either a RequestSpec object or a Mock | |
| 17:41:00 | mriedem | or, to simply mock out _create_and_bind_arqs | |
| 17:41:07 | mriedem | since those existing tests don't care about the details of that method | |
| 17:41:15 | mriedem | and your *new* tests would call _create_and_bind_arqs directly | |
| 17:41:17 | mriedem | with a RequestSpec object | |
| 17:41:32 | mriedem | - Failed: 159 | |
| 17:41:34 | mriedem | yikes | |
| 17:41:51 | mriedem | i'm guessing that's a lot of cumulative failures since it's later in the series | |
| 17:42:06 | mriedem | i.e. https://review.opendev.org/#/c/631243/27 | |
| 17:45:07 | Sundar | mriedem: Thanks. I counted 272 errors of that type over all. Mocking _create_and_bind_arqs is certainly easier. | |
| 17:45:27 | melwitt | Sundar: a bit of advice, what I would do in this situation is write the impl and get the tempest tests passing (I assume something has been or needs to be set up that runs cyborg in it) first, get some review feedback once tempest coverage is passing, then fix the unit test and func test failures after you get some code review. that way you don't have to go through the work to fix hundreds of tests when the impl might change during | |
| 17:45:27 | melwitt | code review | |
| 17:46:43 | mriedem | yeah, or just mock out the thing semi-globally to start, | |
| 17:47:08 | mriedem | e.g. in the patch that is failing tests on _create_and_bind_arqs, just stub that out in the test class setUp | |
| 17:47:41 | mriedem | melwitt: the tricky thing with tempest and this series is there is no tempest coverage with cyborg yet, | |
| 17:47:47 | mriedem | when there is, it'll be a fake cyborg driver, | |
| 17:47:53 | mriedem | but cyborg isn't hooked up with all of that yet, not even placement | |
| 17:47:59 | mriedem | so tempest is a loooong ways off | |
| 17:48:03 | melwitt | oh :\ ok | |
| 17:48:27 | Sundar | melwitt: Makes sense. mriedem: We are setting up tempest: https://review.opendev.org/#/c/670999/ | |
| 17:49:12 | Sundar | But I agree it is tough to predict when it'll land. | |
| 17:49:44 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider config file schema and loader https://review.opendev.org/673341 | |
| 17:52:03 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider config file schema and loader https://review.opendev.org/673341 | |
| 18:07:44 | mriedem | melwitt: looks like https://review.opendev.org/#/c/668123/3 is ready to go in queens | |
| 18:07:50 | mriedem | lee was wanting that one | |
| 18:09:20 | melwitt | a-ha, cool. will review | |
| 18:10:21 | openstackgerrit | Merged openstack/nova master: Get rid of args to RBDDriver.__init__() https://review.opendev.org/668564 | |
| 18:15:21 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Handle alternative UEFI firmware binary paths https://review.opendev.org/348394 | |
| 18:15:21 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities https://review.opendev.org/673790 | |
| 18:23:53 | nafiux | Team, what should I do to properly configure the overcommit of resources? I’m getting this error: “Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.” | |
| 18:30:04 | mriedem | efried_rollin: since i'm not sure how this whole series works, i didn't vote on https://review.opendev.org/#/c/671072/2 but not sure if that should be using a set or list for forbidden_aggregates | |
| 18:30:33 | mriedem | nafiux: you said you listed some resource providers, did you check their inventories to see that they have the allocation_ratio values you configured in nova? you also didn't say what release you were using. | |
| 18:30:47 | mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list | |
| 18:31:55 | nafiux | Thanks @mriedem. I’m using stein. Let me take a look on that documentation. | |
| 18:33:06 | nafiux | Here what I have: https://pastebin.com/MdXsJYs3 | |
| 18:33:13 | mriedem | nafiux: and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios if you didn't read that yet | |
| 18:33:34 | spatel | sean-k-mooney: are you there? | |
| 18:33:45 | mriedem | yeah so your compute nodes configured ram_allocation_ratio is being reported in the MEMORY_MB inventory for the provider | |
| 18:33:46 | mriedem | | MEMORY_MB | 5.0 | 49139 | 512 | 1 | 1 | 49139 | | |
| 18:33:49 | mriedem | that looks fine | |
| 18:33:56 | sean-k-mooney | spatel: kind of yes | |
| 18:34:08 | mriedem | nafiux: is this a fresh deploy or do you already have some servers on that node? | |
| 18:35:02 | nafiux | It’s a fresh deployment, I don’t have any instance running at this point. I used kolla-ansible. I checked the /etc/nova/nova.conf inside the nova_scheduler container and I was able to see the allocation values: | |
| 18:35:11 | nafiux | docker exec -it nova_scheduler bash | |
| 18:35:20 | spatel | Good, I have machine with 64G and with this setting in grub hugepagesz=2M hugepages=27500 | |
| 18:35:49 | spatel | I have this flavor setting - properties | hw:cpu_policy='dedicated', hw:mem_page_size='large', hw:numa_nodes='2' | |
| 18:35:51 | nafiux | vi /etc/nova/nova.conf, and I can see inside [DEFAULT] the cpu_allocation_ratio = 16.0 and ram_allocation_ratio = 5.0 | |
| 18:36:13 | nafiux | I’m able to launch a server with 16GB of Ram and 8 VPCU, but I’m not able to launch a second one. | |
| 18:36:21 | nafiux | Let me launch one, and provide you the report again. | |
| 18:37:14 | spatel | flavor has 8vcpu / 12G memory | |
| 18:37:38 | spatel | trying to create 2 VM but getting error, but i am able to create 1 vm ( not able to create 2) | |
| 18:38:39 | sean-k-mooney | you should have enough hugepages although the kerenl may not alwasys blance it perfectly | |
| 18:38:58 | sean-k-mooney | you have enough clearance however that that should not be the issue | |
| 18:39:09 | sean-k-mooney | how many cpu to you have | |
| 18:39:29 | spatel | 32 core | |
| 18:39:53 | nafiux | @mriedem: I was able to launch the first one, but whe I try to launch the second one I got the “Got no allocation candidates from the Placement API.” error | |
| 18:40:39 | sean-k-mooney | you have ~20G free per numa node and you need 12 so it should not be hugepage related | |
| 18:41:15 | sean-k-mooney | you have 32 cores is htat 2 16 core cpus or 2 8 core cpus with hyperthread or 1 32 core cpus | |
| 18:41:25 | mriedem | nafiux: how much disk in the flavor? or are these volume-backed servers? | |
| 18:41:50 | mriedem | nafiux: the placement-api also logs the allocation candidate filtering it does so you should trace the requestid through the placement logs - might need to enable debug | |
| 18:41:58 | spatel | sean-k-mooney: that is what i am trying to find.. what is wrong here :( | |
| 18:42:00 | sean-k-mooney | spatel: it would be useful to know what the vcpu_pin_set is set to in your nova.conf | |
| 18:42:02 | mriedem | that should tell you why it kicked out the provider | |
| 18:42:43 | nafiux | Ouch, I think that is it! Is the hard disk. Which actually doesn’t overcommit. | |
| 18:42:54 | nafiux | I have assigned a lot in the flavor, let me reduce it and test it again. | |
| 18:43:13 | mriedem | yeah your provider only has ~300 GB of disk | |
| 18:43:23 | spatel | grep vcpu_pin_set /etc/nova/nova.conf <--- empty | |
| 18:43:42 | spatel | I have queen cluster and same config works there... | |
| 18:43:48 | mriedem | nafiux: if you were doing volume-backed servers then you're probably fine, lots of vcpu and ram on that node | |
| 18:43:48 | spatel | i am running new cluster in stein | |
| 18:43:50 | mriedem | but very little disk | |
| 18:43:55 | spatel | which causing issue | |
| 18:44:45 | nafiux | @mriedem yes, that is true, only 300 GB free, and the flavor has 160 GB, so only 1 instance with that capacity will be allowed. | |
| 18:45:04 | mriedem | \o/ | |
| 18:46:36 | nafiux | Does "volume-backed" means the actuall physical hard disk of the hypervisor/nova node? | |
| 18:46:48 | mriedem | no | |
| 18:46:55 | mriedem | means the root disk is a cinder volume | |
| 18:47:11 | mriedem | https://docs.openstack.org/nova/latest/user/launch-instance-from-volume.html | |
| 18:48:09 | nafiux | Oh, I don’t have installed cinder. That will help a lot, if I understand correctly? | |
| 18:48:31 | mriedem | i don't know how to answer that question :) | |
| 18:48:46 | mriedem | if you have a compute node with 300 GB of disk and flavors with 160 GB of disk, you're not going to get far | |
| 18:48:54 | mriedem | so you'd have to carve down your flavors to have smaller disk, | |
| 18:49:00 | mriedem | or add more storage to your compute nodes, | |
| 18:49:07 | mriedem | or use cinder to back the root disk | |
| 18:50:02 | nafiux | Lol, yeah. How servers backed with cinder volumes behave? Do they use as needed and not the actual size specified at creation time? | |
| 18:53:13 | mriedem | for volume-backed servers the root gb on the flavor doesn't really apply | |