| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-24 | |||
| 14:21:36 | stephenfin | dansmith: Barely. I'm seeing image retrieval failures in n-cpu | |
| 14:21:39 | stephenfin | yeah, those ^ | |
| 14:22:13 | dansmith | hmm, maybe the first test I picked was a rando failure then | |
| 14:22:15 | stephenfin | but there are a couple of patches in that series failing and I don't think they're related to the code | |
| 14:23:15 | sean-k-mooney | if those tests are uploading new images maybe they are not ready when the boot is started because the import/conversion takes longer or something | |
| 14:23:43 | dansmith | ah yeah, I see now | |
| 14:24:25 | dansmith | sean-k-mooney: yeah that could be | |
| 14:24:36 | dansmith | I think we should still be able to GET the image though | |
| 14:25:06 | sean-k-mooney | looks like that is not the case for rescure at least https://github.com/openstack/tempest/blob/257f3b009f7978723a8748f9f5b413aa8eb38e3a/tempest/api/compute/servers/test_server_rescue.py#L55-L67 | |
| 14:26:12 | dansmith | sean-k-mooney: what is not the case for rescue? | |
| 14:26:15 | sean-k-mooney | ya it just does rescue without specifying an image so it will use the image the vm was booted with or the image specifid in the config. i wonder if it failed before that | |
| 14:26:31 | sean-k-mooney | dansmith: the rescue test is not uploading any images | |
| 14:26:32 | dansmith | are you looking at a different fail? | |
| 14:27:04 | sean-k-mooney | tempest.api.compute.servers.test_server_rescue.ServerRescueTestJSON.test_rescue_unrescue_instance | |
| 14:27:13 | sean-k-mooney | its the second failure in the test report | |
| 14:27:39 | dansmith | ack, the first thing you linked is to an ImagesTest not rescue right? | |
| 14:27:48 | dansmith | it's definitely doing a snapshot | |
| 14:29:15 | sean-k-mooney | actully looking at the server uuid its not in the ncpu log so the novalid host looks like it really could not fit | |
| 14:29:55 | dansmith | sean-k-mooney: right, that's what I was saying, I just picked poorly on the first test to look at :) | |
| 14:29:56 | dansmith | sean-k-mooney: hmm, I see a DELETE of the image just before the failed GET in the glance logs, for that snapshot one, which is odd | |
| 14:30:12 | sean-k-mooney | yep Got no allocation candidates from the Placement API. | |
| 14:31:05 | sean-k-mooney | oh downstream call | |
| 14:31:20 | dansmith | ah | |
| 14:31:22 | dansmith | so, | |
| 14:31:29 | dansmith | I think that stack trace from sean-k-mooney is a red herring | |
| 14:31:44 | dansmith | I think that's an images test that tries to delete the image whilst snapshotting or something | |
| 14:31:51 | dansmith | it's not even one of the tests that failed in the testr report :) | |
| 14:32:10 | dansmith | all three of those tests are novalidhost | |
| 14:32:32 | dansmith | so maybe we're actually reporting something different to placement and running out of disk or something? | |
| 14:32:56 | sean-k-mooney | ya maybe | |
| 14:33:12 | sean-k-mooney | we have 80G of disk in the ci vms but it may not all be avaible int /opt | |
| 14:33:21 | sean-k-mooney | so i dont know we might have ran out of space | |
| 14:34:10 | dansmith | well, | |
| 14:34:17 | dansmith | it might be a reporting thing or something and not actually out of space, | |
| 14:34:28 | dansmith | because we're not seeing problems, just placement is refusing to find space | |
| 14:36:59 | dansmith | Jul 24 12:44:22.575632 ubuntu-bionic-ovh-bhs1-0018770257 devstack@placement-api.service[50512]: DEBUG placement.wsgi_wrapper [req-eeb6d563-2483-4e4f-91e8-2dc3a694ade4 req-c57d5bd6-fc4e-469d-9784-cdfe1652d653 service placement] Placement API returning an error response: Unable to allocate inventory: Unable to create allocation for 'DISK_GB' on resource provider 'e786426a-5ae2-4732-8cf6-16325fd2bf2a'. The requested amount would exceed | |
| 14:37:00 | dansmith | the capacity. {{(pid=50513) call_func /opt/stack/placement/placement/wsgi_wrapper.py:31}} | |
| 14:37:16 | dansmith | Over capacity for DISK_GB on resource provider e786426a-5ae2-4732-8cf6-16325fd2bf2a. Needed: 1, Used: 10, Capacity: 10.0 | |
| 14:37:22 | dansmith | 10G doesn't sound right | |
| 14:40:45 | mriedem | random drive by comment but https://review.opendev.org/#/c/586363/ | |
| 14:41:06 | mriedem | anyway related to ceph ci jobs? | |
| 14:41:30 | dansmith | I'm trying to figure out, but we are running a ceph df right before we report inventory | |
| 14:41:40 | openstackgerrit | Alex Deiter proposed openstack/nova master: Detach is broken for multi-attached fs-based volumes https://review.opendev.org/741712 | |
| 14:54:48 | sean-k-mooney | dansmith: by the way if the traceback is unrelated then we likely have another silent bug as we are not catching the excpetion in the missing image case | |
| 14:54:56 | dansmith | sean-k-mooney: yep | |
| 14:55:06 | dansmith | so we're calling ceph df to get the total size of the pool and reporting that | |
| 14:55:07 | sean-k-mooney | dansmith: i think you are right that its unrelated | |
| 14:55:16 | dansmith | as best I can tell, the ceph is backed by a 24G partition | |
| 14:55:27 | dansmith | so I dunno where the 10G is coming from | |
| 14:56:04 | sean-k-mooney | this is using the ceph image backend in nova so the local_GB should be the ceph pool size right | |
| 14:56:25 | dansmith | well, it should be yes | |
| 14:56:43 | dansmith | ceph has 24G, so I'm trying to find where our images pool would be limited to 10G but not seeing it | |
| 14:56:47 | dansmith | one thing that might explain this, | |
| 14:57:09 | dansmith | is that our normal ceph job was using qcow on rbd, which is not what you're supposed to do, | |
| 14:57:24 | sean-k-mooney | oh ya because we have to flatten it | |
| 14:57:31 | sean-k-mooney | it should be raw | |
| 14:57:36 | sean-k-mooney | to get the cow optimization | |
| 14:57:40 | dansmith | and so we convert the image to raw, which is 44M per image instead of 12 or something.. although we shouldn't really be using that much space, so... hmm | |
| 14:57:53 | dansmith | and this is just placement saying we're out of space, not ceph | |
| 14:58:17 | dansmith | I wonder if glance is incorrectly determining the size of the new image after it flattens or something | |
| 14:58:19 | sean-k-mooney | well with after teh first image import is all cow clones in ceph right | |
| 14:58:23 | dansmith | and telling us we need a lot more than we do or something | |
| 14:58:30 | dansmith | right | |
| 15:01:20 | dansmith | check this out: Jul 24 12:44:22.270293 ubuntu-bionic-ovh-bhs1-0018770257 nova-scheduler[55176]: WARNING nova.scheduler.host_manager [None req-eeb6d563-2483-4e4f-91e8-2dc3a694ade4 tempest-MultipleCreateTestJSON-968818181 tempest-MultipleCreateTestJSON-968818181] Host ubuntu-bionic-ovh-bhs1-0018770257 has more disk space than database expected (8 GB > 1 GB) | |
| 15:01:37 | sean-k-mooney | reserved_host_disk_mb IS 0 TOO | |
| 15:02:46 | sean-k-mooney | that is strange do we have the hoststate update enabled | |
| 15:03:09 | sean-k-mooney | im pretty sure we do | |
| 15:03:44 | sean-k-mooney | ya we do | |
| 15:04:23 | sean-k-mooney | disk_allocation_ratio=1.0,disk_available_least=8,free_disk_gb=10,f | |
| 15:04:33 | dansmith | we're only asking placement for DISK_GB=1 allocation so I don't think we're getting a bad number from glance or anything | |
| 15:06:15 | sean-k-mooney | what do our flavor look like | |
| 15:06:27 | sean-k-mooney | actully no never mind | |
| 15:06:30 | sean-k-mooney | this is not bfv | |
| 15:06:51 | sean-k-mooney | the flavor should be either 1 or 2GB per instance i think | |
| 15:07:20 | dansmith | and it seems like 1 since we're asking for that size allocation | |
| 15:09:44 | sean-k-mooney | ya its based on teh image size https://github.com/openstack/devstack/blob/2ecd1823850ae0e00ad0ecebbbceb312be60ccf4/lib/tempest#L204-L206 | |
| 15:09:53 | sean-k-mooney | so for cirros image it will be 1g | |
| 15:10:04 | dansmith | sudo ceph -c /etc/ceph/ceph.conf osd pool create vms 8 8 | |
| 15:10:10 | dansmith | that's 8G for the vms pool | |
| 15:10:18 | dansmith | I dunno where we're getting 10G | |
| 15:10:26 | sean-k-mooney | i dont think that is the size | |
| 15:10:43 | sean-k-mooney | i think that is the buckest to share it in | |
| 15:10:45 | sean-k-mooney | let me check | |
| 15:11:15 | dansmith | hmm, okay it seems like size | |
| 15:11:35 | sean-k-mooney | i think its the placment groups but its been a while | |
| 15:11:41 | dansmith | okay yeah, maybe you're right | |
| 15:12:39 | sean-k-mooney | ceph osd pool create <pool-name> <pg-num> <pgp-num> [replicated] \ | |
| 15:12:41 | sean-k-mooney | [crush-ruleset-name] [expected-num-objects] | |
| 15:12:52 | sean-k-mooney | so ya its not the size | |
| 15:13:04 | dansmith | yeah | |
| 15:13:22 | dansmith | I still dunno where we're getting 10G, | |
| 15:13:34 | sean-k-mooney | same | |
| 15:14:27 | dansmith | because CEPH_LOOPBACK_DISK_SIZE=24G | |
| 15:14:57 | sean-k-mooney | so it should be 8 https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/settings#L17 | |
| 15:15:01 | sean-k-mooney | by default | |
| 15:15:12 | dansmith | it's overridden in our job somewhere | |
| 15:15:16 | dansmith | you can see in the devstacklog | |
| 15:15:24 | sean-k-mooney | CEPH_LOOPBACK_DISK_SIZE is | |
| 15:15:31 | sean-k-mooney | is VOLUME_BACKING_FILE_SIZE | |
| 15:15:34 | dansmith | ues | |