| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-20 | |||
| 13:54:01 | dansmith | seems clear, | |
| 13:54:08 | cdent | mriedem: I think i was following along okay, was just trying to pause bauzas | |
| 13:54:09 | tssurya | I have understood this to be something needed as a regression fix for the code before this patch | |
| 13:54:09 | gibi | nova meeting stars in 6 minutes on #openstack-meeting | |
| 13:54:13 | tssurya | so its clear to me too | |
| 13:54:18 | bauzas | wait wait | |
| 13:54:22 | mriedem | so bauzas is the odd man out | |
| 13:54:25 | dansmith | we are not filtering the list of placement results against compute nodes that exist | |
| 13:54:38 | mnaser | yes, we're getting *all* including deleted ones | |
| 13:54:40 | bauzas | can someone explains me what's the problem ? | |
| 13:54:45 | tssurya | yep yep | |
| 13:54:49 | dansmith | bauzas: wait for my bug fix | |
| 13:54:49 | mriedem | dansmith: yes, but we were only given those uuids for deleted records because mnaser doesn't have https://review.openstack.org/#/c/563698/ | |
| 13:54:51 | dansmith | it'll be clear then | |
| 13:54:52 | bauzas | see my BBIAB 30 mins before :) | |
| 13:54:56 | bauzas | dansmith: ok cool | |
| 13:54:57 | dansmith | mriedem: right | |
| 13:55:01 | mriedem | which would have deleted the RP in placement when the compute service was deleted | |
| 13:55:04 | mriedem | it's just a simple side effect | |
| 13:55:33 | mnaser | bauzas: if a compute service is deleted (but still somehow in placement and has host_mappings, something which a bug has fixed not long ago), then nova ComputeNodeList.get_all_by_uuids returns that, it shouldn't | |
| 13:55:37 | dansmith | mriedem: yeah, but we could need to delete a compute node when placement is down or something, so definitely should fix this issue | |
| 13:55:39 | bauzas | jroll: meh, I do have a guitar as well at home... | |
| 13:56:02 | jroll | ++ | |
| 13:56:03 | mnaser | bauzas: and the reason was because get_all_by_uuids() doesn't use model_query so it doesn't filter against soft deleted stuff | |
| 13:56:17 | mnaser | that's what i got out of it and it makes sense enough for me :p | |
| 13:56:24 | mriedem | dansmith: sure i agree | |
| 13:56:42 | bauzas | mnaser: we don't have services in placement, just compute nodes, but okay I understand you :) | |
| 13:56:53 | mnaser | i'm trying here | |
| 13:56:53 | mnaser | haha | |
| 13:57:03 | bauzas | 3 mins dude | |
| 13:57:06 | bauzas | too early | |
| 13:57:08 | mnaser | one last thing: is someone pushing up a patch for this or is it on me? | |
| 13:57:13 | mriedem | dan is | |
| 13:57:13 | mnaser | (i dont mind) | |
| 13:57:16 | mnaser | ok cool | |
| 13:57:18 | bauzas | i think dansmith said | |
| 13:57:20 | mnaser | thanks dansmith | |
| 13:57:21 | bauzas | yeah that | |
| 13:57:29 | mriedem | *head explodes* | |
| 13:57:48 | bauzas | just throw me tomatoes now if I regressed :) | |
| 13:57:59 | dansmith | bauzas: you did regress | |
| 13:58:03 | dansmith | but I will fix | |
| 13:58:11 | efried | stephenfin: Can you please see if I used the correct arguments here? https://review.openstack.org/#/c/603993/ | |
| 13:58:28 | bauzas | dansmith: you're the hero and I'm the villain then ;) | |
| 13:58:38 | dansmith | per usual | |
| 13:58:58 | bauzas | I can snip fingers, don't tempt me | |
| 13:59:01 | bauzas | snap* | |
| 14:37:41 | openstackgerrit | Dan Smith proposed openstack/nova master: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604108 | |
| 14:45:07 | mriedem | mdbooth: did you see the evacuate integration tests are passing? | |
| 14:45:53 | mdbooth | mriedem: I didn't, but I will as soon as poss :) | |
| 14:46:39 | mriedem | not much to see - we evacuate with the down compute so it fails, and then bring libvirtd back up and evacuate and it passes | |
| 14:46:54 | mriedem | only question is if it should be horked into the live migration job which is convenient but sorta confusing | |
| 14:47:23 | mdbooth | mriedem: Ideally we'd have it in a few places: would be good to cover both shared and non-shared. | |
| 14:47:46 | mdbooth | Perfectly we'd cover 'all 4' cases, but I don't think we have all 4 in CI. | |
| 14:48:24 | mriedem | all 4 is what again? volume-backed, local disk not shared, local disk shared | |
| 14:48:26 | mriedem | what's the other? | |
| 14:48:40 | mdbooth | 2 axes are: shared instance, shared disk. | |
| 14:48:41 | mriedem | right now the patch is running on ceph local shared | |
| 14:49:11 | mriedem | the script could be updated to create a volume-backed server and evacuate again | |
| 14:49:42 | mnaser | is there a reason why running debug=True doesn't show the http requests done by placement client? | |
| 14:49:50 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: libvirt: Turn off max_unit https://review.openstack.org/604110 | |
| 14:50:02 | mnaser | rather than reconstructing the http request, it'd be nice to know what nova-scheduler is asking placement | |
| 14:50:27 | mriedem | we call placement over ksa, | |
| 14:50:41 | mriedem | so our debug=True logging is likely not carrying over to ksa | |
| 14:51:19 | mnaser | because before debug=True would show you the filters running through things | |
| 14:51:27 | mnaser | and you'd be able to debug scheduler issues | |
| 14:51:43 | mriedem | you just want to see like the GET /allocation_candidates request right? | |
| 14:51:45 | mnaser | now it just starts filtering a subset and you don't know where that came from so yeah | |
| 14:51:59 | mnaser | stuff like that, yes, so that i know why this live migration is not getting scheduled on one of the hosts | |
| 14:52:07 | mnaser | there's empty capacity but it's not adding it | |
| 14:52:21 | mnaser | probably something along the lines of placement being inconsistent | |
| 14:52:57 | mriedem | are you passing the specific target host for the live migration to the compute API? | |
| 14:53:05 | mriedem | because i know of another bug related to that... | |
| 14:53:15 | mnaser | nope im not passing one | |
| 14:53:30 | mriedem | ok, you just expect a certain host to be picked b/c of weights and capacity etc? | |
| 14:53:54 | mnaser | yeah, it's not even hitting weight/capacity, placement is not returning a host that has capacity | |
| 14:54:08 | mriedem | i think we'd have to add something like this https://github.com/openstack/nova/blob/master/nova/config.py#L39 | |
| 14:54:09 | efried | Does the placement log not show the full command line? | |
| 14:54:11 | mnaser | which tells me there's probably incorrect allocations in placement that make that host look 'overloaded' | |
| 14:54:22 | mnaser | ah that's a good alternative, checking uwsgi logs on placement side | |
| 14:54:30 | mriedem | there should be the same request id on both sides | |
| 14:54:46 | mriedem | that's your workaround for now | |
| 14:54:53 | efried | The placement logs by default won't show the payloads (request or response) but the queryparams are there I thought. | |
| 14:55:01 | mriedem | yes they should be | |
| 14:55:09 | mnaser | i should be able to rerun that http request | |
| 14:55:24 | mriedem | http://logs.openstack.org/75/470775/38/check/tempest-full/a97793a/controller/logs/screen-placement-api.txt.gz#_Sep_06_13_02_32_181234 | |
| 14:55:25 | efried | sorry, "command line" was a bogus phrase | |
| 14:55:29 | mriedem | Sep 06 13:02:32.181234 ubuntu-xenial-ovh-bhs1-0001824138 devstack@placement-api.service[6991]: DEBUG nova.api.openstack.placement.requestlog [req-68f9280e-61b2-446e-8622-a64e27e7c617 req-df696f1f-3435-4b6c-8a3b-c6dc1c58c1ba service placement] Starting request: 158.69.79.172 "GET /placement/allocation_candidates?limit=1000&resources=MEMORY_MB%3A64%2CVCPU%3A1" {{(pid=6993) __call__ /opt/stack/nova/nova/api/openstack/placement/r | |
| 14:55:29 | mriedem | stlog.py:38}} | |
| 14:55:54 | mriedem | must be a bfv instance ^ because there is no DISK_GB requested | |
| 14:56:35 | mnaser | yup /allocation_candidates?limit=1000&resources=DISK_GB%3A10%2CMEMORY_MB%3A2048%2CVCPU%3A4 | |
| 14:56:36 | mnaser | lets see | |
| 14:58:20 | mnaser | openstack allocation candidate list --resource DISK_GB=10 --resource MEMORY_MB=2048 --resource VCPU=4 => Operation or argument is not supported with version 1.0 | |
| 14:58:25 | mnaser | mriedem: do i need some special env variable? | |
| 14:58:49 | mnaser | sigh | |
| 14:58:49 | mnaser | --os-placement-api-version 1.10 | |
| 14:58:52 | mnaser | ill read docs from now on | |
| 14:59:47 | gibi | FYI, I've added the bp use-nested-allocation-candidates to the 3rd runway slot | |
| 14:59:51 | gibi | First part of https://blueprints.launchpad.net/nova/+spec/use-nested-allocation-candidates (gibi) [END: 2018-10-04] | |
| 15:00:01 | cdent | i guess I need to learn osc-placement so I can parse this stuff. I'm still back on raw curl which is nice and comfy and explicit | |