Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-26
14:01:55 dansmith ah, okay
14:12:31 mriedem mdbooth: if you're busy do you care if i add the unit test for https://review.openstack.org/#/c/645546/ ?
14:14:19 trident Hmm. Is trait:CUSTOM_FOO=forbidden set as flavor property expected to work in stable/rocky? I get instances spun up with flavors with that set ending up on resource providers (compute nodes) with that trait set. The other way around seem to work - a flavor with trait:CUSTOM_FOO=required makes instances with that flavor end up on the resource providers with that trait set.
14:15:51 mriedem trident: says it should https://docs.openstack.org/nova/rocky/user/flavors.html
14:15:58 mriedem "Added in the 18.0.0 Rocky release."
14:16:49 mdbooth mriedem: I'd be delighted :) I'm currently working on https://bugs.launchpad.net/nova/+bug/1821373
14:16:50 openstack Launchpad bug 1821373 in OpenStack Compute (nova) "Most instance actions can be called concurrently" [Undecided,New]
14:17:52 trident mriedem: Yeah, that's what I thought as well, so I am a bit surprised. Didn't really find any bugs that seem relevant either.
14:17:58 mriedem mdbooth: ok will do. another thing was https://review.openstack.org/#/c/551349/ just needs some tweaks i think, but it looks like maybe lyarwood is working that now
14:18:08 mriedem unless lyarwood is stuck in tripleo limbo for the next month
14:18:27 openstackgerrit Sylvain Bauza proposed openstack/nova master: Add doc on VGPU allocs and inventories for nrp https://review.openstack.org/647519
14:18:46 mdbooth mriedem: He's away this week, but he has taken that patch.
14:19:29 mriedem trident: if it's helpful, here is a functional test that does traits-based scheduling https://github.com/openstack/nova/blob/358776a303b0c5dba3c279f242c82b7a12ff5ce8/nova/tests/functional/test_servers.py#L4451
14:20:09 trident mriedem: Butt still the API call to placement /allocation_candidates doesn't seem to pass the forbidden trait (which if I understand it correctly should be "required=!CUSTOM_FOO" while it does pass it for the flavor with trait required: "required=CUSTOM_FOO"
14:21:08 trident mriedem: Thanks, I'll keep looking then, just wanted to check that I am not missing anything obvious.
14:21:33 mriedem hmm, looking at the test in there that uses a forbidden trait on a flavor https://github.com/openstack/nova/blob/358776a303b0c5dba3c279f242c82b7a12ff5ce8/nova/tests/functional/test_servers.py#L4824 it doesn't seem like the forbidden trait is actually testing anything useful
14:22:26 mriedem meaning, a better test would be to have something like 2 computes, one with trait CUSTOM_FOO and one without, and disable the one without so we can't schedule there, then schedule with forbidden CUSTOM_FOO and it should have a NoValidHost result,
14:22:33 mriedem but you're saying it sounds like it might just pass
14:23:42 trident mriedem: I'll see what I find continuing the troubleshooting and get back...
14:24:13 mriedem ack, this is the code that builds the GET /allocation_candidates query string fwiw https://github.com/openstack/nova/blob/358776a303b0c5dba3c279f242c82b7a12ff5ce8/nova/scheduler/utils.py#L257
14:28:03 trident mriedem: Thanks. that's exactly where I am currently looking :)
15:01:36 brandor5 hello everyone: I'm trying to use hypervisor_hostname as a scheduler hint and it's worked in the past but I'm running into issues now... I get a 400 bad request back... anyone have any ideas why that could be?
15:02:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix incomplete instance data returned after build failure https://review.openstack.org/645546
15:02:14 mriedem mdbooth: done ^
15:03:47 mdbooth mriedem: Thanks! You ought to add a co-authored line, esp considering the tests are longer than the fix :)
15:04:42 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix incomplete instance data returned after build failure https://review.openstack.org/645546
15:06:13 mriedem melwitt: want to hit ^ when you get a chance?
15:06:15 mriedem goes back to pike
15:06:48 mriedem brandor5: would be helpful if you shared the actual 400 error message
15:07:15 brandor5 mriedem: I'm trying to dig it out, 1 second :)
15:07:59 mriedem scheduler hints are open-ended for the most part, and hypervisor_hostname is not a known hint https://github.com/openstack/nova/blob/358776a303b0c5dba3c279f242c82b7a12ff5ce8/nova/api/openstack/compute/schemas/servers.py#L93
15:08:07 mriedem so i'm assuming you're using some out of tree scheduler filter for this hint?
15:08:27 brandor5 http://sprunge.us/9pGzX9
15:08:37 mriedem like forcing to build on a specific baremetal node?
15:08:44 brandor5 exactly
15:08:51 mriedem that doesn't give the error message
15:08:59 brandor5 query: '["=","$hypervisor_hostname","b23bbc4a-9a43-4ed8-b028-43d2aca3a6d7"]'
15:09:05 brandor5 that's the only 400 I see anywhere
15:09:17 brandor5 when i'm running the command
15:09:26 mriedem you don't get an error message on the client side?
15:09:29 brandor5 ansible returns: msg: 'BadRequestException: 400'
15:09:47 brandor5 if I remove the filter from the call it works fine
15:09:48 mriedem there is nothing else in the logs for req-f5f66dc0-8035-4c07-a631-a438fb79ba42 ?
15:10:07 mriedem so you're hitting something with this https://github.com/openstack/nova/blob/358776a303b0c5dba3c279f242c82b7a12ff5ce8/nova/api/openstack/compute/schemas/servers.py#L125
15:10:24 mriedem i want to say i've heard this reported bfore
15:10:26 mriedem *before
15:11:13 brandor5 nothing else found in the logs with that request
15:11:32 brandor5 the 400 would be coming from nova, correct?
15:11:37 brandor5 I shouldn't be looking down in ironic?
15:11:46 mriedem it's coming from nova
15:11:57 brandor5 that's what I thought
15:12:34 mdbooth Is it worth writing new code whose only consumer would be cellsv1?
15:12:52 mriedem mdbooth: nope
15:12:58 mdbooth Specifically, it looks like the only user of expected_vm_state is cellsv1
15:13:13 mdbooth i.e. instance.save(expected_vm_state=FOO)
15:13:15 mriedem probably because cells v1 was racy
15:13:26 mriedem syncing state between child and parent cell
15:13:55 mdbooth Ok. It'll look a bit odd to add new expected_task_state code without expected_vm_state code, though, so I'll add a comment
15:15:53 mriedem brandor5: i don't know what would be returning that 400 except the json schema validation code, but that query hint is just marked as a string or object, and you're passing a json string
15:16:22 mriedem the actual query hint isn't used until we hit the JsonFilter scheduler filter, which is long after you'd get that 400
15:16:43 mriedem so my guess is the request schema validation is somehow puking
15:16:54 brandor5 ok, i'll keep digging and will report back if I figure it out
15:17:08 mriedem i very much doubt we have any functional testing for JsonFilter
15:17:35 mriedem brandor5: i think the last person that had this problem just used availability_zone=nova::b23bbc4a-9a43-4ed8-b028-43d2aca3a6d7
15:17:45 mriedem the only problem with that is it bypasses the filters and forces to that node
15:17:55 brandor5 that would be okay in this case
15:18:04 mriedem https://docs.openstack.org/nova/latest/admin/availability-zones.html
15:18:05 brandor5 so I'll try that if the thing im about to test doesn't work
15:18:07 mriedem then force away
15:23:38 brandor5 mriedem: it turns out it was a bug in ansible, updating to latest and I don't see that issue now
15:26:18 mriedem huh
15:26:29 mriedem well you made me find our docs are busted at least :) https://bugs.launchpad.net/nova/+bug/1821764
15:26:30 brandor5 yeah, weird
15:26:30 openstack Launchpad bug 1821764 in OpenStack Compute (nova) "docs: JsonFilter query hint examples do not use valid json strings" [Medium,Triaged]
15:26:46 brandor5 hehe glad to be of assistance :D
15:28:55 sean-k-mooney mriedem: [">=", "$free_ram_mb", 1024] is a vlaid json array
15:29:21 mriedem sean-k-mooney: did you read the bug?
15:30:08 sean-k-mooney im just looking at it but '[">=", "$free_ram_mb", 1024]' is not the same thing as json.loads("[>=,$free_ram_mb,1024]")
15:30:57 mriedem serenity now
15:32:06 openstackgerrit Merged openstack/nova stable/pike: Fix WeighedHost logging regression https://review.openstack.org/641398
15:33:48 sean-k-mooney oh you are saying its correct in the schduler doc but not in the create server example
15:34:40 mriedem i don't see how what i wrote in the bug report is unclear
15:36:48 sean-k-mooney i was confused by the fact that it was correct in the osc command but not in the api example in the json filter doc
15:37:46 sean-k-mooney i intially looked at it and didnt see teh api example was wrong as my brain just filled in teh missing quots ared the args
15:38:53 sean-k-mooney its why im terible are prof readign my own stuff if i read two things that are almost identical and expect them to be the same i have troble seeign the difference
15:38:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix JsonFilter query hint examples in docs https://review.openstack.org/647778
15:42:44 mriedem ooo dansmith i could use the JsonFilter to schedule directly to a cell, that's fun
15:43:22 sean-k-mooney mriedem: you can you the json filter to do a lot of horribel things
15:43:57 dansmith mriedem: you can probably use it to do all kinds of things
15:45:36 sean-k-mooney as far as i rememebr it give you acess to any value contined in the hoststate object
15:46:51 mriedem correct
15:48:31 mriedem ima add a warning in the docs
15:49:09 sean-k-mooney the main usecase i was aware of was shduing based in the cpu info or hypervior type.
15:49:30 mriedem i think the ImagePropertiesFilter already handles hv type
15:49:39 mriedem and cpu info should be solved with traits
15:49:44 sean-k-mooney ya it does
15:50:48 sean-k-mooney the json filter is really a filter that we shoudl be able to remove in the near future but some peopel still use it. i think oath have an internal version too.
15:57:39 mdbooth Hmm, my grand plan to fix instance.save() is scuppered because I forgot we don't have access to the old values of fields which have changed but not saved.
16:01:40 sean-k-mooney mdbooth: can you not just always save all changed field when we call save and simply mark any filed we write to as chaged ignoring if the value is the same or not
16:02:19 mdbooth sean-k-mooney: No, because we've already lost the old value before we called save()

Earlier   Later