| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-17 | |||
| 13:57:08 | gibi | mriedem: done | |
| 13:57:18 | mriedem | yay | |
| 13:57:58 | melwitt | nova meeting in 3 min | |
| 14:09:17 | openstackgerrit | Sylvain Bauza proposed openstack/nova-specs master: Proposes Multiple GPU types https://review.openstack.org/557065 | |
| 14:17:13 | mriedem | tssurya: check the max_attempts config option, if it's 1 then we shouldn't be calculating alternate hosts in the scheduler | |
| 14:17:23 | mriedem | and belmorei_ said in boston that cern doesn't do retries | |
| 14:17:41 | dansmith | mriedem: well, let me just hand you my thunder then | |
| 14:18:28 | mriedem | tssurya: dansmith reminded me that max_attempts=1 means no retries and dansmith reminded me that belmorei_ said in boston that cern doesn't do retries | |
| 14:18:40 | dansmith | haha | |
| 14:20:44 | belmorei_ | mriedem: correct :) we have max_attempts=1 | |
| 14:21:31 | dansmith | so if it's trying to get alternates in that case, it's probably a bug | |
| 14:22:28 | mriedem | yeah need to figure that part out, although i thought edleafe added unit tests for that scenario to assert that we didn't try calculating alternates if max_attempest=1 | |
| 14:22:36 | mriedem | tempest on the brain | |
| 14:23:01 | dansmith | ah, | |
| 14:23:08 | dansmith | max_attempts grabs one alternate it looks like | |
| 14:23:15 | dansmith | instead of max_attempts-1 ? | |
| 14:24:06 | dansmith | https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L161-L161 | |
| 14:24:16 | edleafe | num_alts = CONF.scheduler.max_attempts if return_alternates else 0 | |
| 14:24:36 | edleafe | so yeah, it will return 1 alternate with max_attempts=1 | |
| 14:24:52 | dansmith | which is wrong | |
| 14:25:20 | mriedem | arvindn05: how are you coming on that traits-in-images + rebuild patch? | |
| 14:25:38 | dansmith | we also would do too much extra work if num_alts is zero | |
| 14:26:06 | dansmith | belmorei_: tssurya: if you can confirm that with max_attempts the scheduler is asking for one extra alternate that would be good | |
| 14:26:29 | dansmith | would mean about an exact doubling of placement and filter load I guess | |
| 14:26:42 | mriedem | shouldn't be any more placement load, | |
| 14:26:44 | mriedem | just filtering | |
| 14:26:58 | mriedem | the alternates are pulled from the compute node list we get after placement allocation candidates are retrieved | |
| 14:26:59 | dansmith | it asks placement again doesn't it? | |
| 14:27:07 | mriedem | not for new allocation candidates | |
| 14:27:09 | edleafe | alternates only get re-filtered when building > 1 instance per request | |
| 14:27:10 | mriedem | or it shouldn't... | |
| 14:27:19 | belmorei_ | dansmith: will do | |
| 14:27:23 | dansmith | mriedem: okay | |
| 14:29:09 | mriedem | there are some unit tests setting max_attempts to 1 but i'm having a hard time seeing what they assert | |
| 14:29:28 | mriedem | probably just need a simple tests where max_attempts=1, single instance in the request, and make sure we don't mess with alternate calculation | |
| 14:29:34 | mriedem | *test | |
| 14:29:42 | openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Rename spec to match blueprint https://review.openstack.org/568870 | |
| 14:29:43 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Fix interpretation of max_attempts for scheduling alternates https://review.openstack.org/569127 | |
| 14:29:48 | mriedem | dansmith: want to keep this backport train a comin https://review.openstack.org/#/c/568369/ | |
| 14:29:51 | dansmith | belmorei_: tssurya ^ is what I expect we'll do | |
| 14:30:00 | dansmith | *we'll need | |
| 14:30:05 | belmorei_ | this is an interesting issue. We get in average 20s for scheduling... We are debugging and let you know when we have more info | |
| 14:30:27 | dansmith | 20s to schedule one? that's.. uh, terrible | |
| 14:33:49 | melwitt | yeah ... | |
| 14:37:01 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: get_inventory => update_provider_tree https://review.openstack.org/560444 | |
| 14:37:02 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Don't report DISK_GB if sharing https://review.openstack.org/560459 | |
| 14:37:03 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Normalize inventory from update_provider_tree https://review.openstack.org/569132 | |
| 14:37:18 | efried | bhagyashris_: ^ | |
| 14:42:49 | dansmith | belmorei_: tssurya: something else to maybe look at.. by default we ask for 1000 placement results on each schedule, which is probably insane | |
| 14:43:04 | dansmith | you might try tweaking down max_placement_results to see if that makes a difference | |
| 14:43:12 | dansmith | depending on where you see the hotspot is | |
| 14:43:48 | belmorei_ | in our case that shouldn't happen because we use the request_filter | |
| 14:44:10 | dansmith | what do you mean? | |
| 14:44:20 | dansmith | you mean because you're limiting to a cell with only 200 hosts? | |
| 14:44:32 | belmorei_ | the request_filter to use placement_aggregates | |
| 14:44:42 | dansmith | yeah, okay | |
| 14:45:04 | dansmith | well, still, on an empty cell you'd get back the whole cell, but fair enough | |
| 14:45:10 | mriedem | are the compute node resource providers in placement within aggregates? | |
| 14:45:24 | dansmith | mriedem: I think that's the implication :) | |
| 14:45:24 | mriedem | in other words, are you getting back 200 allocation candidates or 10K? | |
| 14:45:37 | mriedem | sure but have they asserted that's all linked up properly | |
| 14:45:45 | dansmith | of they didn't, they would get back zero | |
| 14:45:51 | mriedem | sorry, i mean 1K | |
| 14:45:56 | dansmith | unless they set required=false | |
| 14:46:20 | belmorei_ | depends in the project... but should get in average less than 200 | |
| 14:46:20 | mriedem | because we'd say give me allocation candidates that are a member of this aggregate, and if the rp isn't in that aggregate we'd get back 0, right | |
| 14:46:37 | dansmith | mriedem: yeah | |
| 14:57:47 | openstackgerrit | Merged openstack/osc-placement master: Fix error message asserts in functional test https://review.openstack.org/569048 | |
| 15:01:22 | melwitt | lyarwood: thanks for jumping on https://bugs.launchpad.net/nova/+bug/1771700 | |
| 15:01:23 | openstack | Launchpad bug 1771700 in OpenStack Compute (nova) "nova-lvm tempest job failing with InvalidDiskInfo" [High,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 15:01:38 | lyarwood | melwitt: well, sorry for breaking nova-lvm ^_^ | |
| 15:02:09 | melwitt | mriedem: lyarwood had an idea of running the nova-lvm job non-voting on nova/virt/libvirt/* changes. what do you think? | |
| 15:02:26 | melwitt | no worries | |
| 15:03:42 | mriedem | why non-voting? | |
| 15:03:46 | mriedem | i think it is stable otherwise | |
| 15:04:02 | mriedem | we just don't gate on it because it's not a highly deployed image backend | |
| 15:04:16 | melwitt | I dunno. voting then? | |
| 15:04:21 | mriedem | we also had a raw imagebackend job at one point | |
| 15:04:34 | mriedem | or maybe that was just a dnm patch i had... | |
| 15:05:24 | mriedem | let's see how it looks on lee's fix first | |
| 15:05:40 | mriedem | 1 hour and 20 minutes... | |
| 15:05:43 | mriedem | but it passes | |
| 15:05:50 | mriedem | maybe we should just run the nova-lvm job on compute api tests | |
| 15:05:53 | mriedem | non-voting to start | |
| 15:05:57 | mriedem | and only on libvirt changes | |
| 15:06:23 | melwitt | I think that would be helpful | |
| 15:07:38 | openstackgerrit | Merged openstack/osc-placement master: Resource class set (v1.7) https://review.openstack.org/514644 | |
| 15:09:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: test raw image live migration with vfat config drive https://review.openstack.org/567860 | |
| 15:09:40 | mriedem | lyarwood: let's make sure we don't regress the raw + vfat config drive + live migration case :) ^ | |
| 15:11:14 | lyarwood | mriedem: thanks, I'd like to get off of this regression merry-go-round eventually :) | |
| 15:12:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deduplicate config/policy reference docs from main index https://review.openstack.org/564846 | |
| 15:12:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement granular policy rules for placement https://review.openstack.org/524425 | |
| 15:12:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for /resource_classes* https://review.openstack.org/565578 | |
| 15:12:36 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for resource providers inventories https://review.openstack.org/568666 | |
| 15:12:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add granular policy rules for usages https://review.openstack.org/568706 | |
| 15:12:38 | mriedem | lyarwood: job security! | |
| 15:13:39 | mriedem | per my comments in https://review.openstack.org/#/c/567860/ i'd also like to diversify the live migration tests we run | |
| 15:13:43 | mriedem | to mix up the image format | |
| 15:13:53 | mriedem | and to use config drive | |
| 15:14:39 | mriedem | we already run the nova-live-migration job and the neutron-tempest-multinode-full job runs live migration, | |