| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-27 | |||
| 15:52:28 | mriedem | well, | |
| 15:52:32 | mriedem | we don't have CI for a lot of things | |
| 15:52:50 | mriedem | and we still support them, <cough>evacuate</cough> | |
| 15:53:28 | sean-k-mooney | yes and i assume they are broken by default unless proven otherwise by it working when i use it and being happy | |
| 15:53:31 | dansmith | evacuate is hard to test for legit reasons, but this shared thing is not | |
| 15:53:43 | dansmith | and it's also often broken | |
| 15:54:10 | mriedem | yup | |
| 15:54:57 | mriedem | btw, yes, forced host live migrate/evacuate will drop the DISK_GB allocation on the shared provider | |
| 15:56:09 | dansmith | mriedem: from your test? | |
| 15:57:18 | mriedem | no just looking at teh code | |
| 15:57:27 | mriedem | https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/scheduler/utils.py#L500 | |
| 15:57:34 | mriedem | we only get the allocations for the instance against the source node | |
| 15:57:39 | dansmith | oh | |
| 15:57:40 | mriedem | and copy those to the dest node for the instance | |
| 15:57:42 | mriedem | double up | |
| 15:58:00 | mriedem | doesn't put anything on the migration record in the force cas | |
| 15:58:03 | mriedem | *case | |
| 15:58:53 | mriedem | hmm, which makes me wonder if we ever cleanup the dest host allocations on a failed live migration | |
| 15:58:55 | mriedem | that is forced | |
| 16:00:14 | mriedem | looks like post_live_migration will give you a warning but remove the doubled allocation | |
| 16:00:46 | mriedem | https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/compute/manager.py#L6638L6669 | |
| 16:00:49 | mriedem | oops | |
| 16:01:21 | mriedem | i'll write a functional test for the rollback forced live migration case | |
| 16:06:08 | mriedem | https://bugs.launchpad.net/nova/+bug/1784022 | |
| 16:06:08 | openstack | Launchpad bug 1784022 in OpenStack Compute (nova) "Failed forced live migration does not rollback doubled up allocations in placement" [High,Triaged] | |
| 16:07:32 | mriedem | looks like we regressed that in queens | |
| 16:08:44 | mriedem | blarg https://review.openstack.org/#/c/507638/25/nova/compute/manager.py@6252 | |
| 16:11:21 | dansmith | mriedem: are you saying we don't have a migration record if we do a forced? | |
| 16:11:30 | mriedem | dansmith: we do, but we don't put the allocations on it | |
| 16:11:38 | mriedem | b/c we don't go through the scheduler for forced | |
| 16:11:46 | dansmith | um | |
| 16:11:51 | mriedem | this is just one of the many reasons for the dreaded -5 in dublin | |
| 16:12:30 | mriedem | dansmith: forced live migration calls this method to double up the allocations from the source to the forced dest https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/scheduler/utils.py#L473 | |
| 16:12:40 | mriedem | that's from pike when doubling was all the rage | |
| 16:12:49 | dansmith | okay, so you're saying on forced we don't do the migration allocations, we just allocate against the newhost, then if we have to revert, we don't have the migration allocations to revert to the instance? | |
| 16:12:55 | melwitt | is it safe to recheck yet? I didn't see another status update | |
| 16:13:00 | mriedem | dansmith: correct | |
| 16:13:04 | mriedem | melwitt: yeah i just did | |
| 16:13:19 | melwitt | ok | |
| 16:13:27 | mriedem | dansmith: i'll write a functional test for it when i'm back from lunch | |
| 16:13:36 | dansmith | mriedem: okay but the doubling is not intentional, just incidental since we didn't replace the instance allocs with the migration one yeah? | |
| 16:13:51 | mriedem | it's intentional | |
| 16:13:59 | mriedem | it mimics the behavior of doubling in the scheduler from before quens | |
| 16:14:00 | mriedem | *queens | |
| 16:14:08 | dansmith | right, but we shouldn't be doing any doubling anymore | |
| 16:14:17 | mriedem | sure, | |
| 16:14:19 | mriedem | but we are :) | |
| 16:14:20 | mriedem | for forced | |
| 16:14:24 | mriedem | b/c forced is FUN | |
| 16:14:26 | mriedem | -20! | |
| 16:14:33 | dansmith | I'm saying we shouldn't intend to be doing that, | |
| 16:14:40 | mriedem | not anymore no | |
| 16:14:44 | dansmith | which means it's a case we missed in converting to non-doubling | |
| 16:14:47 | mriedem | but we missed it in queens with your bp | |
| 16:14:49 | mriedem | yes | |
| 16:14:52 | dansmith | right, that's what I mean | |
| 16:14:55 | dansmith | unintentional | |
| 16:14:57 | mriedem | yeah | |
| 16:15:04 | mriedem | ok lunch | |
| 16:23:40 | sean-k-mooney | mriedem_away: im goint to choose to read -20! as -(20 factoral) to give it the weight it should have | |
| 16:25:07 | Sundar_ | efried: Please ping me when you have the time | |
| 16:26:01 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Revert non-reporting DISK_GB if sharing https://review.openstack.org/586614 | |
| 16:26:09 | fried_rice | mriedem_away, dansmith, cdent, melwitt: ^ | |
| 16:26:31 | fried_rice | Sundar_: Bad timing :( I have to run for a bit. Will you be around in a couple of hours? | |
| 16:27:23 | Sundar_ | NP, sure | |
| 16:40:26 | dansmith | mriedem_away: when you're back: I guess I don't really see the thing requiring the dynamic opts registration as being a bad thing | |
| 16:40:37 | dansmith | mriedem_away: it forces us to think about it when we write new code and the tests for it | |
| 16:49:37 | openstackgerrit | Dan Smith proposed openstack/nova master: Assorted cleanups from numa-aware-vswitches series https://review.openstack.org/582651 | |
| 16:49:38 | openstackgerrit | Dan Smith proposed openstack/nova master: Add additional functional tests for NUMA networks https://review.openstack.org/585385 | |
| 16:52:48 | cdent | melwitt, dansmith, mriedem_away : next week I'm pretty broadly available, so if stuff comes up and you want to wind me up and point me particular places, please ask. | |
| 16:54:05 | melwitt | will do, thanks | |
| 17:47:55 | colby_ | Hey Everyone. Im trying to get metrics based filtering working in nova. I tried enabling compute_monitors but I always get an error in the logs: | |
| 17:47:56 | colby_ | compute_monitors=["nova.compute.monitors.cpu.virt_driver", "numa_mem_bw.virt_driver"] | |
| 17:48:10 | colby_ | 2018-07-27 17:43:14.001 2295696 WARNING nova.compute.monitors [req-51711d41-c626-4af2-92fd-dde09c576fb2 - - - - -] Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors). | |
| 17:48:38 | colby_ | Ive tried variations on the monitor: cpu.virt_driver & just virt_driver. It always gives the same error | |
| 17:49:19 | colby_ | Im on pike, Centos, kvm | |
| 17:49:36 | colby_ | I have gnocchi running and collecting resource | |
| 17:50:11 | colby_ | 2018-07-27 17:44:36.110 2800963 INFO nova.filters [req-0e8215e5-e029-4104-8578-a917bf9edddc e28435e0a66740968c523e6376c57f68 18882d9c32ba42aeaa33c4703ad84b2c - default default] Filter MetricsFilter returned 0 hosts | |
| 17:50:33 | colby_ | Not sure where the problem is | |
| 17:51:20 | colby_ | weight_setting=compute.node.cpu.percent=-1.0 | |
| 17:51:32 | dansmith | colby_: I really can't help you, but I can tell you that metrics have nothing to do with gnocchi/ceilo | |
| 17:51:52 | colby_ | ok I thought I read somewhere that it used the gnocchi metrics... | |
| 17:51:53 | dansmith | colby_: the computes have to be configured to report them in order to use the filter | |
| 17:51:56 | dansmith | nope | |
| 17:52:21 | colby_ | ok so then the compute_monitors is the issue then | |
| 17:52:25 | dansmith | the metrics come from libvirt, reported by the compute, used by the filter | |
| 17:53:05 | colby_ | ok then Im not sure why Im not getting the metrics | |
| 17:53:12 | colby_ | besides the filed driver load | |
| 17:53:26 | colby_ | or monitor load I mean | |
| 17:53:30 | dansmith | yeah, I can't really help beyond that | |
| 17:54:27 | sean-k-mooney | dansmith: colby_ if you enable the metric reporting on the compute node ceilometer is able to read them form the message bus and store them but that is a sideffect | |
| 17:55:51 | colby_ | Ok so does that mean my metrics reporting is working? | |
| 17:56:05 | sean-k-mooney | colby_: by the way memory bandwith monitoring is broken on skylake. both read and write metrics are are actully read... | |
| 17:56:28 | colby_ | Im actually just interested in the cpu.percent | |
| 17:57:00 | colby_ | I want to not put instances on nodes with high cpu usage. We have a large memory node and the scheduler always puts instances there even when its way overcommited on cpu | |
| 17:58:12 | sean-k-mooney | ah well you could just change the order of the weigher to prefer weighing on cpus before memory. am but i have not used the metric based weigher myself so i have not tried to configure it before | |
| 17:59:07 | colby_ | ah ok. I treid weight_setting=cpu.percent=-1.0 | |
| 17:59:21 | colby_ | but I got zero hosts returned with metrics filter enabled | |
| 18:00:48 | colby_ | I was not aware that changing weigher order made any difference | |
| 18:01:24 | colby_ | I just used: nova.scheduler.weights.all_weighers | |