| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-27 | |||
| 16:00:49 | mriedem | oops | |
| 16:01:21 | mriedem | i'll write a functional test for the rollback forced live migration case | |
| 16:06:08 | openstack | Launchpad bug 1784022 in OpenStack Compute (nova) "Failed forced live migration does not rollback doubled up allocations in placement" [High,Triaged] | |
| 16:06:08 | mriedem | https://bugs.launchpad.net/nova/+bug/1784022 | |
| 16:07:32 | mriedem | looks like we regressed that in queens | |
| 16:08:44 | mriedem | blarg https://review.openstack.org/#/c/507638/25/nova/compute/manager.py@6252 | |
| 16:11:21 | dansmith | mriedem: are you saying we don't have a migration record if we do a forced? | |
| 16:11:30 | mriedem | dansmith: we do, but we don't put the allocations on it | |
| 16:11:38 | mriedem | b/c we don't go through the scheduler for forced | |
| 16:11:46 | dansmith | um | |
| 16:11:51 | mriedem | this is just one of the many reasons for the dreaded -5 in dublin | |
| 16:12:30 | mriedem | dansmith: forced live migration calls this method to double up the allocations from the source to the forced dest https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/scheduler/utils.py#L473 | |
| 16:12:40 | mriedem | that's from pike when doubling was all the rage | |
| 16:12:49 | dansmith | okay, so you're saying on forced we don't do the migration allocations, we just allocate against the newhost, then if we have to revert, we don't have the migration allocations to revert to the instance? | |
| 16:12:55 | melwitt | is it safe to recheck yet? I didn't see another status update | |
| 16:13:00 | mriedem | dansmith: correct | |
| 16:13:04 | mriedem | melwitt: yeah i just did | |
| 16:13:19 | melwitt | ok | |
| 16:13:27 | mriedem | dansmith: i'll write a functional test for it when i'm back from lunch | |
| 16:13:36 | dansmith | mriedem: okay but the doubling is not intentional, just incidental since we didn't replace the instance allocs with the migration one yeah? | |
| 16:13:51 | mriedem | it's intentional | |
| 16:13:59 | mriedem | it mimics the behavior of doubling in the scheduler from before quens | |
| 16:14:00 | mriedem | *queens | |
| 16:14:08 | dansmith | right, but we shouldn't be doing any doubling anymore | |
| 16:14:17 | mriedem | sure, | |
| 16:14:19 | mriedem | but we are :) | |
| 16:14:20 | mriedem | for forced | |
| 16:14:24 | mriedem | b/c forced is FUN | |
| 16:14:26 | mriedem | -20! | |
| 16:14:33 | dansmith | I'm saying we shouldn't intend to be doing that, | |
| 16:14:40 | mriedem | not anymore no | |
| 16:14:44 | dansmith | which means it's a case we missed in converting to non-doubling | |
| 16:14:47 | mriedem | but we missed it in queens with your bp | |
| 16:14:49 | mriedem | yes | |
| 16:14:52 | dansmith | right, that's what I mean | |
| 16:14:55 | dansmith | unintentional | |
| 16:14:57 | mriedem | yeah | |
| 16:15:04 | mriedem | ok lunch | |
| 16:23:40 | sean-k-mooney | mriedem_away: im goint to choose to read -20! as -(20 factoral) to give it the weight it should have | |
| 16:25:07 | Sundar_ | efried: Please ping me when you have the time | |
| 16:26:01 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Revert non-reporting DISK_GB if sharing https://review.openstack.org/586614 | |
| 16:26:09 | fried_rice | mriedem_away, dansmith, cdent, melwitt: ^ | |
| 16:26:31 | fried_rice | Sundar_: Bad timing :( I have to run for a bit. Will you be around in a couple of hours? | |
| 16:27:23 | Sundar_ | NP, sure | |
| 16:40:26 | dansmith | mriedem_away: when you're back: I guess I don't really see the thing requiring the dynamic opts registration as being a bad thing | |
| 16:40:37 | dansmith | mriedem_away: it forces us to think about it when we write new code and the tests for it | |
| 16:49:37 | openstackgerrit | Dan Smith proposed openstack/nova master: Assorted cleanups from numa-aware-vswitches series https://review.openstack.org/582651 | |
| 16:49:38 | openstackgerrit | Dan Smith proposed openstack/nova master: Add additional functional tests for NUMA networks https://review.openstack.org/585385 | |
| 16:52:48 | cdent | melwitt, dansmith, mriedem_away : next week I'm pretty broadly available, so if stuff comes up and you want to wind me up and point me particular places, please ask. | |
| 16:54:05 | melwitt | will do, thanks | |
| 17:47:55 | colby_ | Hey Everyone. Im trying to get metrics based filtering working in nova. I tried enabling compute_monitors but I always get an error in the logs: | |
| 17:47:56 | colby_ | compute_monitors=["nova.compute.monitors.cpu.virt_driver", "numa_mem_bw.virt_driver"] | |
| 17:48:10 | colby_ | 2018-07-27 17:43:14.001 2295696 WARNING nova.compute.monitors [req-51711d41-c626-4af2-92fd-dde09c576fb2 - - - - -] Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors). | |
| 17:48:38 | colby_ | Ive tried variations on the monitor: cpu.virt_driver & just virt_driver. It always gives the same error | |
| 17:49:19 | colby_ | Im on pike, Centos, kvm | |
| 17:49:36 | colby_ | I have gnocchi running and collecting resource | |
| 17:50:11 | colby_ | 2018-07-27 17:44:36.110 2800963 INFO nova.filters [req-0e8215e5-e029-4104-8578-a917bf9edddc e28435e0a66740968c523e6376c57f68 18882d9c32ba42aeaa33c4703ad84b2c - default default] Filter MetricsFilter returned 0 hosts | |
| 17:50:33 | colby_ | Not sure where the problem is | |
| 17:51:20 | colby_ | weight_setting=compute.node.cpu.percent=-1.0 | |
| 17:51:32 | dansmith | colby_: I really can't help you, but I can tell you that metrics have nothing to do with gnocchi/ceilo | |
| 17:51:52 | colby_ | ok I thought I read somewhere that it used the gnocchi metrics... | |
| 17:51:53 | dansmith | colby_: the computes have to be configured to report them in order to use the filter | |
| 17:51:56 | dansmith | nope | |
| 17:52:21 | colby_ | ok so then the compute_monitors is the issue then | |
| 17:52:25 | dansmith | the metrics come from libvirt, reported by the compute, used by the filter | |
| 17:53:05 | colby_ | ok then Im not sure why Im not getting the metrics | |
| 17:53:12 | colby_ | besides the filed driver load | |
| 17:53:26 | colby_ | or monitor load I mean | |
| 17:53:30 | dansmith | yeah, I can't really help beyond that | |
| 17:54:27 | sean-k-mooney | dansmith: colby_ if you enable the metric reporting on the compute node ceilometer is able to read them form the message bus and store them but that is a sideffect | |
| 17:55:51 | colby_ | Ok so does that mean my metrics reporting is working? | |
| 17:56:05 | sean-k-mooney | colby_: by the way memory bandwith monitoring is broken on skylake. both read and write metrics are are actully read... | |
| 17:56:28 | colby_ | Im actually just interested in the cpu.percent | |
| 17:57:00 | colby_ | I want to not put instances on nodes with high cpu usage. We have a large memory node and the scheduler always puts instances there even when its way overcommited on cpu | |
| 17:58:12 | sean-k-mooney | ah well you could just change the order of the weigher to prefer weighing on cpus before memory. am but i have not used the metric based weigher myself so i have not tried to configure it before | |
| 17:59:07 | colby_ | ah ok. I treid weight_setting=cpu.percent=-1.0 | |
| 17:59:21 | colby_ | but I got zero hosts returned with metrics filter enabled | |
| 18:00:48 | colby_ | I was not aware that changing weigher order made any difference | |
| 18:01:24 | colby_ | I just used: nova.scheduler.weights.all_weighers | |
| 18:02:06 | colby_ | I thought it was all just based on multipliers | |
| 18:02:57 | sean-k-mooney | colby_: well stickly speaking it does not but what i ment was listing only the weighers you care about and then setting there multipliers | |
| 18:03:50 | sean-k-mooney | if you only care about cpus then you can simploy only enable the cpu Weigher | |
| 18:04:06 | colby_ | hmm ok | |
| 18:04:09 | colby_ | thanks | |
| 18:04:11 | melwitt | colby_: are you specifying compute_monitors= under the [DEFAULT] section of the nova.conf? | |
| 18:04:23 | colby_ | yes | |
| 18:04:55 | colby_ | but I get the error: Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors) | |
| 18:05:11 | melwitt | okay. the log message you posted earlier is saying it doesn't find the monitor in the list from the conf option. hm | |
| 18:06:28 | colby_ | oh wait...there is a typo <smacks head> | |
| 18:08:25 | sean-k-mooney | colby_: you could proably get a similar effect by setting ram_weight_multiplier=0 or 0.1 so that ram is basically ignored when weighing if that does not work | |
| 18:08:54 | colby_ | ok thanks for your help! | |
| 18:19:47 | sean-k-mooney | melwitt: mriedem1 https://review.openstack.org/#/c/586568/ hit the retry_limit issue after your last recheck. is that issue(retry_limit) still happening in the gate | |
| 18:19:59 | melwitt | I think it's been fixed | |
| 18:20:35 | sean-k-mooney | well there is no gate job for that patch at the moment. will i retry it? | |
| 18:21:03 | melwitt | yeah, go ahead. I didn't realize that one hadn't been rechecked | |
| 18:22:02 | sean-k-mooney | melwitt: it had. you did it at 5:14 but it hit the error again | |
| 18:22:24 | sean-k-mooney | you proably missed the fix by a few minutes | |
| 18:22:30 | melwitt | yeah, guh | |
| 18:22:37 | mriedem1 | dansmith: danicus, i have good pleasurable news | |
| 18:22:48 | dansmith | um | |