| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-27 | |||
| 16:13:59 | mriedem | it mimics the behavior of doubling in the scheduler from before quens | |
| 16:14:00 | mriedem | *queens | |
| 16:14:08 | dansmith | right, but we shouldn't be doing any doubling anymore | |
| 16:14:17 | mriedem | sure, | |
| 16:14:19 | mriedem | but we are :) | |
| 16:14:20 | mriedem | for forced | |
| 16:14:24 | mriedem | b/c forced is FUN | |
| 16:14:26 | mriedem | -20! | |
| 16:14:33 | dansmith | I'm saying we shouldn't intend to be doing that, | |
| 16:14:40 | mriedem | not anymore no | |
| 16:14:44 | dansmith | which means it's a case we missed in converting to non-doubling | |
| 16:14:47 | mriedem | but we missed it in queens with your bp | |
| 16:14:49 | mriedem | yes | |
| 16:14:52 | dansmith | right, that's what I mean | |
| 16:14:55 | dansmith | unintentional | |
| 16:14:57 | mriedem | yeah | |
| 16:15:04 | mriedem | ok lunch | |
| 16:23:40 | sean-k-mooney | mriedem_away: im goint to choose to read -20! as -(20 factoral) to give it the weight it should have | |
| 16:25:07 | Sundar_ | efried: Please ping me when you have the time | |
| 16:26:01 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Revert non-reporting DISK_GB if sharing https://review.openstack.org/586614 | |
| 16:26:09 | fried_rice | mriedem_away, dansmith, cdent, melwitt: ^ | |
| 16:26:31 | fried_rice | Sundar_: Bad timing :( I have to run for a bit. Will you be around in a couple of hours? | |
| 16:27:23 | Sundar_ | NP, sure | |
| 16:40:26 | dansmith | mriedem_away: when you're back: I guess I don't really see the thing requiring the dynamic opts registration as being a bad thing | |
| 16:40:37 | dansmith | mriedem_away: it forces us to think about it when we write new code and the tests for it | |
| 16:49:37 | openstackgerrit | Dan Smith proposed openstack/nova master: Assorted cleanups from numa-aware-vswitches series https://review.openstack.org/582651 | |
| 16:49:38 | openstackgerrit | Dan Smith proposed openstack/nova master: Add additional functional tests for NUMA networks https://review.openstack.org/585385 | |
| 16:52:48 | cdent | melwitt, dansmith, mriedem_away : next week I'm pretty broadly available, so if stuff comes up and you want to wind me up and point me particular places, please ask. | |
| 16:54:05 | melwitt | will do, thanks | |
| 17:47:55 | colby_ | Hey Everyone. Im trying to get metrics based filtering working in nova. I tried enabling compute_monitors but I always get an error in the logs: | |
| 17:47:56 | colby_ | compute_monitors=["nova.compute.monitors.cpu.virt_driver", "numa_mem_bw.virt_driver"] | |
| 17:48:10 | colby_ | 2018-07-27 17:43:14.001 2295696 WARNING nova.compute.monitors [req-51711d41-c626-4af2-92fd-dde09c576fb2 - - - - -] Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors). | |
| 17:48:38 | colby_ | Ive tried variations on the monitor: cpu.virt_driver & just virt_driver. It always gives the same error | |
| 17:49:19 | colby_ | Im on pike, Centos, kvm | |
| 17:49:36 | colby_ | I have gnocchi running and collecting resource | |
| 17:50:11 | colby_ | 2018-07-27 17:44:36.110 2800963 INFO nova.filters [req-0e8215e5-e029-4104-8578-a917bf9edddc e28435e0a66740968c523e6376c57f68 18882d9c32ba42aeaa33c4703ad84b2c - default default] Filter MetricsFilter returned 0 hosts | |
| 17:50:33 | colby_ | Not sure where the problem is | |
| 17:51:20 | colby_ | weight_setting=compute.node.cpu.percent=-1.0 | |
| 17:51:32 | dansmith | colby_: I really can't help you, but I can tell you that metrics have nothing to do with gnocchi/ceilo | |
| 17:51:52 | colby_ | ok I thought I read somewhere that it used the gnocchi metrics... | |
| 17:51:53 | dansmith | colby_: the computes have to be configured to report them in order to use the filter | |
| 17:51:56 | dansmith | nope | |
| 17:52:21 | colby_ | ok so then the compute_monitors is the issue then | |
| 17:52:25 | dansmith | the metrics come from libvirt, reported by the compute, used by the filter | |
| 17:53:05 | colby_ | ok then Im not sure why Im not getting the metrics | |
| 17:53:12 | colby_ | besides the filed driver load | |
| 17:53:26 | colby_ | or monitor load I mean | |
| 17:53:30 | dansmith | yeah, I can't really help beyond that | |
| 17:54:27 | sean-k-mooney | dansmith: colby_ if you enable the metric reporting on the compute node ceilometer is able to read them form the message bus and store them but that is a sideffect | |
| 17:55:51 | colby_ | Ok so does that mean my metrics reporting is working? | |
| 17:56:05 | sean-k-mooney | colby_: by the way memory bandwith monitoring is broken on skylake. both read and write metrics are are actully read... | |
| 17:56:28 | colby_ | Im actually just interested in the cpu.percent | |
| 17:57:00 | colby_ | I want to not put instances on nodes with high cpu usage. We have a large memory node and the scheduler always puts instances there even when its way overcommited on cpu | |
| 17:58:12 | sean-k-mooney | ah well you could just change the order of the weigher to prefer weighing on cpus before memory. am but i have not used the metric based weigher myself so i have not tried to configure it before | |
| 17:59:07 | colby_ | ah ok. I treid weight_setting=cpu.percent=-1.0 | |
| 17:59:21 | colby_ | but I got zero hosts returned with metrics filter enabled | |
| 18:00:48 | colby_ | I was not aware that changing weigher order made any difference | |
| 18:01:24 | colby_ | I just used: nova.scheduler.weights.all_weighers | |
| 18:02:06 | colby_ | I thought it was all just based on multipliers | |
| 18:02:57 | sean-k-mooney | colby_: well stickly speaking it does not but what i ment was listing only the weighers you care about and then setting there multipliers | |
| 18:03:50 | sean-k-mooney | if you only care about cpus then you can simploy only enable the cpu Weigher | |
| 18:04:06 | colby_ | hmm ok | |
| 18:04:09 | colby_ | thanks | |
| 18:04:11 | melwitt | colby_: are you specifying compute_monitors= under the [DEFAULT] section of the nova.conf? | |
| 18:04:23 | colby_ | yes | |
| 18:04:55 | colby_ | but I get the error: Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors) | |
| 18:05:11 | melwitt | okay. the log message you posted earlier is saying it doesn't find the monitor in the list from the conf option. hm | |
| 18:06:28 | colby_ | oh wait...there is a typo <smacks head> | |
| 18:08:25 | sean-k-mooney | colby_: you could proably get a similar effect by setting ram_weight_multiplier=0 or 0.1 so that ram is basically ignored when weighing if that does not work | |
| 18:08:54 | colby_ | ok thanks for your help! | |
| 18:19:47 | sean-k-mooney | melwitt: mriedem1 https://review.openstack.org/#/c/586568/ hit the retry_limit issue after your last recheck. is that issue(retry_limit) still happening in the gate | |
| 18:19:59 | melwitt | I think it's been fixed | |
| 18:20:35 | sean-k-mooney | well there is no gate job for that patch at the moment. will i retry it? | |
| 18:21:03 | melwitt | yeah, go ahead. I didn't realize that one hadn't been rechecked | |
| 18:22:02 | sean-k-mooney | melwitt: it had. you did it at 5:14 but it hit the error again | |
| 18:22:24 | sean-k-mooney | you proably missed the fix by a few minutes | |
| 18:22:30 | melwitt | yeah, guh | |
| 18:22:37 | mriedem1 | dansmith: danicus, i have good pleasurable news | |
| 18:22:48 | dansmith | um | |
| 18:23:01 | openstack | bug 1784022 in OpenStack Compute (nova) queens "Failed forced live migration does not rollback doubled up allocations in placement" [High,Triaged] https://launchpad.net/bugs/1784022 | |
| 18:23:01 | mriedem | bug 1784022 isn't a problem | |
| 18:23:03 | mriedem | it's handled | |
| 18:23:08 | dansmith | oh yeah? | |
| 18:23:15 | dansmith | that is indeed pleasurable | |
| 18:24:54 | melwitt | dansmith: wanna ack this? https://review.openstack.org/586614 | |
| 18:25:53 | dansmith | yup | |
| 18:26:11 | melwitt | hooray | |
| 18:27:07 | openstack | Launchpad bug 1708433 in OpenStack Compute (nova) "Attaching sriov nic VM fail with keyError pci_slot" [Undecided,New] | |
| 18:27:07 | melwitt | dangit, missed artom again. I had wanted to ask him about https://bugs.launchpad.net/nova/+bug/1708433 | |
| 18:28:51 | mriedem | dansmith: i'll push up the functional test anyway since it didn't look like we had one, only for the non-forced rollback checks | |
| 18:28:58 | dansmith | okay | |
| 18:29:07 | dansmith | mriedem: did you see my comment above about stephen's set? | |
| 18:29:13 | dansmith | and I pushed up the other fixes to that, btw | |
| 18:29:24 | dansmith | since you hadn't and seemingly got distracted with this other thing | |
| 18:29:56 | dansmith | oh I see you did | |
| 18:29:57 | dansmith | cool | |
| 18:33:18 | mriedem | sure did | |
| 18:33:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for forced live migration rollback allocs https://review.openstack.org/586636 | |
| 18:35:21 | mriedem | well, just in time for us to kill the shared storage provider support, i got it passing the ceph job http://logs.openstack.org/63/586363/3/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/569c574/ | |
| 18:36:50 | dansmith | presumably because we're left with broken allocations after a revert or something, but don't check/assert them? | |