| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-27 | |||
| 16:49:37 | openstackgerrit | Dan Smith proposed openstack/nova master: Assorted cleanups from numa-aware-vswitches series https://review.openstack.org/582651 | |
| 16:49:38 | openstackgerrit | Dan Smith proposed openstack/nova master: Add additional functional tests for NUMA networks https://review.openstack.org/585385 | |
| 16:52:48 | cdent | melwitt, dansmith, mriedem_away : next week I'm pretty broadly available, so if stuff comes up and you want to wind me up and point me particular places, please ask. | |
| 16:54:05 | melwitt | will do, thanks | |
| 17:47:55 | colby_ | Hey Everyone. Im trying to get metrics based filtering working in nova. I tried enabling compute_monitors but I always get an error in the logs: | |
| 17:47:56 | colby_ | compute_monitors=["nova.compute.monitors.cpu.virt_driver", "numa_mem_bw.virt_driver"] | |
| 17:48:10 | colby_ | 2018-07-27 17:43:14.001 2295696 WARNING nova.compute.monitors [req-51711d41-c626-4af2-92fd-dde09c576fb2 - - - - -] Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors). | |
| 17:48:38 | colby_ | Ive tried variations on the monitor: cpu.virt_driver & just virt_driver. It always gives the same error | |
| 17:49:19 | colby_ | Im on pike, Centos, kvm | |
| 17:49:36 | colby_ | I have gnocchi running and collecting resource | |
| 17:50:11 | colby_ | 2018-07-27 17:44:36.110 2800963 INFO nova.filters [req-0e8215e5-e029-4104-8578-a917bf9edddc e28435e0a66740968c523e6376c57f68 18882d9c32ba42aeaa33c4703ad84b2c - default default] Filter MetricsFilter returned 0 hosts | |
| 17:50:33 | colby_ | Not sure where the problem is | |
| 17:51:20 | colby_ | weight_setting=compute.node.cpu.percent=-1.0 | |
| 17:51:32 | dansmith | colby_: I really can't help you, but I can tell you that metrics have nothing to do with gnocchi/ceilo | |
| 17:51:52 | colby_ | ok I thought I read somewhere that it used the gnocchi metrics... | |
| 17:51:53 | dansmith | colby_: the computes have to be configured to report them in order to use the filter | |
| 17:51:56 | dansmith | nope | |
| 17:52:21 | colby_ | ok so then the compute_monitors is the issue then | |
| 17:52:25 | dansmith | the metrics come from libvirt, reported by the compute, used by the filter | |
| 17:53:05 | colby_ | ok then Im not sure why Im not getting the metrics | |
| 17:53:12 | colby_ | besides the filed driver load | |
| 17:53:26 | colby_ | or monitor load I mean | |
| 17:53:30 | dansmith | yeah, I can't really help beyond that | |
| 17:54:27 | sean-k-mooney | dansmith: colby_ if you enable the metric reporting on the compute node ceilometer is able to read them form the message bus and store them but that is a sideffect | |
| 17:55:51 | colby_ | Ok so does that mean my metrics reporting is working? | |
| 17:56:05 | sean-k-mooney | colby_: by the way memory bandwith monitoring is broken on skylake. both read and write metrics are are actully read... | |
| 17:56:28 | colby_ | Im actually just interested in the cpu.percent | |
| 17:57:00 | colby_ | I want to not put instances on nodes with high cpu usage. We have a large memory node and the scheduler always puts instances there even when its way overcommited on cpu | |
| 17:58:12 | sean-k-mooney | ah well you could just change the order of the weigher to prefer weighing on cpus before memory. am but i have not used the metric based weigher myself so i have not tried to configure it before | |
| 17:59:07 | colby_ | ah ok. I treid weight_setting=cpu.percent=-1.0 | |
| 17:59:21 | colby_ | but I got zero hosts returned with metrics filter enabled | |
| 18:00:48 | colby_ | I was not aware that changing weigher order made any difference | |
| 18:01:24 | colby_ | I just used: nova.scheduler.weights.all_weighers | |
| 18:02:06 | colby_ | I thought it was all just based on multipliers | |
| 18:02:57 | sean-k-mooney | colby_: well stickly speaking it does not but what i ment was listing only the weighers you care about and then setting there multipliers | |
| 18:03:50 | sean-k-mooney | if you only care about cpus then you can simploy only enable the cpu Weigher | |
| 18:04:06 | colby_ | hmm ok | |
| 18:04:09 | colby_ | thanks | |
| 18:04:11 | melwitt | colby_: are you specifying compute_monitors= under the [DEFAULT] section of the nova.conf? | |
| 18:04:23 | colby_ | yes | |
| 18:04:55 | colby_ | but I get the error: Excluding nova.compute.monitors.cpu monitor virt_driver. Not in the list of enabled monitors (CONF.compute_monitors) | |
| 18:05:11 | melwitt | okay. the log message you posted earlier is saying it doesn't find the monitor in the list from the conf option. hm | |
| 18:06:28 | colby_ | oh wait...there is a typo <smacks head> | |
| 18:08:25 | sean-k-mooney | colby_: you could proably get a similar effect by setting ram_weight_multiplier=0 or 0.1 so that ram is basically ignored when weighing if that does not work | |
| 18:08:54 | colby_ | ok thanks for your help! | |
| 18:19:47 | sean-k-mooney | melwitt: mriedem1 https://review.openstack.org/#/c/586568/ hit the retry_limit issue after your last recheck. is that issue(retry_limit) still happening in the gate | |
| 18:19:59 | melwitt | I think it's been fixed | |
| 18:20:35 | sean-k-mooney | well there is no gate job for that patch at the moment. will i retry it? | |
| 18:21:03 | melwitt | yeah, go ahead. I didn't realize that one hadn't been rechecked | |
| 18:22:02 | sean-k-mooney | melwitt: it had. you did it at 5:14 but it hit the error again | |
| 18:22:24 | sean-k-mooney | you proably missed the fix by a few minutes | |
| 18:22:30 | melwitt | yeah, guh | |
| 18:22:37 | mriedem1 | dansmith: danicus, i have good pleasurable news | |
| 18:22:48 | dansmith | um | |
| 18:23:01 | openstack | bug 1784022 in OpenStack Compute (nova) queens "Failed forced live migration does not rollback doubled up allocations in placement" [High,Triaged] https://launchpad.net/bugs/1784022 | |
| 18:23:01 | mriedem | bug 1784022 isn't a problem | |
| 18:23:03 | mriedem | it's handled | |
| 18:23:08 | dansmith | oh yeah? | |
| 18:23:15 | dansmith | that is indeed pleasurable | |
| 18:24:54 | melwitt | dansmith: wanna ack this? https://review.openstack.org/586614 | |
| 18:25:53 | dansmith | yup | |
| 18:26:11 | melwitt | hooray | |
| 18:27:07 | openstack | Launchpad bug 1708433 in OpenStack Compute (nova) "Attaching sriov nic VM fail with keyError pci_slot" [Undecided,New] | |
| 18:27:07 | melwitt | dangit, missed artom again. I had wanted to ask him about https://bugs.launchpad.net/nova/+bug/1708433 | |
| 18:28:51 | mriedem | dansmith: i'll push up the functional test anyway since it didn't look like we had one, only for the non-forced rollback checks | |
| 18:28:58 | dansmith | okay | |
| 18:29:07 | dansmith | mriedem: did you see my comment above about stephen's set? | |
| 18:29:13 | dansmith | and I pushed up the other fixes to that, btw | |
| 18:29:24 | dansmith | since you hadn't and seemingly got distracted with this other thing | |
| 18:29:56 | dansmith | oh I see you did | |
| 18:29:57 | dansmith | cool | |
| 18:33:18 | mriedem | sure did | |
| 18:33:29 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for forced live migration rollback allocs https://review.openstack.org/586636 | |
| 18:35:21 | mriedem | well, just in time for us to kill the shared storage provider support, i got it passing the ceph job http://logs.openstack.org/63/586363/3/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/569c574/ | |
| 18:36:50 | dansmith | presumably because we're left with broken allocations after a revert or something, but don't check/assert them? | |
| 18:37:22 | mriedem | right tempest won't assert any of that stuff, | |
| 18:37:41 | mriedem | we do have a post-test hook in the nova-next job for making sure there are no orphaned allocations but only on compute node providers | |
| 18:37:50 | dansmith | we had some sanity checking and logging in the RT when we removed the healing.. maybe there is some evidence in there? | |
| 18:38:47 | mriedem | oh nvm it's not just computes, it's all resource providers | |
| 18:38:51 | mriedem | but we don't run it on that job | |
| 18:39:13 | dansmith | http://logs.openstack.org/63/586363/3/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/569c574/logs/screen-n-cpu.txt.gz#_Jul_27_17_31_35_337258 | |
| 18:40:23 | mriedem | yeah i don't see any obvious warnings related to allocatoins | |
| 18:40:52 | mriedem | i think if we ran our post-test leaked allocation hook on this job it would fail | |
| 18:41:11 | mriedem | well, maybe not for single node | |
| 18:42:56 | dansmith | yeah, so there are 133 logs of instance fd563ed2-d42c-4dc1-a614-8700c6e6c8fd | |
| 18:43:06 | dansmith | having non-cleaned-up allocations | |
| 18:43:36 | dansmith | although really the allocations that we'd destroy wouldn't be against the compute node, | |
| 18:43:39 | dansmith | and would be gone not stale | |
| 18:43:43 | dansmith | so even your check probably wouldn't catch it | |
| 18:43:54 | dansmith | because we'd be _losing_ not _leaking_ disk allocations | |
| 18:45:15 | dansmith | also, um | |
| 18:45:25 | dansmith | I just noticed that we're logging an entire console log out of privsep somewhere | |
| 18:45:34 | dansmith | http://logs.openstack.org/63/586363/3/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/569c574/logs/screen-n-cpu.txt.gz#_Jul_27_18_07_23_550670 | |
| 18:46:01 | dansmith | you could argue that is a security issue if instances log sensitive info to their console | |
| 18:46:31 | mriedem | nice, 9 of those | |
| 18:46:49 | mriedem | you can open that bug | |
| 18:47:47 | dansmith | okay | |
| 18:48:04 | dansmith | does privsep daemon log everything over the channel or something? | |
| 18:49:51 | sean-k-mooney | dansmith: that log is becasue seting a route in teh guest failed http://logs.openstack.org/63/586363/3/check/legacy-tempest-dsvm-full-devstack-plugin-ceph/569c574/logs/screen-n-cpu.txt.gz#_Jul_27_18_07_23_555618 | |
| 18:50:31 | dansmith | not sure about that | |