| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-19 | |||
| 18:25:31 | spatel | i had same concern so i didn't bother to install | |
| 18:26:09 | sean-k-mooney | ya so in general i would check out collectd and see if it meets your needs | |
| 18:26:26 | spatel | I am going to try collectd sure! | |
| 18:26:49 | sean-k-mooney | my former team enhanced the collectd libvirt plugin to allow reporting stats for vms using the uuid field instead of the id field | |
| 18:27:02 | sean-k-mooney | the uuid filed is set to the nova instance uuid | |
| 18:27:17 | sean-k-mooney | so its easy to do a 1:1 mapping between the stats and the workload | |
| 18:27:40 | spatel | hmm! interesting i think i have to look into collectd now | |
| 18:28:04 | spatel | also i was thinking push collectd data to influx + grafana so i have good dashboard | |
| 18:28:08 | sean-k-mooney | you can also have collectd use its network plugin to send the stats to influxdb and then use graphana to visualise the results | |
| 18:28:15 | sean-k-mooney | jinks | |
| 18:28:32 | sean-k-mooney | :) that is a good solution | |
| 18:28:49 | spatel | yup! | |
| 18:29:15 | sean-k-mooney | are you familar with OPNFV? | |
| 18:29:23 | spatel | no much | |
| 18:29:54 | sean-k-mooney | thats ok its rather vast in scope like openstack | |
| 18:30:12 | sean-k-mooney | spatel: i just wanted to highlight the barometer project https://wiki.opnfv.org/display/fastpath/Barometer+Home | |
| 18:31:01 | spatel | reading.. | |
| 18:32:13 | sean-k-mooney | spatel: they have be working with integrating collect graphana and other tools for openstack monitoring for telco/NFV usecases | |
| 18:32:16 | sean-k-mooney | https://www.youtube.com/watch?v=-82UShFiBBM | |
| 18:32:26 | spatel | sean-k-mooney: very interesting... also i had question about SR-IOV VF nic stats, it doesn't tell you how much data flowing through your VF | |
| 18:32:57 | sean-k-mooney | spatel: that depend on the nic | |
| 18:33:18 | spatel | I have Qlogic so i doubt it has that feature | |
| 18:33:23 | sean-k-mooney | so most nics dont have enough hardware counters to gatar stats on all the VFs from the host side | |
| 18:34:00 | sean-k-mooney | spatel: if you are able to run collectd in the guest however it can monitor the kernel stats and give you that info | |
| 18:34:50 | spatel | Yes! that is what i am doing currently! guest base snmp monitoring | |
| 18:34:59 | sean-k-mooney | spatel: one other tool that you might also find userful in this space is skydive. http://skydive.network/ | |
| 18:35:35 | spatel | hmm! looking cool | |
| 18:35:51 | spatel | we are using observium and Cisco DCNM | |
| 18:36:49 | spatel | hmm! they have openstack support too | |
| 18:37:11 | sean-k-mooney | skydive was developed in the last 2 years ago specificly for cloud and conterised enviornments but it began to mature just after i deployed my last cloud so i have not used it myself | |
| 18:38:29 | spatel | Can i use for Cisco switches and router? | |
| 18:38:52 | spatel | look like they use agent do i doubt | |
| 18:38:58 | sean-k-mooney | i belive so. it uses multiple protocols to do its monitoring | |
| 18:40:05 | sean-k-mooney | if thet cisco switch support sflow i belive it would work | |
| 18:40:39 | spatel | We have all Cisco nexus switches and they do sflow :) i am going to explore it now | |
| 18:41:41 | spatel | Do you guys using collectd to monitor your compute nodes? | |
| 18:41:46 | mriedem_afk | awaugama: as far as i know if you set the cpu_allocation_ratio in nova.conf on the nova-compute service, it should mirror that to the new OR existing resource provider in placement | |
| 18:41:57 | mriedem | if it's nit mirroring those updates i think that's a bug | |
| 18:42:16 | mriedem | SteelyDan: on that bug, tbc, i think they got confused about "scheduling" | |
| 18:42:32 | awaugama | mriedem: that's what I thought would happen. sean-k-mooney you think it won't update existing ones? | |
| 18:42:32 | mriedem | it's not scheduling anything, they just power on an instance on a stopped compute and the rpc cast is sent to the void | |
| 18:43:09 | sean-k-mooney | awaugama: im not sure. | |
| 18:43:22 | mriedem | cfriesen: yes the bug is saying they power on the instance, the api changes the task_state, but b/c the compute is down, we never finish the task and it's "stuck" | |
| 18:43:46 | sean-k-mooney | awaugama: it will do one of two things either it will overright it allways with the config value or it will only use the config value on creattion fo the rp | |
| 18:43:47 | mriedem | fixing that would mean needing to check the compute service status for every instance action... | |
| 18:44:19 | sean-k-mooney | awaugama: both cases are "wronge" depending on who you ask hench the sepc for initall allcoation ratios https://review.openstack.org/#/c/552105/ | |
| 18:44:56 | mriedem | on every periodic, we update the resource class inventory allocation ratio based on config that we send to placement https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/compute/resource_tracker.py#L108 | |
| 18:45:09 | mriedem | https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/compute/resource_tracker.py#L952 | |
| 18:45:24 | mriedem | the question is if prov_tree.update_inventory(nodename, inv_data) has a bug thinking that nothing changed | |
| 18:45:25 | mriedem | b/c it's a cache | |
| 18:45:34 | mriedem | and the reportclient itself has a cache of the provider tree | |
| 18:46:04 | mriedem | https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1501 | |
| 18:46:08 | mriedem | "The specified ProviderTree is compared against the local cache. Any changes are flushed back to the placement service. " | |
| 18:47:37 | awaugama | makes sense | |
| 18:48:23 | mriedem | the bug is probably here https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1575-L1576 | |
| 18:48:40 | mriedem | if we have a single compute node resource provider and that doesn't change, both of those sets will be empty | |
| 18:48:43 | cfriesen | mriedem: as I said in the bug, that won't actually fix it, just make the race window smaller. | |
| 18:48:47 | mriedem | and the for loops below won't flush any changes to placement | |
| 18:48:54 | mriedem | cfriesen: isure | |
| 18:48:56 | mriedem | *sure | |
| 18:48:59 | mriedem | b/c of the service group api | |
| 18:49:03 | mriedem | unless you force down the service | |
| 18:49:47 | mriedem | awaugama: in queens we weren't using that provider tree stuff | |
| 18:49:47 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/compute/resource_tracker.py#L883 | |
| 18:50:11 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1112 | |
| 18:50:33 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L850 | |
| 18:50:34 | cdent | because it is friday and I don't feel like filtering, can I just say: god I hate caches, why do we do caches? | |
| 18:50:44 | mriedem | idk | |
| 18:50:53 | mriedem | to avoid calling the placement API 500 times per periodic? | |
| 18:51:13 | mriedem | i just know caches are very tricky | |
| 18:51:15 | mriedem | brittle | |
| 18:51:19 | sean-k-mooney | cdent: so the processor has another way to mess with your view of a sequtially constent exectuion of your program | |
| 18:51:41 | awaugama | mriedem: so basically it's a stale value and placement is never refreshing to pick up the new conf setting? | |
| 18:52:01 | awaugama | or the rp isn't refreshing? | |
| 18:52:31 | sean-k-mooney | cdent: hehe since you are here can i get your input on https://review.openstack.org/#/c/610034/ | |
| 18:52:35 | mriedem | https://github.com/openstack/nova/blob/stable/queens/nova/compute/provider_tree.py#L124 | |
| 18:52:51 | sean-k-mooney | cdent: actully perhaps the placement channel would be better. | |
| 18:53:00 | mriedem | awaugama: i think the scheduler report client in master/rocky is thinking nothing is changing | |
| 18:53:04 | cdent | yeah, join me over there because I think that may be fixed | |
| 18:53:22 | mriedem | i just had doritos, you probably don't want to kiss me | |
| 18:53:25 | mriedem | at least not open mouth | |
| 18:53:57 | awaugama | this channel gets weird on Fridays | |
| 18:54:25 | mriedem | awaugama: i think on master/rocky the problem is we're not getting here https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1646 | |
| 18:54:41 | mriedem | awaugama: it should be pretty easy to recreate this | |
| 18:55:02 | awaugama | mriedem: I think finucannot was able to reproduce on his system | |
| 18:55:13 | awaugama | think he was just using devstack | |
| 18:55:45 | mriedem | that or https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1112 is short circuirting | |
| 18:55:47 | mriedem | *circuiting | |
| 18:56:06 | mriedem | at one point i had a debug patch for a bunch of this b/c it's really hard to know wtf is going on without any logs | |
| 18:56:51 | awaugama | mriedem, I had to redeploy my system for another feature test, I can see about reproducing next week | |
| 18:56:54 | mriedem | i'll see if i can dredge that up | |
| 18:56:54 | awaugama | with logs | |
| 18:57:55 | mriedem | awaugama: https://review.openstack.org/#/c/597560/ | |
| 18:59:23 | awaugama | cool, I'll make a note of that patch and see if I can get it applied | |
| 18:59:36 | mriedem | the stuff in here is probably still useful https://review.openstack.org/#/c/597560/6/nova/scheduler/client/report.py | |
| 18:59:43 | mriedem | the rest was for debugging a specific thing that is now fixed | |
| 18:59:53 | mriedem | maybe i'll restore and rev that to clean it up | |
| 19:00:05 | mriedem | on top of https://review.openstack.org/#/c/597553/ | |
| 19:19:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log the operation when updating generation in ProviderTree https://review.openstack.org/597553 | |
| 19:19:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add debug logs for when provider inventory changes https://review.openstack.org/597560 | |