| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-19 | |||
| 17:42:12 | sean-k-mooney | awaugama: he is usually enjoying his weekend by now :) | |
| 17:50:39 | awaugama | sean-k-mooney, yeah, I figure he'll get the scrollback at somepoint | |
| 17:57:41 | sean-k-mooney | awaugama: the inital allocation ratios spec is likely the one that is most relevent to you https://review.openstack.org/#/c/552105/ | |
| 18:04:31 | spatel | sean-k-mooney: hey! how are you doing :) | |
| 18:04:54 | sean-k-mooney | spatel: quite well thank you. how are you :) | |
| 18:04:57 | spatel | after longtime i am seeing you online or may be i was not paying attention | |
| 18:05:13 | sean-k-mooney | i was on company traing most of this week | |
| 18:05:21 | spatel | I am great!! and my openstack also going great | |
| 18:05:46 | sean-k-mooney | so ya i was not online much. that is good to hear. | |
| 18:06:01 | spatel | I have added 80 SR-IOV compute nodes and put them in production :) | |
| 18:06:31 | sean-k-mooney | wow that was fast. are you happy with the feature set/perfromance you have achived? | |
| 18:06:51 | spatel | Yup!! i am not seeing any performance impact for my application so far | |
| 18:07:38 | sean-k-mooney | you decided to go with vnic_type direct with a flavor with hugepages and pinned cpus in the end? | |
| 18:07:59 | spatel | I have create two AZ group as we spoken last time, tor-1 and tor-2 and spreading application across two AZ | |
| 18:08:30 | spatel | Yes vnic_type=direct / hugepages / cpu pinning / numa=2 | |
| 18:08:46 | spatel | whatever setting you suggested last time | |
| 18:09:03 | sean-k-mooney | cool that should be a good starting point for any VNF deployed on openstack | |
| 18:09:23 | spatel | I am happy with those setting :) | |
| 18:09:27 | sean-k-mooney | and for the tor config did you split the control and dataplane across the tors or keep them on the same tor | |
| 18:11:08 | spatel | That i didn't tested yet because of deployment urgency! | |
| 18:11:18 | spatel | i need to test that in LAB first | |
| 18:11:55 | spatel | that is in my to-do list | |
| 18:12:32 | sean-k-mooney | well the original config you proposed should work but i was not certen the failure mode was optimal. that said you intended to add a deicated manament switch at a later poit so that will likely be the best longterm solution in anycase | |
| 18:12:38 | spatel | as soon as i get time i want to run test on dpdk (SR-IOV is still painful even its better) | |
| 18:14:17 | spatel | sean-k-mooney: yes i have plan in future to isolate mgmt traffic using extra nic | |
| 18:16:48 | spatel | sean-k-mooney: Quick question, how do you guys monitor instance stats? like CPU/memory from KVM | |
| 18:18:09 | spatel | currently i have snmp agent running on compute node and also inside instance but that won't give you 100% result right? i want to monitor hypervisor level stats | |
| 18:18:39 | spatel | KVM level monitoring in short | |
| 18:19:58 | sean-k-mooney | am you have several options | |
| 18:20:17 | sean-k-mooney | in generall i would recommened collectd for host level perfomance monitoring | |
| 18:20:53 | sean-k-mooney | if you have deployed the openstack ceilometer project you can also confure it to monitro the libvirt isntances | |
| 18:21:48 | sean-k-mooney | i belive the prometious project has good monitoring capablites for the openstack servcies but it does not have good monitoring capablites for the host or the vms as far as i am aware | |
| 18:22:21 | spatel | I didn't configure ceilometer because i wasn't sure i need that component because its private cloud and we don't need billing | |
| 18:22:55 | sean-k-mooney | spatel: that is a common missconceptions ceilometer is for telemetry not billing | |
| 18:23:09 | sean-k-mooney | that said its not nessisarly the best solution for telemetry either | |
| 18:24:10 | spatel | i was worried it will eat my resources | |
| 18:24:19 | sean-k-mooney | if you decide to use collectd you could look into https://github.com/openstack/collectd-openstack-plugins to plublis metrics to celometer or gnocci but collectd can also advertise the stata it monitors over snmp or toher protocols | |
| 18:24:54 | sean-k-mooney | spatel: yes that is a concern with celimiter. it does not scale well which is why i generally dont recomend it as my first choice | |
| 18:25:14 | spatel | Agreed ++ | |
| 18:25:31 | spatel | i had same concern so i didn't bother to install | |
| 18:26:09 | sean-k-mooney | ya so in general i would check out collectd and see if it meets your needs | |
| 18:26:26 | spatel | I am going to try collectd sure! | |
| 18:26:49 | sean-k-mooney | my former team enhanced the collectd libvirt plugin to allow reporting stats for vms using the uuid field instead of the id field | |
| 18:27:02 | sean-k-mooney | the uuid filed is set to the nova instance uuid | |
| 18:27:17 | sean-k-mooney | so its easy to do a 1:1 mapping between the stats and the workload | |
| 18:27:40 | spatel | hmm! interesting i think i have to look into collectd now | |
| 18:28:04 | spatel | also i was thinking push collectd data to influx + grafana so i have good dashboard | |
| 18:28:08 | sean-k-mooney | you can also have collectd use its network plugin to send the stats to influxdb and then use graphana to visualise the results | |
| 18:28:15 | sean-k-mooney | jinks | |
| 18:28:32 | sean-k-mooney | :) that is a good solution | |
| 18:28:49 | spatel | yup! | |
| 18:29:15 | sean-k-mooney | are you familar with OPNFV? | |
| 18:29:23 | spatel | no much | |
| 18:29:54 | sean-k-mooney | thats ok its rather vast in scope like openstack | |
| 18:30:12 | sean-k-mooney | spatel: i just wanted to highlight the barometer project https://wiki.opnfv.org/display/fastpath/Barometer+Home | |
| 18:31:01 | spatel | reading.. | |
| 18:32:13 | sean-k-mooney | spatel: they have be working with integrating collect graphana and other tools for openstack monitoring for telco/NFV usecases | |
| 18:32:16 | sean-k-mooney | https://www.youtube.com/watch?v=-82UShFiBBM | |
| 18:32:26 | spatel | sean-k-mooney: very interesting... also i had question about SR-IOV VF nic stats, it doesn't tell you how much data flowing through your VF | |
| 18:32:57 | sean-k-mooney | spatel: that depend on the nic | |
| 18:33:18 | spatel | I have Qlogic so i doubt it has that feature | |
| 18:33:23 | sean-k-mooney | so most nics dont have enough hardware counters to gatar stats on all the VFs from the host side | |
| 18:34:00 | sean-k-mooney | spatel: if you are able to run collectd in the guest however it can monitor the kernel stats and give you that info | |
| 18:34:50 | spatel | Yes! that is what i am doing currently! guest base snmp monitoring | |
| 18:34:59 | sean-k-mooney | spatel: one other tool that you might also find userful in this space is skydive. http://skydive.network/ | |
| 18:35:35 | spatel | hmm! looking cool | |
| 18:35:51 | spatel | we are using observium and Cisco DCNM | |
| 18:36:49 | spatel | hmm! they have openstack support too | |
| 18:37:11 | sean-k-mooney | skydive was developed in the last 2 years ago specificly for cloud and conterised enviornments but it began to mature just after i deployed my last cloud so i have not used it myself | |
| 18:38:29 | spatel | Can i use for Cisco switches and router? | |
| 18:38:52 | spatel | look like they use agent do i doubt | |
| 18:38:58 | sean-k-mooney | i belive so. it uses multiple protocols to do its monitoring | |
| 18:40:05 | sean-k-mooney | if thet cisco switch support sflow i belive it would work | |
| 18:40:39 | spatel | We have all Cisco nexus switches and they do sflow :) i am going to explore it now | |
| 18:41:41 | spatel | Do you guys using collectd to monitor your compute nodes? | |
| 18:41:46 | mriedem_afk | awaugama: as far as i know if you set the cpu_allocation_ratio in nova.conf on the nova-compute service, it should mirror that to the new OR existing resource provider in placement | |
| 18:41:57 | mriedem | if it's nit mirroring those updates i think that's a bug | |
| 18:42:16 | mriedem | SteelyDan: on that bug, tbc, i think they got confused about "scheduling" | |
| 18:42:32 | awaugama | mriedem: that's what I thought would happen. sean-k-mooney you think it won't update existing ones? | |
| 18:42:32 | mriedem | it's not scheduling anything, they just power on an instance on a stopped compute and the rpc cast is sent to the void | |
| 18:43:09 | sean-k-mooney | awaugama: im not sure. | |
| 18:43:22 | mriedem | cfriesen: yes the bug is saying they power on the instance, the api changes the task_state, but b/c the compute is down, we never finish the task and it's "stuck" | |
| 18:43:46 | sean-k-mooney | awaugama: it will do one of two things either it will overright it allways with the config value or it will only use the config value on creattion fo the rp | |
| 18:43:47 | mriedem | fixing that would mean needing to check the compute service status for every instance action... | |
| 18:44:19 | sean-k-mooney | awaugama: both cases are "wronge" depending on who you ask hench the sepc for initall allcoation ratios https://review.openstack.org/#/c/552105/ | |
| 18:44:56 | mriedem | on every periodic, we update the resource class inventory allocation ratio based on config that we send to placement https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/compute/resource_tracker.py#L108 | |
| 18:45:09 | mriedem | https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/compute/resource_tracker.py#L952 | |
| 18:45:24 | mriedem | the question is if prov_tree.update_inventory(nodename, inv_data) has a bug thinking that nothing changed | |
| 18:45:25 | mriedem | b/c it's a cache | |
| 18:45:34 | mriedem | and the reportclient itself has a cache of the provider tree | |
| 18:46:04 | mriedem | https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1501 | |
| 18:46:08 | mriedem | "The specified ProviderTree is compared against the local cache. Any changes are flushed back to the placement service. " | |
| 18:47:37 | awaugama | makes sense | |
| 18:48:23 | mriedem | the bug is probably here https://github.com/openstack/nova/blob/5b815eec4c5fc8c19863aa38b1d1920705b17bfa/nova/scheduler/client/report.py#L1575-L1576 | |
| 18:48:40 | mriedem | if we have a single compute node resource provider and that doesn't change, both of those sets will be empty | |
| 18:48:43 | cfriesen | mriedem: as I said in the bug, that won't actually fix it, just make the race window smaller. | |
| 18:48:47 | mriedem | and the for loops below won't flush any changes to placement | |
| 18:48:54 | mriedem | cfriesen: isure | |
| 18:48:56 | mriedem | *sure | |
| 18:48:59 | mriedem | b/c of the service group api | |