| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-03 | |||
| 09:40:29 | openstackgerrit | Michael Still proposed openstack/nova master: Move simple execute call to processutils. https://review.openstack.org/621527 | |
| 09:40:30 | openstackgerrit | Michael Still proposed openstack/nova master: Move setting mac addresses for network devices to privsep. https://review.openstack.org/621529 | |
| 09:45:49 | izza_ | hi..can anyone help me here about error that we've encountered upon testing vgpu in devstack using tensorflow | |
| 09:46:16 | izza_ | encountering below error: The TensorFlow library was compiled to use SSE4.1 instructions, but these aren't available on your machine. | |
| 10:05:13 | bauzas | izza_: good morning | |
| 10:05:34 | bauzas | izza_: when you say "testing vgpus", you mean using https://docs.openstack.org/nova/latest/admin/virtual-gpu.html ? | |
| 10:06:05 | bauzas | izza_: I guess you use the libvirt driver with KVM ? | |
| 10:08:03 | izza_ | able to detect the vgpu...the thing is when we tried to deploy tensorflow in devstack, we're encountering such error.. yes, it is already set as KVM | |
| 10:08:26 | bauzas | izza_: I guess you probably don't have the right CPU model | |
| 10:11:02 | bauzas | izza_: which CPU mode do you use for all your computes ? | |
| 10:12:04 | bauzas | context is https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.cpu_mode | |
| 10:13:00 | bauzas | AFAIR, using "None" for your CPU mode in Nova makes you get a very old CPU model | |
| 10:13:08 | bauzas | for migration reasons | |
| 10:23:04 | bauzas | izza_: if you wanna get more on CPU models, see https://www.berrange.com/posts/2018/06/29/cpu-model-configuration-for-qemu-kvm-on-x86-hosts/ | |
| 10:32:31 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: DNM: troubleshoot cells v1 tests on gate https://review.openstack.org/621538 | |
| 10:38:23 | sean-k-mooney | bauzas: using none for cpu mode should result in host model which will be the most recent cpu model that matches your host | |
| 10:40:30 | izza_ | so it should be none? | |
| 10:41:34 | sean-k-mooney | izza_: you should always set it explctly to a supported value or not at all. | |
| 10:41:51 | sean-k-mooney | none will indrectly result in host-model | |
| 10:43:44 | sean-k-mooney | oh actully maybe bauzas is correct https://github.com/openstack/nova/blob/master/nova/conf/libvirt.py#L490-L493 | |
| 10:46:15 | sean-k-mooney | it looks like we treat the string "none" differently form that value None | |
| 10:46:18 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3796-L3815 | |
| 10:47:48 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: Add workaround to remove instance dir during cleanup with rbd https://review.openstack.org/618478 | |
| 10:48:07 | sean-k-mooney | izza_: try setting it to host-model explcitly | |
| 10:49:05 | sean-k-mooney | izza_: bauzas may already have covered this but you might also need to use a different vGPU type to enable tensorflow to work with gpu accleration. | |
| 10:49:17 | sean-k-mooney | only some models support opencl/cuda | |
| 10:50:48 | bauzas | sean-k-mooney: izza_: the conf option help says that cpu_mode defaults to host-model if KVM | |
| 10:51:07 | bauzas | so you should get the SSE4 instructions if your CPU is recent | |
| 10:51:25 | bauzas | but worth checking the guest XML | |
| 10:51:49 | izza_ | im checking it now | |
| 10:52:23 | bauzas | izza_: please tell us the config options values for [libvirt]/cpu_mode and [libvirt]/cpu_model | |
| 10:52:32 | bauzas | izza_: besides the virt type | |
| 10:52:36 | sean-k-mooney | sse4 is pretty old. i would be surprised if the host did not supported it | |
| 10:52:41 | bauzas | I agree | |
| 10:53:06 | bauzas | the only thing I could think of would be cpu_mode explicitely set to None | |
| 10:53:48 | izza_ | [libvirt] | |
| 10:53:50 | bauzas | which would make the guest using a very old CPU model that doesn't support SSE1 | |
| 10:53:53 | bauzas | SSE4* | |
| 10:54:04 | izza_ | live_migration_uri = qemu+ssh://stack@%s/system | |
| 10:54:10 | izza_ | cpu_mode = host-passthrough | |
| 10:54:16 | izza_ | cpu_model_extra_flags = sse4_1 | |
| 10:54:22 | izza_ | virt_type = kvm | |
| 10:54:47 | sean-k-mooney | izza_: host-passthrough will give you the same feature flags as the host. you dod not need the extra flags | |
| 10:55:14 | izza_ | noted on this..we will remove it.. | |
| 10:55:23 | sean-k-mooney | izza_: you have sse in the hosts /proc/cpuinfo right? | |
| 10:55:39 | izza_ | but we tried to remove it and set is as passthrough without flags | |
| 10:55:48 | izza_ | but still unable to get the other flags | |
| 10:56:07 | sean-k-mooney | every xeon since nehelem has had sse 4.2+ | |
| 10:56:26 | izza_ | yes, sse4.1 is available in /proc/cpuinfo | |
| 10:56:52 | sean-k-mooney | izza_: and your setting cpu_mode = host-passthrough in the /etc/nova/nova-cpu.conf | |
| 10:58:01 | izza_ | it is set as "none" in /etc/nova/nova-cpu.conf | |
| 10:58:12 | sean-k-mooney | izza_: that the issue | |
| 10:58:30 | izza_ | what should i do? | |
| 10:58:36 | sean-k-mooney | /etc/nova/nova-cpu.conf is used by the compute agent /etc/nova/nova.conf is used by the schduer/conductor | |
| 10:59:07 | izza_ | ok will try to change it | |
| 10:59:57 | izza_ | we will advise you for the result | |
| 11:00:08 | sean-k-mooney | [[post-config|$NOVA_CPU_CONF]] | |
| 11:00:11 | sean-k-mooney | [libvirt] | |
| 11:00:13 | sean-k-mooney | cpu_mode = host-passthrough | |
| 11:00:15 | sean-k-mooney | virt_type = kvm | |
| 11:00:19 | sean-k-mooney | that is how to set it in your local.conf by the way | |
| 11:00:51 | izza_ | ok noted on that | |
| 11:00:54 | izza_ | we'll try | |
| 11:01:00 | sean-k-mooney | you can change it locally and then restart the nova services with sudo systemctl restart devstack@n-* | |
| 11:01:20 | bauzas | what sean-k-mooney said | |
| 11:01:30 | sean-k-mooney | * by locally i mean in /etc/nova/nova-cpu.conf | |
| 11:01:34 | bauzas | izza_: your n-cpu service uses the wrong file | |
| 11:01:36 | izza_ | thanks guys | |
| 11:04:40 | sean-k-mooney | bauzas: well a different file we use /etc/nova/nova-cpu.conf instead of /etc/nova/nova.conf intentionally | |
| 11:04:49 | izza_ | got it...already capture the cpu details after changing the said file | |
| 11:05:11 | bauzas | 'wrong' is maybe not the right word | |
| 11:05:28 | bauzas | what I mean is that n-cpu uses another conf file by default | |
| 11:05:31 | bauzas | but you can change this | |
| 11:05:45 | sean-k-mooney | yep | |
| 11:05:56 | bauzas | just don't use systemctl in this case | |
| 11:06:11 | sean-k-mooney | ya not? | |
| 11:06:25 | sean-k-mooney | *why not? | |
| 11:06:50 | bauzas | just becauses it precludes the config file name | |
| 11:07:08 | bauzas | sometimes I prefer stopping the service and starting it by hand | |
| 11:07:21 | bauzas | maybe because I'm an old guy | |
| 11:07:40 | sean-k-mooney | systemctl is the correct way to do that in devstack for like 2-3 releases | |
| 11:08:12 | sean-k-mooney | infact that devstack docs use n-cpu as the example https://docs.openstack.org/devstack/latest/systemd.html#manipulating-units | |
| 11:14:37 | jangutter | bauzas: I'm still sometimes using 'ifconfig', where I should be using 'ip', and really, really, should be using 'nmcli'. | |
| 11:14:50 | bauzas | sean-k-mooney: you misunderstood me | |
| 11:15:19 | bauzas | sean-k-mooney: I'm not saying to not use systemctl (although there is still a devstack option for it :p ) | |
| 11:15:56 | bauzas | sean-k-mooney: I'm saying that, most of the times, stopping the incriminated service and restarting it by hand is preferrable | |
| 11:16:07 | sean-k-mooney | bauzas: noop the screen option was removed in rocky | |
| 11:16:15 | bauzas | gtk | |
| 11:16:44 | sean-k-mooney | oh you mean sudo systemctl stop devstack@n-cpu.service the nova-coumpute ... | |
| 11:17:29 | sean-k-mooney | runing the service directly can be usefaul for debuging | |
| 11:20:06 | jangutter | One thing I like is that I can go 'systemctl status PID' and get all the startup (and a bit of logs) for a service. | |
| 11:20:21 | jangutter | Is there something similar for docker/containers? | |
| 11:20:55 | sean-k-mooney | jangutter: docker logs <container name or id> | |
| 11:21:47 | sean-k-mooney | or depeneding on your container it may log to the hosts journal so journalctl -u <service in contienr> | |
| 11:30:41 | ShilpaSD | hi: All, please help me to understand, how this section, table format of notificaiotns, is added here 'https://docs.openstack.org/nova/rocky/reference/notifications.html#existing-versioned-notifications' | |
| 11:31:48 | izza_ | hi | |
| 11:32:28 | izza_ | encounetered error upon execution of tensorflow session | |
| 11:32:29 | izza_ | >>> sess = tf.Session() | |
| 11:33:01 | izza_ | 2018-12-03 11:30:11.110534: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA | |
| 11:33:08 | izza_ | raceback (most recent call last): | |