| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-15 | |||
| 12:18:56 | kashyap | Although nothing is stopping me from practising it myself. | |
| 12:19:50 | cdent | good luck | |
| 12:30:51 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: docs: Secure live migration with QEMU-native TLS https://review.openstack.org/629627 | |
| 12:31:39 | brinzhang | Kevin_Zheng: Thanks for your help, my devstack | |
| 12:31:51 | brinzhang | is ok now :) | |
| 12:36:09 | Kevin_Zheng | yw | |
| 12:50:09 | kashyap | stephenfin: Squashed all the additional into the main doc: https://review.openstack.org/629627 | |
| 12:50:33 | kashyap | Rendered version: | |
| 12:50:34 | kashyap | http://logs.openstack.org/27/629627/6/check/openstack-tox-docs/39c0a0b/html/admin/secure-live-migration-with-qemu-native-tls.html | |
| 12:50:42 | kashyap | (All the damn hyperlinks work. :D) | |
| 12:51:29 | dpawlik | kashyap: Hi, does Openstack documentation have any information, which cpu_model_extra_flags are "recommended" to enable? | |
| 12:51:51 | kashyap | dpawlik: Hi, I wrote some docuemntation for it, and linked to it as well | |
| 12:51:53 | kashyap | Let me find | |
| 12:52:08 | dpawlik | kashyap, thx | |
| 12:53:26 | kashyap | dpawlik: A couple of URLs: | |
| 12:53:37 | kashyap | (1) See slides 35 (76 of 80) here: https://kashyapc.fedorapeople.org/Effective-Virtual-CPU-Configuration-in-Nova-Berlin2018.pdf | |
| 12:53:41 | kashyap | (2) https://www.berrange.com/posts/2018/06/29/cpu-model-configuration-for-qemu-kvm-on-x86-hosts/ | |
| 12:53:50 | kashyap | (3) https://www.qemu.org/2018/02/14/qemu-2-11-1-and-spectre-update/ | |
| 12:54:16 | dpawlik | kashyap: thank you :) | |
| 12:54:24 | kashyap | (4) `grep` for "cpu_model_extra_flags" here: https://docs.openstack.org/nova/rocky/configuration/config.html | |
| 12:54:29 | kashyap | That's it. | |
| 12:56:14 | dpawlik | kashyap: I guess for now I will test just those PCID, VMX, pdpe1gb | |
| 12:56:40 | dpawlik | I was thinking that there are more of them, that are recommended | |
| 12:57:22 | kashyap | dpawlik: For Intel hosts, don't forget: "spec-ctrl", "ssbd" as well. | |
| 12:57:48 | amorin | yes | |
| 12:58:42 | dpawlik | kashyap: it will be nice to have "default suggested" features that should be enabled | |
| 12:59:02 | dpawlik | right now I just add PCID and pdpe1gb | |
| 12:59:19 | dpawlik | but about "spec-ctrl", "ssbd" i didn't know that they should be also enabled | |
| 12:59:22 | kashyap | dpawlik: That "suggestion" is not as easy as it sounds -- because that requires several checks to be done. | |
| 12:59:45 | kashyap | E.g. what host microcode you have, what host/guest kernel, what version of libvirt and QEMU | |
| 13:00:00 | kashyap | And _then_, some magic code among libvirt/QEMU should do what you have and _then_ suggest something. | |
| 13:00:29 | kashyap | We had some discussion about this with the KVM/QEMU/libvirt folks at one of the recent confereces. More details to emerge... | |
| 13:00:29 | dpawlik | wow | |
| 13:01:01 | kashyap | Until dpawlik Until then, I only have: "x86: Updating to patched vCPU models" section | |
| 13:01:04 | dpawlik | it seems like table with "which features are available in which kernel version and qemu/libvirt" | |
| 13:01:36 | kashyap | But that has to wait... | |
| 13:02:13 | dpawlik | kashyap: ack. Thanks a lot | |
| 13:02:45 | kashyap | NP. | |
| 13:26:38 | ShilpaSD | mriedem: Hi, replied to your comment and uploaded patch https://review.openstack.org/#/c/612626/2, Please give your feedback on the latest patch. | |
| 13:48:01 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: Clean up versioned object backlevelling code https://review.openstack.org/630976 | |
| 13:51:34 | openstackgerrit | Maria Luisa Arches proposed openstack/nova master: Update network info during confirmed cold migration https://review.openstack.org/630978 | |
| 14:03:07 | alex_xu | will NUMACell.memory include the size of hugepages? | |
| 14:08:09 | stephenfin | alex_xu: No, it's just a basic values, i.e. what you'd get from 'cat /sys/devices/system/node/nodeN/meminfo | grep MemTotal' | |
| 14:08:13 | stephenfin | *value | |
| 14:08:22 | sean-k-mooney | actully i think it will | |
| 14:08:35 | sean-k-mooney | the memory field should be the total memory on that numa node | |
| 14:08:56 | sean-k-mooney | mem total include hugepages | |
| 14:09:43 | stephenfin | Oh, I read that directly (as in would it have information about hugepages, which is actually reported via 'NUMACell.mempages') | |
| 14:09:48 | alex_xu | I doesn't configure hugepage on my pc, so can't verify that... | |
| 14:10:10 | sean-k-mooney | one sec ill post a past bin of the output | |
| 14:10:14 | stephenfin | alex_xu: I can give you info from one of my machine, if you'd like | |
| 14:10:24 | stephenfin | Or, better yet, let sean-k-mooney do it :) | |
| 14:10:40 | alex_xu | hah, I just want an answer :) | |
| 14:10:41 | sean-k-mooney | http://paste.openstack.org/show/742643/ | |
| 14:10:58 | sean-k-mooney | so i have 64GB of ram on that numa node | |
| 14:11:06 | sean-k-mooney | Node 0 MemTotal: 65998248 kB | |
| 14:11:16 | alex_xu | just want to ensure whether we have this problem https://review.openstack.org/#/c/629281/1/nova/virt/hardware.py@1050 or not | |
| 14:11:54 | sean-k-mooney | i have hugepages allocated too "HugePages_Total: 27136" these are 2MB huge pages | |
| 14:12:05 | stephenfin | alex_xu: I saw the comment, but I'm not sure I understand the actual problem. What do you think could go wrong? | |
| 14:12:28 | stephenfin | Just saw you replied. Oops | |
| 14:13:11 | sean-k-mooney | instance_cell.memory is the amoung of guest ram | |
| 14:13:18 | stephenfin | alex_xu: So I think you're saying that https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L982 could be wrong? | |
| 14:13:20 | sean-k-mooney | on that virtual numa node | |
| 14:13:40 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: A few miscellaneous items related to "native TLS" https://review.openstack.org/630980 | |
| 14:13:52 | stephenfin | alex_xu: Or rather that 'host_cell.avail_memory' includes memory that's not actually available because it's used for hugepages | |
| 14:13:58 | sean-k-mooney | this will be compared to the amount of ram available for the page size selected | |
| 14:16:05 | stephenfin | alex_xu: Do I understand correctly? If so, I guess this would be easily checked. On a 64GB host, I create 34 1GB huge pages then try to boot a 32GB pinned instance. If it doesn't fail, we're likely overcommitting | |
| 14:16:26 | stephenfin | though if that is the case, it's a different bug again that I think we should probably handle in a follow-up | |
| 14:16:39 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: A few miscellaneous items related to "native TLS" https://review.openstack.org/630980 | |
| 14:17:20 | alex_xu | stephenfin: yes, that is the case I talk about | |
| 14:17:22 | sean-k-mooney | host_cell.available is compareing free memory not total but the page size check is done by https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1021-L1056 which is run after | |
| 14:17:47 | kashyap | Damn it, one last respin | |
| 14:17:52 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: A few miscellaneous items related to "native TLS" https://review.openstack.org/630980 | |
| 14:18:03 | kashyap | (Sorry for the spam, folks.) | |
| 14:18:22 | stephenfin | sean-k-mooney: Right, but I'm changing that here https://review.openstack.org/#/c/629281/1/nova/virt/hardware.py@1050 | |
| 14:18:31 | stephenfin | sean-k-mooney: to allow overcommit if using the smallest pagesize | |
| 14:19:15 | sean-k-mooney | yes i think it si still correct i like the wrong peice of code one sec | |
| 14:19:36 | stephenfin | sean-k-mooney: Ah, yes. https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L982 is available memory, as you say, not total memory | |
| 14:21:51 | stephenfin | but it's based on total memory (which could be include memory used for hugepages) minus memory consumed by instances | |
| 14:22:00 | stephenfin | ugh, I need to check this locally again | |
| 14:22:50 | sean-k-mooney | so you are changing _pack_instance_onto_cores to allow oversubsciption you are not modifying _numa_fit_instance_cell | |
| 14:24:22 | sean-k-mooney | _pack_instance_onto_cores is called from _numa_fit_instance_cell_with_pinning which is called in _numa_fit_instance_cell after the check on line 1046 | |
| 14:24:58 | sean-k-mooney | sorry no ignore that | |
| 14:25:21 | sean-k-mooney | gerrit is mseeing with me you are modifying _numa_fit_instance_cell | |
| 14:26:08 | stephenfin | sean-k-mooney: I'm modifying _numa_cell_supports_pagesize_request which is called by _numa_fit_instance_cell | |
| 14:27:31 | sean-k-mooney | stephenfin: in either case i think the code you have is correct | |
| 14:29:33 | sean-k-mooney | can_fit_pagesize will only consider teh total or availbe memory for the specified page size which will always be less or equal to the total or available ram for the numa node | |
| 14:30:33 | alex_xu | emm...the problem is overcommit, not less or equal to the total or avail ram | |
| 14:31:20 | sean-k-mooney | this code is only ment to ensure the vm does not request more ram then is available for that pagesize on that numa node | |
| 14:31:53 | sean-k-mooney | over commit from multiple vms is intended to be allowed for the smallest page size as we allow ram allocation ratios > 1 | |
| 14:34:23 | sean-k-mooney | alex_xu: i take it that your are conserned that https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1085-L1106 will not be correct with this change | |
| 14:35:11 | alex_xu | yes, but that patch allow the overcommit for each numa cell, but in the later, we found it is cpu pinned instance, we only ensure it isn't overcommit by the amount memory for all the numa cells | |
| 14:35:57 | alex_xu | sean-k-mooney: no, I concern this branch https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1076-L1083... | |
| 14:36:26 | sean-k-mooney | ah well i just spotted that stpehn has not allowed oversubscition here also which shoudl be allowed https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L982-L990 | |
| 14:37:05 | sean-k-mooney | or rather this should not be checking host_cell.avail_memory it should be checking host_cell.memory | |
| 14:37:22 | sean-k-mooney | or better it should be doing a page size aware check | |
| 14:37:26 | stephenfin | I don't think I changed that | |
| 14:37:41 | sean-k-mooney | stephenfin: you didnt that is my point | |
| 14:38:11 | sean-k-mooney | if you dont chenge that pinned instance can still not have over subsribed memory which is part of the original bug | |
| 14:38:34 | stephenfin | sean-k-mooney: Yes, that's what alex_xu is saying, I think :) | |