| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-07 | |||
| 13:42:08 | artom | lyarwood, there's the one below it as well ;) https://review.openstack.org/#/c/587248/1 | |
| 13:43:08 | artom | Cheers! | |
| 13:43:21 | lyarwood | artom: np LGTM | |
| 15:06:17 | tomtom001 | lyarwood: hello I was wondering you could say how the encryption on nova works with nova libvirt and ceph... we've noticed really slow performance when using luks encryption... I've turned off xts,and only use aes but it doesn't seem to change anything, what specifically needs to be in place to get a fast encryption for disks? | |
| 15:07:33 | lyarwood | tomtom001: Which version are you using and slow in what sense? Read / Writes / time taken to attach volumes etc? | |
| 15:12:26 | sean-k-mooney | tomtom001: does the host system have aes-ni processor instruction or is this older hardware. there are a log of vairibles that can come into play here | |
| 15:18:01 | tomtom001 | So they are DellFx2 servers, so recent hardware. We are using Queens as installed from Openstack-ansible and using Ceph luminous. i/o performance is 1/20th the speed of an unencrypted volume. | |
| 15:18:33 | tomtom001 | volume creation and attach time seems normal. | |
| 15:22:20 | sean-k-mooney | tomtom001: that would seam to imply that the perfromace pelenty is coming form the encryption overhead which should be low if its hardware offloaded but would be non trivial if aes-ni was not available | |
| 15:24:12 | cdent | efried: do open questions remain on your refresh stack or is it fairly straightforward from here? I'm trying to optimize my time before leaving for berlin (tomorrow) | |
| 15:25:32 | efried | cdent: I'm kind of a mess, need to go back through the emails and the series. But I *think* the only remaining questions are whether to make the timer zero by default for new installs; and/or make it the *only* thing - i.e. remove the option entirely and only refresh the cache when something happens. | |
| 15:26:26 | efried | cdent: and I've convinced the ptb that, if either of those things comes to pass, they should be done at least separately, and probably later once these have had some field time with CERN/Vexx | |
| 15:26:48 | cdent | makes sense | |
| 15:30:24 | sean-k-mooney | efried: for what its worth im fine with waiting to change the default but in T we proably shoudl change the default to 0 | |
| 15:31:05 | efried | sean-k-mooney: I don't even mind doing it earlier than that if we can demonstrate some traction in a prod env or two. | |
| 15:31:17 | sean-k-mooney | that said cern/vexxhost wont deploy stein right away so it would be late in T before we got any feedback | |
| 15:32:03 | efried | sean-k-mooney: I may be able to backport this code. | |
| 15:32:12 | efried | sean-k-mooney: Not to merge necessarily, but at least so they can pull it into their env. | |
| 15:32:18 | sean-k-mooney | right the catalyst for changing the default would production use showing it works for them | |
| 15:39:20 | tomtom001 | sean-k-mooney - right so hardware offloaded would just be hardware encryption, we are looking at it from the cinder luks ceph side. | |
| 15:40:10 | sean-k-mooney | tomtom001: well hardware offload would acclerate the luks part | |
| 15:40:14 | mriedem | bauzas: do you care about this at all? you added the original lazy-load scheduler report client stuff https://review.openstack.org/#/c/615641/ | |
| 15:40:32 | sean-k-mooney | cinder and ceph should not chagne if you enable encryption | |
| 15:40:45 | bauzas | mriedem: for sure, I'm interested but I have a long customer issue | |
| 15:41:55 | efried | mriedem: bauzas and I discussed it yesterday. I'm not sure whether I succeeded in convincing him that we weren't actually deferring importing the modules until method call. But he also admitted the only reason we were doing it at all was to avoid a circular import, which is clearly not an issue anymore. | |
| 15:43:31 | efried | btw, at some point when I get breathing room, I'm going to rip out the SchedulerClient entirely. At the top of my series, it only has methods to the query client (all report client passthrough methods are gone). And I want to suck up to jaypipes some more anyway. | |
| 15:43:51 | mriedem | i think that's the nit todo i just left on that change | |
| 15:43:54 | mriedem | eventually remove the proxy | |
| 15:44:22 | cdent | I'm glad we seem to have agreement on that idea | |
| 15:44:31 | efried | yeah, for sure. I've looked into it before and it was going to be nontrivial, which is why I haven't done it in this series. | |
| 15:44:42 | efried | but this series will make it a little bit easier. | |
| 15:44:46 | bauzas | efried: mriedem: +Wd | |
| 15:45:02 | mriedem | well looky there | |
| 15:45:02 | efried | bauzas: Thank you sir | |
| 15:45:18 | bauzas | sorry for not having a lot of time for upstream :( | |
| 15:56:42 | tomtom001 | sean-k-mooney Sorry i'm not understanding, from what I think is supposed to happen, is that qemu is supposed to handle luks encryption on the fly on a disk correct? | |
| 15:59:03 | sean-k-mooney | tomtom001: correct but that was only added to openstack in pike see https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/libvirt-qemu-native-luks.html | |
| 15:59:30 | tomtom001 | ok, so I'm running queens so that should be there. | |
| 15:59:42 | tomtom001 | or no it's not implemented yet? | |
| 16:00:02 | sean-k-mooney | tomtom001: no it should be there | |
| 16:01:02 | tomtom001 | so on one cpu host i've removed the xts module to ensure only the aes module is loaded, is there anything else I can check in qemu or libvirt or nova to ensure that encryption is operating normally? | |
| 16:01:45 | tomtom001 | qemu 2.11 is installed | |
| 16:02:14 | tomtom001 | libvirt 4.0.0 and nova compute from stale/queens | |
| 16:02:19 | tomtom001 | *stable | |
| 16:02:52 | sean-k-mooney | there versions of the software you are running should all support it | |
| 16:03:42 | tomtom001 | are there settings I need to check for this? | |
| 16:08:21 | sean-k-mooney | not that i a aware of but lyarwood might be able to provide more info | |
| 16:08:56 | sean-k-mooney | you will see the encyption nebaled in the xml/qemu command line but i dont know of anything you need to do to improve performance | |
| 16:19:36 | tomtom001 | sean-k-mooney thank you, lyarwood, you have anything I can check or something you'd like to look at to tell you? | |
| 16:42:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send rp_uuid in the port binding https://review.openstack.org/569459 | |
| 16:42:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 16:42:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | |
| 16:42:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations to neturonv2 api https://review.openstack.org/616240 | |
| 16:50:21 | efried | mriedem: Update on the nova-next business... | |
| 16:50:41 | efried | https://review.openstack.org/#/c/616033/ see latest comment | |
| 16:50:58 | efried | tldr I don't think the tweak is working, or I'm a dolt and looking in the wrong place. | |
| 16:52:34 | efried | However, having looked at the placement logs for the nova-next runs in the whole series, I see a major drop at the "Reduce calls to placement from _ensure" patch, which is what I would expect; and actually, I wouldn't expect any of the CI jobs to see a significant drop from setting refresh=0 because they're getting refreshed all the time because they're going crazy doing real stuffs, so the refreshing is happening anyway and the | |
| 16:53:25 | efried | The refresh=0 thing is really only going to see a big benefit in steady state, and (afaik) we don't have any CI jobs that just... sit there. | |
| 16:54:57 | mriedem | i've had issues with using post-config and NOVA_CPU_CONF in the nova-live-migration job... | |
| 16:56:12 | efried | mriedem: I found a cells job that was using something similar and it seemed to be working (was able to verify the conf change was made) | |
| 16:56:54 | efried | mriedem: http://logs.openstack.org/33/616033/1/check/nova-cells-v1/fb84e57/logs/etc/nova/nova-cells.conf.txt.gz is getting vif_plugging_timeout = 0 from playbooks/legacy/nova-cells-v1/run.yaml | |
| 16:58:08 | mriedem | devstack is processing the post-config | |
| 16:58:08 | mriedem | http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_24_44_044 | |
| 17:00:19 | efried | so... wtf? | |
| 17:00:57 | mriedem | it looks like that post-config happens first, | |
| 17:01:03 | mriedem | and then other stuff is writing to nova-cpu.conf | |
| 17:01:13 | mriedem | so my guess is that devstack is overwriting it... | |
| 17:01:51 | efried | ah http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_26_29_702 | |
| 17:01:55 | efried | we blow it away later :( | |
| 17:02:09 | efried | what's the magic env var for the nova.conf? | |
| 17:02:15 | mriedem | yup http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_26_29_702 | |
| 17:02:24 | mriedem | $NOVA_CONF | |
| 17:03:37 | mriedem | definitely a bug, but just do $NOVA_CONF for now and comment in the playbook | |
| 17:03:44 | efried | ack | |
| 17:04:04 | mriedem | i can report a devstack back | |
| 17:04:14 | efried | bug? | |
| 17:04:16 | efried | cool, thanks. | |
| 17:06:20 | mriedem | https://bugs.launchpad.net/devstack/+bug/1802143 | |
| 17:06:20 | openstack | Launchpad bug 1802143 in devstack "Changes from [[post-config|$NOVA_CPU_CONF]] are overwritten in "superconductor" mode" [Undecided,New] | |
| 17:06:21 | mriedem | *yeah bug | |
| 17:07:35 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Turn off rp association refresh in nova-next https://review.openstack.org/616033 | |
| 17:07:41 | efried | mriedem: done ^ | |
| 17:08:07 | efried | mriedem: but as I said, I don't actually expect to see a significant difference, because active state. | |
| 17:08:28 | efried | though making sure it actually works and doesn't blow up the world will be good :) | |
| 17:10:43 | mriedem | hey now that the dems took over the house the world is going to end anyway | |
| 17:11:30 | mriedem | heh, *braces | |
| 17:11:50 | mriedem | sorry, trump is doing a news conference talking about the lack of republican losers that didn't accept his warm embrace | |
| 17:25:11 | tssurya | sean-k-mooney: I had a question, would you know the working of virsh capabilities (ca you point me to the code of how this xml is generated)? | |
| 17:25:22 | tssurya | couldn't find the exact source code | |
| 17:27:26 | mriedem | kashyap: ^ | |
| 17:29:27 | dansmith | tssurya: it comes from libvirt, obviously | |
| 17:29:36 | dansmith | tssurya: there is a lot of documentation about what they all mean | |
| 17:29:49 | tssurya | dansmith: yea I tried to go into the libvirt code,, but in vain | |
| 17:29:55 | dansmith | tssurya: https://libvirt.org/formatcaps.html | |
| 17:30:06 | dansmith | tssurya: you want to know how it's determining one thing or another, or just what everything means? | |
| 17:30:09 | tssurya | I already saw that link | |
| 17:30:35 | dansmith | example of what you're trying to figure out? | |
| 17:30:49 | tssurya | I want to know how this Core(s) per socket in the topology is calculated | |
| 17:31:04 | tssurya | or the code in libvirt that gets this part | |
| 17:31:58 | dansmith | it should all be straight out of /sys I would think | |