Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-07
13:43:21 lyarwood artom: np LGTM
15:06:17 tomtom001 lyarwood: hello I was wondering you could say how the encryption on nova works with nova libvirt and ceph... we've noticed really slow performance when using luks encryption... I've turned off xts,and only use aes but it doesn't seem to change anything, what specifically needs to be in place to get a fast encryption for disks?
15:07:33 lyarwood tomtom001: Which version are you using and slow in what sense? Read / Writes / time taken to attach volumes etc?
15:12:26 sean-k-mooney tomtom001: does the host system have aes-ni processor instruction or is this older hardware. there are a log of vairibles that can come into play here
15:18:01 tomtom001 So they are DellFx2 servers, so recent hardware. We are using Queens as installed from Openstack-ansible and using Ceph luminous. i/o performance is 1/20th the speed of an unencrypted volume.
15:18:33 tomtom001 volume creation and attach time seems normal.
15:22:20 sean-k-mooney tomtom001: that would seam to imply that the perfromace pelenty is coming form the encryption overhead which should be low if its hardware offloaded but would be non trivial if aes-ni was not available
15:24:12 cdent efried: do open questions remain on your refresh stack or is it fairly straightforward from here? I'm trying to optimize my time before leaving for berlin (tomorrow)
15:25:32 efried cdent: I'm kind of a mess, need to go back through the emails and the series. But I *think* the only remaining questions are whether to make the timer zero by default for new installs; and/or make it the *only* thing - i.e. remove the option entirely and only refresh the cache when something happens.
15:26:26 efried cdent: and I've convinced the ptb that, if either of those things comes to pass, they should be done at least separately, and probably later once these have had some field time with CERN/Vexx
15:26:48 cdent makes sense
15:30:24 sean-k-mooney efried: for what its worth im fine with waiting to change the default but in T we proably shoudl change the default to 0
15:31:05 efried sean-k-mooney: I don't even mind doing it earlier than that if we can demonstrate some traction in a prod env or two.
15:31:17 sean-k-mooney that said cern/vexxhost wont deploy stein right away so it would be late in T before we got any feedback
15:32:03 efried sean-k-mooney: I may be able to backport this code.
15:32:12 efried sean-k-mooney: Not to merge necessarily, but at least so they can pull it into their env.
15:32:18 sean-k-mooney right the catalyst for changing the default would production use showing it works for them
15:39:20 tomtom001 sean-k-mooney - right so hardware offloaded would just be hardware encryption, we are looking at it from the cinder luks ceph side.
15:40:10 sean-k-mooney tomtom001: well hardware offload would acclerate the luks part
15:40:14 mriedem bauzas: do you care about this at all? you added the original lazy-load scheduler report client stuff https://review.openstack.org/#/c/615641/
15:40:32 sean-k-mooney cinder and ceph should not chagne if you enable encryption
15:40:45 bauzas mriedem: for sure, I'm interested but I have a long customer issue
15:41:55 efried mriedem: bauzas and I discussed it yesterday. I'm not sure whether I succeeded in convincing him that we weren't actually deferring importing the modules until method call. But he also admitted the only reason we were doing it at all was to avoid a circular import, which is clearly not an issue anymore.
15:43:31 efried btw, at some point when I get breathing room, I'm going to rip out the SchedulerClient entirely. At the top of my series, it only has methods to the query client (all report client passthrough methods are gone). And I want to suck up to jaypipes some more anyway.
15:43:51 mriedem i think that's the nit todo i just left on that change
15:43:54 mriedem eventually remove the proxy
15:44:22 cdent I'm glad we seem to have agreement on that idea
15:44:31 efried yeah, for sure. I've looked into it before and it was going to be nontrivial, which is why I haven't done it in this series.
15:44:42 efried but this series will make it a little bit easier.
15:44:46 bauzas efried: mriedem: +Wd
15:45:02 efried bauzas: Thank you sir
15:45:02 mriedem well looky there
15:45:18 bauzas sorry for not having a lot of time for upstream :(
15:56:42 tomtom001 sean-k-mooney Sorry i'm not understanding, from what I think is supposed to happen, is that qemu is supposed to handle luks encryption on the fly on a disk correct?
15:59:03 sean-k-mooney tomtom001: correct but that was only added to openstack in pike see https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/libvirt-qemu-native-luks.html
15:59:30 tomtom001 ok, so I'm running queens so that should be there.
15:59:42 tomtom001 or no it's not implemented yet?
16:00:02 sean-k-mooney tomtom001: no it should be there
16:01:02 tomtom001 so on one cpu host i've removed the xts module to ensure only the aes module is loaded, is there anything else I can check in qemu or libvirt or nova to ensure that encryption is operating normally?
16:01:45 tomtom001 qemu 2.11 is installed
16:02:14 tomtom001 libvirt 4.0.0 and nova compute from stale/queens
16:02:19 tomtom001 *stable
16:02:52 sean-k-mooney there versions of the software you are running should all support it
16:03:42 tomtom001 are there settings I need to check for this?
16:08:21 sean-k-mooney not that i a aware of but lyarwood might be able to provide more info
16:08:56 sean-k-mooney you will see the encyption nebaled in the xml/qemu command line but i dont know of anything you need to do to improve performance
16:19:36 tomtom001 sean-k-mooney thank you, lyarwood, you have anything I can check or something you'd like to look at to tell you?
16:42:08 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send rp_uuid in the port binding https://review.openstack.org/569459
16:42:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239
16:42:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
16:42:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: Pass allocations to neturonv2 api https://review.openstack.org/616240
16:50:21 efried mriedem: Update on the nova-next business...
16:50:41 efried https://review.openstack.org/#/c/616033/ see latest comment
16:50:58 efried tldr I don't think the tweak is working, or I'm a dolt and looking in the wrong place.
16:52:34 efried However, having looked at the placement logs for the nova-next runs in the whole series, I see a major drop at the "Reduce calls to placement from _ensure" patch, which is what I would expect; and actually, I wouldn't expect any of the CI jobs to see a significant drop from setting refresh=0 because they're getting refreshed all the time because they're going crazy doing real stuffs, so the refreshing is happening anyway and the
16:53:25 efried The refresh=0 thing is really only going to see a big benefit in steady state, and (afaik) we don't have any CI jobs that just... sit there.
16:54:57 mriedem i've had issues with using post-config and NOVA_CPU_CONF in the nova-live-migration job...
16:56:12 efried mriedem: I found a cells job that was using something similar and it seemed to be working (was able to verify the conf change was made)
16:56:54 efried mriedem: http://logs.openstack.org/33/616033/1/check/nova-cells-v1/fb84e57/logs/etc/nova/nova-cells.conf.txt.gz is getting vif_plugging_timeout = 0 from playbooks/legacy/nova-cells-v1/run.yaml
16:58:08 mriedem http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_24_44_044
16:58:08 mriedem devstack is processing the post-config
17:00:19 efried so... wtf?
17:00:57 mriedem it looks like that post-config happens first,
17:01:03 mriedem and then other stuff is writing to nova-cpu.conf
17:01:13 mriedem so my guess is that devstack is overwriting it...
17:01:51 efried ah http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_26_29_702
17:01:55 efried we blow it away later :(
17:02:09 efried what's the magic env var for the nova.conf?
17:02:15 mriedem yup http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_26_29_702
17:02:24 mriedem $NOVA_CONF
17:03:37 mriedem definitely a bug, but just do $NOVA_CONF for now and comment in the playbook
17:03:44 efried ack
17:04:04 mriedem i can report a devstack back
17:04:14 efried bug?
17:04:16 efried cool, thanks.
17:06:20 openstack Launchpad bug 1802143 in devstack "Changes from [[post-config|$NOVA_CPU_CONF]] are overwritten in "superconductor" mode" [Undecided,New]
17:06:20 mriedem https://bugs.launchpad.net/devstack/+bug/1802143
17:06:21 mriedem *yeah bug
17:07:35 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Turn off rp association refresh in nova-next https://review.openstack.org/616033
17:07:41 efried mriedem: done ^
17:08:07 efried mriedem: but as I said, I don't actually expect to see a significant difference, because active state.
17:08:28 efried though making sure it actually works and doesn't blow up the world will be good :)
17:10:43 mriedem hey now that the dems took over the house the world is going to end anyway
17:11:30 mriedem heh, *braces
17:11:50 mriedem sorry, trump is doing a news conference talking about the lack of republican losers that didn't accept his warm embrace
17:25:11 tssurya sean-k-mooney: I had a question, would you know the working of virsh capabilities (ca you point me to the code of how this xml is generated)?
17:25:22 tssurya couldn't find the exact source code
17:27:26 mriedem kashyap: ^
17:29:27 dansmith tssurya: it comes from libvirt, obviously
17:29:36 dansmith tssurya: there is a lot of documentation about what they all mean
17:29:49 tssurya dansmith: yea I tried to go into the libvirt code,, but in vain
17:29:55 dansmith tssurya: https://libvirt.org/formatcaps.html
17:30:06 dansmith tssurya: you want to know how it's determining one thing or another, or just what everything means?
17:30:09 tssurya I already saw that link
17:30:35 dansmith example of what you're trying to figure out?
17:30:49 tssurya I want to know how this Core(s) per socket in the topology is calculated
17:31:04 tssurya or the code in libvirt that gets this part
17:31:58 dansmith it should all be straight out of /sys I would think
17:32:25 dansmith you're talking about host caps in that case right? not how a guest's topo gets determined (by nova) but how libvirt reports the host topo?
17:32:49 tssurya yea the host caps (I think)

Earlier   Later