| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-07 | |||
| 13:18:21 | lyarwood | artom: yeah tis and I'll get to it after this call | |
| 13:18:23 | sean-k-mooney | gibi: if its not recoverable you should end the program | |
| 13:18:34 | artom | lyarwood, much thanks! | |
| 13:18:44 | sean-k-mooney | gibi: so that means you should be able to call the function again. | |
| 13:19:23 | gibi | sean-k-mooney: but allowing to call the function _again_ is something the run_once decorator should not allow at least that is what the name of the decorator suggests | |
| 13:20:13 | sean-k-mooney | gibi: the contract i wanted to give is this fucntion will run once sucessfully | |
| 13:20:14 | gibi | sean-k-mooney: I don't want to block your CI so I'm fine if you can document the behavior of the decorator precisely | |
| 13:20:30 | sean-k-mooney | sure ill add more documentaion | |
| 13:21:54 | sean-k-mooney | i could may be make the setting of the called value explcit but i was concered that people would complain it was over enginerred. my oringinal solution was a try block | |
| 13:22:14 | sean-k-mooney | then an if and then i was asked for a decorator | |
| 13:23:27 | gibi | sean-k-mooney: I'm one of the guy who feels it is overengineered so please do not add more configurability to the decorator, add just more documentation what is the intention of the decorator. Like that a run only counts if it wasn't raised | |
| 13:24:25 | sean-k-mooney | ok will do thanks ill get this back up in about 2 hours | |
| 13:24:39 | gibi | sean-k-mooney: ping me and I will try to +2 it quickly | |
| 13:42:08 | artom | lyarwood, there's the one below it as well ;) https://review.openstack.org/#/c/587248/1 | |
| 13:43:08 | artom | Cheers! | |
| 13:43:21 | lyarwood | artom: np LGTM | |
| 15:06:17 | tomtom001 | lyarwood: hello I was wondering you could say how the encryption on nova works with nova libvirt and ceph... we've noticed really slow performance when using luks encryption... I've turned off xts,and only use aes but it doesn't seem to change anything, what specifically needs to be in place to get a fast encryption for disks? | |
| 15:07:33 | lyarwood | tomtom001: Which version are you using and slow in what sense? Read / Writes / time taken to attach volumes etc? | |
| 15:12:26 | sean-k-mooney | tomtom001: does the host system have aes-ni processor instruction or is this older hardware. there are a log of vairibles that can come into play here | |
| 15:18:01 | tomtom001 | So they are DellFx2 servers, so recent hardware. We are using Queens as installed from Openstack-ansible and using Ceph luminous. i/o performance is 1/20th the speed of an unencrypted volume. | |
| 15:18:33 | tomtom001 | volume creation and attach time seems normal. | |
| 15:22:20 | sean-k-mooney | tomtom001: that would seam to imply that the perfromace pelenty is coming form the encryption overhead which should be low if its hardware offloaded but would be non trivial if aes-ni was not available | |
| 15:24:12 | cdent | efried: do open questions remain on your refresh stack or is it fairly straightforward from here? I'm trying to optimize my time before leaving for berlin (tomorrow) | |
| 15:25:32 | efried | cdent: I'm kind of a mess, need to go back through the emails and the series. But I *think* the only remaining questions are whether to make the timer zero by default for new installs; and/or make it the *only* thing - i.e. remove the option entirely and only refresh the cache when something happens. | |
| 15:26:26 | efried | cdent: and I've convinced the ptb that, if either of those things comes to pass, they should be done at least separately, and probably later once these have had some field time with CERN/Vexx | |
| 15:26:48 | cdent | makes sense | |
| 15:30:24 | sean-k-mooney | efried: for what its worth im fine with waiting to change the default but in T we proably shoudl change the default to 0 | |
| 15:31:05 | efried | sean-k-mooney: I don't even mind doing it earlier than that if we can demonstrate some traction in a prod env or two. | |
| 15:31:17 | sean-k-mooney | that said cern/vexxhost wont deploy stein right away so it would be late in T before we got any feedback | |
| 15:32:03 | efried | sean-k-mooney: I may be able to backport this code. | |
| 15:32:12 | efried | sean-k-mooney: Not to merge necessarily, but at least so they can pull it into their env. | |
| 15:32:18 | sean-k-mooney | right the catalyst for changing the default would production use showing it works for them | |
| 15:39:20 | tomtom001 | sean-k-mooney - right so hardware offloaded would just be hardware encryption, we are looking at it from the cinder luks ceph side. | |
| 15:40:10 | sean-k-mooney | tomtom001: well hardware offload would acclerate the luks part | |
| 15:40:14 | mriedem | bauzas: do you care about this at all? you added the original lazy-load scheduler report client stuff https://review.openstack.org/#/c/615641/ | |
| 15:40:32 | sean-k-mooney | cinder and ceph should not chagne if you enable encryption | |
| 15:40:45 | bauzas | mriedem: for sure, I'm interested but I have a long customer issue | |
| 15:41:55 | efried | mriedem: bauzas and I discussed it yesterday. I'm not sure whether I succeeded in convincing him that we weren't actually deferring importing the modules until method call. But he also admitted the only reason we were doing it at all was to avoid a circular import, which is clearly not an issue anymore. | |
| 15:43:31 | efried | btw, at some point when I get breathing room, I'm going to rip out the SchedulerClient entirely. At the top of my series, it only has methods to the query client (all report client passthrough methods are gone). And I want to suck up to jaypipes some more anyway. | |
| 15:43:51 | mriedem | i think that's the nit todo i just left on that change | |
| 15:43:54 | mriedem | eventually remove the proxy | |
| 15:44:22 | cdent | I'm glad we seem to have agreement on that idea | |
| 15:44:31 | efried | yeah, for sure. I've looked into it before and it was going to be nontrivial, which is why I haven't done it in this series. | |
| 15:44:42 | efried | but this series will make it a little bit easier. | |
| 15:44:46 | bauzas | efried: mriedem: +Wd | |
| 15:45:02 | mriedem | well looky there | |
| 15:45:02 | efried | bauzas: Thank you sir | |
| 15:45:18 | bauzas | sorry for not having a lot of time for upstream :( | |
| 15:56:42 | tomtom001 | sean-k-mooney Sorry i'm not understanding, from what I think is supposed to happen, is that qemu is supposed to handle luks encryption on the fly on a disk correct? | |
| 15:59:03 | sean-k-mooney | tomtom001: correct but that was only added to openstack in pike see https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/libvirt-qemu-native-luks.html | |
| 15:59:30 | tomtom001 | ok, so I'm running queens so that should be there. | |
| 15:59:42 | tomtom001 | or no it's not implemented yet? | |
| 16:00:02 | sean-k-mooney | tomtom001: no it should be there | |
| 16:01:02 | tomtom001 | so on one cpu host i've removed the xts module to ensure only the aes module is loaded, is there anything else I can check in qemu or libvirt or nova to ensure that encryption is operating normally? | |
| 16:01:45 | tomtom001 | qemu 2.11 is installed | |
| 16:02:14 | tomtom001 | libvirt 4.0.0 and nova compute from stale/queens | |
| 16:02:19 | tomtom001 | *stable | |
| 16:02:52 | sean-k-mooney | there versions of the software you are running should all support it | |
| 16:03:42 | tomtom001 | are there settings I need to check for this? | |
| 16:08:21 | sean-k-mooney | not that i a aware of but lyarwood might be able to provide more info | |
| 16:08:56 | sean-k-mooney | you will see the encyption nebaled in the xml/qemu command line but i dont know of anything you need to do to improve performance | |
| 16:19:36 | tomtom001 | sean-k-mooney thank you, lyarwood, you have anything I can check or something you'd like to look at to tell you? | |
| 16:42:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send rp_uuid in the port binding https://review.openstack.org/569459 | |
| 16:42:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 16:42:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239 | |
| 16:42:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass allocations to neturonv2 api https://review.openstack.org/616240 | |
| 16:50:21 | efried | mriedem: Update on the nova-next business... | |
| 16:50:41 | efried | https://review.openstack.org/#/c/616033/ see latest comment | |
| 16:50:58 | efried | tldr I don't think the tweak is working, or I'm a dolt and looking in the wrong place. | |
| 16:52:34 | efried | However, having looked at the placement logs for the nova-next runs in the whole series, I see a major drop at the "Reduce calls to placement from _ensure" patch, which is what I would expect; and actually, I wouldn't expect any of the CI jobs to see a significant drop from setting refresh=0 because they're getting refreshed all the time because they're going crazy doing real stuffs, so the refreshing is happening anyway and the | |
| 16:53:25 | efried | The refresh=0 thing is really only going to see a big benefit in steady state, and (afaik) we don't have any CI jobs that just... sit there. | |
| 16:54:57 | mriedem | i've had issues with using post-config and NOVA_CPU_CONF in the nova-live-migration job... | |
| 16:56:12 | efried | mriedem: I found a cells job that was using something similar and it seemed to be working (was able to verify the conf change was made) | |
| 16:56:54 | efried | mriedem: http://logs.openstack.org/33/616033/1/check/nova-cells-v1/fb84e57/logs/etc/nova/nova-cells.conf.txt.gz is getting vif_plugging_timeout = 0 from playbooks/legacy/nova-cells-v1/run.yaml | |
| 16:58:08 | mriedem | devstack is processing the post-config | |
| 16:58:08 | mriedem | http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_24_44_044 | |
| 17:00:19 | efried | so... wtf? | |
| 17:00:57 | mriedem | it looks like that post-config happens first, | |
| 17:01:03 | mriedem | and then other stuff is writing to nova-cpu.conf | |
| 17:01:13 | mriedem | so my guess is that devstack is overwriting it... | |
| 17:01:51 | efried | ah http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_26_29_702 | |
| 17:01:55 | efried | we blow it away later :( | |
| 17:02:09 | efried | what's the magic env var for the nova.conf? | |
| 17:02:15 | mriedem | yup http://logs.openstack.org/33/616033/1/check/nova-next/6e893b8/logs/devstacklog.txt.gz#_2018-11-07_00_26_29_702 | |
| 17:02:24 | mriedem | $NOVA_CONF | |
| 17:03:37 | mriedem | definitely a bug, but just do $NOVA_CONF for now and comment in the playbook | |
| 17:03:44 | efried | ack | |
| 17:04:04 | mriedem | i can report a devstack back | |
| 17:04:14 | efried | bug? | |
| 17:04:16 | efried | cool, thanks. | |
| 17:06:20 | mriedem | https://bugs.launchpad.net/devstack/+bug/1802143 | |
| 17:06:20 | openstack | Launchpad bug 1802143 in devstack "Changes from [[post-config|$NOVA_CPU_CONF]] are overwritten in "superconductor" mode" [Undecided,New] | |
| 17:06:21 | mriedem | *yeah bug | |
| 17:07:35 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Turn off rp association refresh in nova-next https://review.openstack.org/616033 | |
| 17:07:41 | efried | mriedem: done ^ | |
| 17:08:07 | efried | mriedem: but as I said, I don't actually expect to see a significant difference, because active state. | |
| 17:08:28 | efried | though making sure it actually works and doesn't blow up the world will be good :) | |
| 17:10:43 | mriedem | hey now that the dems took over the house the world is going to end anyway | |
| 17:11:30 | mriedem | heh, *braces | |
| 17:11:50 | mriedem | sorry, trump is doing a news conference talking about the lack of republican losers that didn't accept his warm embrace | |