Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-07
12:57:58 sean-k-mooney that said i also plan to back port this so i what this to be right
12:58:26 sean-k-mooney cdent: nova need to have it done for the nova-api too however
12:58:43 sean-k-mooney do we want two ifs one for each both depending on private data
12:59:30 cdent I don't have a strong preference, except that we move it forward quickly in some way that satisfies us and recognize that it is okay to fix it more later.
13:00:36 sean-k-mooney cdent: ya same. this has been blocking downstream ci for almost a month at this point. i will have to consider applying a donwstream only fix if we dont merge something with the next week
13:01:45 cdent Yeah, we seem to have quite the backlog at the moment
13:05:13 cdent sean-k-mooney: I wish I could help to move things a bit more
13:06:12 sean-k-mooney cdent: i need to run to the bank so ill be back in a while. efried spotted a typo so i was going to resping for that and ill add the extra docs that gibi asked for too. unfortunetly that will cause curn on the placement side is that ok with you?
13:06:18 sean-k-mooney cdent: and no worries
13:06:35 sean-k-mooney you have help move it along alot already
13:07:13 cdent no worries on the placement side, I'll watch that and make sure it's properly rebased etc. The test load from placement is currently very light and fast, so no issues there.
13:07:41 sean-k-mooney thanks :) ok got to run brb
13:13:20 gibi sean-k-mooney: my -1 is mostly due to the problem with raised exception from the wrapped function
13:13:40 gibi sean-k-mooney: the rest is mostly thinking out loud different alternatives
13:14:43 sean-k-mooney gibi: well we dont want to catch exceptions when we invoke the wrapped function the first time. but we could catch exceptions when logging i guess
13:15:05 gibi sean-k-mooney: my problem is that if the first call raises then we allow a second call
13:15:18 gibi sean-k-mooney: because called only set _after_ the wrapped function returns
13:15:38 sean-k-mooney gibi: yes because this is a general decorator and we should not assume that means another call to the same fucntion will also raise
13:16:17 gibi sean-k-mooney: for me the contract of that decorator is that the decorated function will be called at most once. But the actual implementation allows any number of calls to the wrapped function if the wrapped function raises
13:16:38 sean-k-mooney gibi: yes the order of setting called after was very deliberate
13:17:35 gibi so if my func() has some code in it that can only be run once, but also my func() can raise then I cannot use the new decorator
13:17:36 sean-k-mooney gibi: for me it will be called at most once if it succeeds and the fuctor object will be in its orignal state if not
13:17:42 artom lyarwood, could you hit https://review.openstack.org/#/c/587249/ ? It's the pike version of a backport you +2'ed for Queens - I'll try and get Matt R as well when he comes onlone
13:18:01 artom (Pike is still a thing, right? I know Ocata is EM now)
13:18:09 sean-k-mooney gibi: you can but if you caugt the excption that by definitoin means its recoverable
13:18:21 lyarwood artom: yeah tis and I'll get to it after this call
13:18:23 sean-k-mooney gibi: if its not recoverable you should end the program
13:18:34 artom lyarwood, much thanks!
13:18:44 sean-k-mooney gibi: so that means you should be able to call the function again.
13:19:23 gibi sean-k-mooney: but allowing to call the function _again_ is something the run_once decorator should not allow at least that is what the name of the decorator suggests
13:20:13 sean-k-mooney gibi: the contract i wanted to give is this fucntion will run once sucessfully
13:20:14 gibi sean-k-mooney: I don't want to block your CI so I'm fine if you can document the behavior of the decorator precisely
13:20:30 sean-k-mooney sure ill add more documentaion
13:21:54 sean-k-mooney i could may be make the setting of the called value explcit but i was concered that people would complain it was over enginerred. my oringinal solution was a try block
13:22:14 sean-k-mooney then an if and then i was asked for a decorator
13:23:27 gibi sean-k-mooney: I'm one of the guy who feels it is overengineered so please do not add more configurability to the decorator, add just more documentation what is the intention of the decorator. Like that a run only counts if it wasn't raised
13:24:25 sean-k-mooney ok will do thanks ill get this back up in about 2 hours
13:24:39 gibi sean-k-mooney: ping me and I will try to +2 it quickly
13:42:08 artom lyarwood, there's the one below it as well ;) https://review.openstack.org/#/c/587248/1
13:43:08 artom Cheers!
13:43:21 lyarwood artom: np LGTM
15:06:17 tomtom001 lyarwood: hello I was wondering you could say how the encryption on nova works with nova libvirt and ceph... we've noticed really slow performance when using luks encryption... I've turned off xts,and only use aes but it doesn't seem to change anything, what specifically needs to be in place to get a fast encryption for disks?
15:07:33 lyarwood tomtom001: Which version are you using and slow in what sense? Read / Writes / time taken to attach volumes etc?
15:12:26 sean-k-mooney tomtom001: does the host system have aes-ni processor instruction or is this older hardware. there are a log of vairibles that can come into play here
15:18:01 tomtom001 So they are DellFx2 servers, so recent hardware. We are using Queens as installed from Openstack-ansible and using Ceph luminous. i/o performance is 1/20th the speed of an unencrypted volume.
15:18:33 tomtom001 volume creation and attach time seems normal.
15:22:20 sean-k-mooney tomtom001: that would seam to imply that the perfromace pelenty is coming form the encryption overhead which should be low if its hardware offloaded but would be non trivial if aes-ni was not available
15:24:12 cdent efried: do open questions remain on your refresh stack or is it fairly straightforward from here? I'm trying to optimize my time before leaving for berlin (tomorrow)
15:25:32 efried cdent: I'm kind of a mess, need to go back through the emails and the series. But I *think* the only remaining questions are whether to make the timer zero by default for new installs; and/or make it the *only* thing - i.e. remove the option entirely and only refresh the cache when something happens.
15:26:26 efried cdent: and I've convinced the ptb that, if either of those things comes to pass, they should be done at least separately, and probably later once these have had some field time with CERN/Vexx
15:26:48 cdent makes sense
15:30:24 sean-k-mooney efried: for what its worth im fine with waiting to change the default but in T we proably shoudl change the default to 0
15:31:05 efried sean-k-mooney: I don't even mind doing it earlier than that if we can demonstrate some traction in a prod env or two.
15:31:17 sean-k-mooney that said cern/vexxhost wont deploy stein right away so it would be late in T before we got any feedback
15:32:03 efried sean-k-mooney: I may be able to backport this code.
15:32:12 efried sean-k-mooney: Not to merge necessarily, but at least so they can pull it into their env.
15:32:18 sean-k-mooney right the catalyst for changing the default would production use showing it works for them
15:39:20 tomtom001 sean-k-mooney - right so hardware offloaded would just be hardware encryption, we are looking at it from the cinder luks ceph side.
15:40:10 sean-k-mooney tomtom001: well hardware offload would acclerate the luks part
15:40:14 mriedem bauzas: do you care about this at all? you added the original lazy-load scheduler report client stuff https://review.openstack.org/#/c/615641/
15:40:32 sean-k-mooney cinder and ceph should not chagne if you enable encryption
15:40:45 bauzas mriedem: for sure, I'm interested but I have a long customer issue
15:41:55 efried mriedem: bauzas and I discussed it yesterday. I'm not sure whether I succeeded in convincing him that we weren't actually deferring importing the modules until method call. But he also admitted the only reason we were doing it at all was to avoid a circular import, which is clearly not an issue anymore.
15:43:31 efried btw, at some point when I get breathing room, I'm going to rip out the SchedulerClient entirely. At the top of my series, it only has methods to the query client (all report client passthrough methods are gone). And I want to suck up to jaypipes some more anyway.
15:43:51 mriedem i think that's the nit todo i just left on that change
15:43:54 mriedem eventually remove the proxy
15:44:22 cdent I'm glad we seem to have agreement on that idea
15:44:31 efried yeah, for sure. I've looked into it before and it was going to be nontrivial, which is why I haven't done it in this series.
15:44:42 efried but this series will make it a little bit easier.
15:44:46 bauzas efried: mriedem: +Wd
15:45:02 mriedem well looky there
15:45:02 efried bauzas: Thank you sir
15:45:18 bauzas sorry for not having a lot of time for upstream :(
15:56:42 tomtom001 sean-k-mooney Sorry i'm not understanding, from what I think is supposed to happen, is that qemu is supposed to handle luks encryption on the fly on a disk correct?
15:59:03 sean-k-mooney tomtom001: correct but that was only added to openstack in pike see https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/libvirt-qemu-native-luks.html
15:59:30 tomtom001 ok, so I'm running queens so that should be there.
15:59:42 tomtom001 or no it's not implemented yet?
16:00:02 sean-k-mooney tomtom001: no it should be there
16:01:02 tomtom001 so on one cpu host i've removed the xts module to ensure only the aes module is loaded, is there anything else I can check in qemu or libvirt or nova to ensure that encryption is operating normally?
16:01:45 tomtom001 qemu 2.11 is installed
16:02:14 tomtom001 libvirt 4.0.0 and nova compute from stale/queens
16:02:19 tomtom001 *stable
16:02:52 sean-k-mooney there versions of the software you are running should all support it
16:03:42 tomtom001 are there settings I need to check for this?
16:08:21 sean-k-mooney not that i a aware of but lyarwood might be able to provide more info
16:08:56 sean-k-mooney you will see the encyption nebaled in the xml/qemu command line but i dont know of anything you need to do to improve performance
16:19:36 tomtom001 sean-k-mooney thank you, lyarwood, you have anything I can check or something you'd like to look at to tell you?
16:42:08 openstackgerrit Balazs Gibizer proposed openstack/nova master: Send rp_uuid in the port binding https://review.openstack.org/569459
16:42:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317
16:42:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Calculate port_id rp_uuid mapping for binding https://review.openstack.org/616239
16:42:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: Pass allocations to neturonv2 api https://review.openstack.org/616240
16:50:21 efried mriedem: Update on the nova-next business...
16:50:41 efried https://review.openstack.org/#/c/616033/ see latest comment
16:50:58 efried tldr I don't think the tweak is working, or I'm a dolt and looking in the wrong place.
16:52:34 efried However, having looked at the placement logs for the nova-next runs in the whole series, I see a major drop at the "Reduce calls to placement from _ensure" patch, which is what I would expect; and actually, I wouldn't expect any of the CI jobs to see a significant drop from setting refresh=0 because they're getting refreshed all the time because they're going crazy doing real stuffs, so the refreshing is happening anyway and the
16:53:25 efried The refresh=0 thing is really only going to see a big benefit in steady state, and (afaik) we don't have any CI jobs that just... sit there.
16:54:57 mriedem i've had issues with using post-config and NOVA_CPU_CONF in the nova-live-migration job...
16:56:12 efried mriedem: I found a cells job that was using something similar and it seemed to be working (was able to verify the conf change was made)
16:56:54 efried mriedem: http://logs.openstack.org/33/616033/1/check/nova-cells-v1/fb84e57/logs/etc/nova/nova-cells.conf.txt.gz is getting vif_plugging_timeout = 0 from playbooks/legacy/nova-cells-v1/run.yaml
16:58:08 mriedem devstack is processing the post-config

Earlier   Later