Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-31
14:58:52 mriedem dansmith: yeah https://bugs.launchpad.net/openstack-gate/+bug/1790152
14:58:52 openstack Launchpad bug 1790152 in OpenStack-Gate "jobs failing with ansible "ImportError: No module named manager" since 8/30" [Undecided,Fix released]
14:58:57 mriedem already fixed but broke everything last night
15:00:04 dansmith yeah, but are we still on that backlog?
15:00:19 dansmith at 2am this morning they said "go ahead and recheck" -- maybe it's just the influx of everyone doing that/
15:02:25 mriedem i assume so yes - everyone woke up and started rechecking the world
15:02:51 openstackgerrit Merged openstack/nova master: Remove old check_attach version check in API https://review.openstack.org/588348
15:04:36 mriedem ^ feels good to have that gone
15:14:53 openstackgerrit Matt Riedemann proposed openstack/nova master: Don't persist zero allocation ratios in ResourceTracker https://review.openstack.org/598365
15:22:49 cdent sean-k-mooney: in case you're following along at home my car failed its mot rather spectacularly
15:25:56 sean-k-mooney cdent: i was actully wondering about that
15:26:32 cdent i'll find out the details in a few minutes when I go to pick it up. the parts it needs may be impossible to get.
15:26:50 sean-k-mooney cdent: fixable or not viable
15:27:26 cdent well, fixable with a) much money, b) access to parts. Both require some effort to resolve.
15:27:49 sean-k-mooney ya what is it by the way. you mentioned it was 21 years old but not the make or modle
16:34:03 sean-k-mooney melwitt: finucannot since sahid was fine https://review.openstack.org/#/q/topic:bug/1759420+(status:open+OR+status:merged) and melwitt mentioned to not squash unless i respin are they good to go?
16:35:15 melwitt sean-k-mooney: probably. I'd ask mriedem_away if he thinks squashing is important but he's away for atm
16:35:58 finucannot sean-k-mooney: Sure. I've +2d the bottom one but, in hindsight, I might leave both to melwitt seeing as she got there first (assuming that's alright with her)
16:37:18 melwitt finucannot: I'm fine with you taking them, I can +1 them anyway. I want to ask mriedem_away about squashing since it's related to the backport
16:37:50 finucannot melwitt: Yeah, it can wait until next week, in that case
16:38:03 melwitt we routinely backport 2-change deals for functional regression tests, but this is a unit regression test, just want to make sure with him since I'm not sure it matters
16:38:12 sean-k-mooney sure just comment or ping me if you want it squashed. its like 30 seconds to do but the gate is busy at the moment so did not want to push a change for no reason
16:38:42 melwitt ok, I'll let you know
17:43:00 hamzy hey y'all, my ethernet connections in instances installed via TripleO doesn't work. I seem to see all the bridges and connections setup, but some final qemu hardware connection is broken or missing http://paste.openstack.org/show/729241/
17:44:02 sean-k-mooney hamzy: that all seams fine at first glance
17:44:14 sean-k-mooney hamzy: how did you determihn that the connectivity did not work
17:45:58 hamzy sean-k-mooney, on lines 78+ I can't get the IP via DHCP, or when I set it manually, ping the router
17:48:11 sean-k-mooney hamzy: is dhcp enable on the neutron subnet
17:48:15 cfriesen who's a good person to talk to about performance issues?
17:48:25 cfriesen I mean process startup times
17:49:35 sean-k-mooney cfriesen: as in makeing them better or debuging a problem
17:50:47 sean-k-mooney cfriesen: if your debugging a proablem i proably cant help but if you have a proposal on how to make it faster then im happy to be a sound board to bounce ideas off
17:50:49 cfriesen sean-k-mooney: in nova/compute/flavors.py we call "from nova.api.validation import parameter_types". This appears to be really expensive (~6 seconds in a recent test) due to the regex stuff, which makes a controller node startup fairly expensive
17:50:55 hamzy sean-k-mooney, yes I believe that it is... all ports on 'openstack port list' are active
17:51:48 sean-k-mooney cfriesen: ocuch 6 seconds for a singel function call for paramater validation is nuts
17:52:23 sean-k-mooney hamzy: yes they will be active regardless of if dhcp is enable on the subnet
17:52:28 cfriesen sean-k-mooney: that was in vbox, but it's still multiple seconds on bare metal. internally we moved the import down into the actual code so it's only done if we actually do the flavor creation
17:52:31 sean-k-mooney they also will have ips assigned
17:53:15 sean-k-mooney cfriesen: wait the import is executing code? is it building the regex on import or something if so that is terible
17:53:38 hamzy sean-k-mooney, http://paste.openstack.org/show/729253/
17:54:02 hamzy is the DHCP agent only run on the controller?
17:54:56 sean-k-mooney cfriesen: sigh... https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L144-L492 its all at file scope we do this on every import
17:55:08 cfriesen sean-k-mooney: precisely
17:56:24 sean-k-mooney cfriesen: can you open a bug for that. i would have to check what this actully used for but personally i think we should lazy initallies the values or replace them with caching factoryFunctions
17:56:44 cfriesen sean-k-mooney: sure
17:57:06 sean-k-mooney the simplest thing to do is proably make them all properties but not sure that is leagal at file/module scope
17:57:38 sean-k-mooney hamzy: yes it should only be running on the contoller/networking nodes
17:58:33 hamzy sean-k-mooney, I would think that "sudo ip addr add ..." and ping .254 (router) should work as a test for network connectivity, right?
17:58:51 sean-k-mooney hamzy: can you do an "openstack subnet list" and then "openstack subnet show" for the subnet the vm is connected too
17:58:56 leakypipes cfriesen, sean-k-mooney: I'm actually only seeing a single re.compile() in that module...
17:59:22 leakypipes cfriesen, sean-k-mooney: It would seem like it would be more efficient to be re.compile()'ing all the regexes that are in module scope in that module.
18:00:28 sean-k-mooney leakypipes: actully i had assumed the ValidationRegex object was compiling them but i see now its not
18:01:10 leakypipes sean-k-mooney: no it is not...
18:01:33 leakypipes sean-k-mooney: or at least, it isn't AFAICT
18:02:02 sean-k-mooney ya looking over the file level stuff its not really doing anything expecive
18:02:16 cfriesen try running python and then doing "from nova.api.validation import parameter_types"
18:02:45 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/api/validation/__init__.py however might be
18:03:28 sean-k-mooney also not really be i hate when people hide stuff in __init__.py files
18:04:54 cfriesen just retested and it's 3 secs on pretty beefy hardware
18:05:25 sean-k-mooney cfriesen: and if you move the import to the fuction where its used what happens?
18:05:45 cfriesen sean-k-mooney: then you don't hit the delay at process startup, only when you actually run the function
18:06:11 sean-k-mooney cfriesen: yes but what is the delta in start up
18:06:22 sean-k-mooney if its 3 second to 2.9 then do we care
18:06:43 cfriesen no, it's 3 secs just for "from nova.api.validation import parameter_types"
18:06:56 sean-k-mooney oh that is different
18:07:25 leakypipes sean-k-mooney: well, there's no state in the dunderinit file at least... just functions.
18:07:44 sean-k-mooney cfriesen: can you temporally comment out the file level suff just to see if it changes
18:09:07 leakypipes holy fucksticks, Batman.
18:09:08 leakypipes [jaypipes@uberbox nova]$ source .tox/py27/bin/activate
18:09:09 leakypipes (py27) [jaypipes@uberbox nova]$ time python -m nova.api.validation.parameter_types
18:09:09 leakypipes real0m5.653s
18:09:09 leakypipes user0m3.331s
18:09:09 leakypipes sys0m0.201s
18:09:34 sean-k-mooney cfriesen: the only thin that looks even remotely expensive is maybe _build_regex_range and even then the most expecive thing i see there is re.escape
18:10:08 sean-k-mooney i dont really like this https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L128-L142
18:11:08 sean-k-mooney def _get_all_chars():
18:11:10 sean-k-mooney for i in range(0xFFFF):
18:11:12 sean-k-mooney yield six.unichr(i)
18:11:50 sean-k-mooney so that is got to loop 65535 times
18:12:01 sean-k-mooney *going too
18:12:44 sean-k-mooney and we call the function 17 times
18:14:08 sean-k-mooney so that 1.1 million callse to re.escape every time we load that module
18:14:45 sean-k-mooney leakypipes: cfriesen shall i fix this or do one of ye wnat to do it
18:15:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Default AZ for instance if cross_az_attach=False and checking from API https://review.openstack.org/469675
18:15:40 cfriesen sean-k-mooney: go for it. :)
18:16:00 sean-k-mooney cfriesen: good find by the way that is nuts
18:17:25 cfriesen sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1790195
18:17:25 openstack Launchpad bug 1790195 in OpenStack Compute (nova) "performance problems starting up nova process due to regex code" [Undecided,New]
18:18:17 leakypipes cfriesen: ++
18:21:06 cfriesen sean-k-mooney: I think that's likely the culprit. Changing it to "for i in range(0x1):" made "python -m nova.api.validation.parameter_types" take 1.5 sec instead of 7.
18:21:19 mriedem hmm, so https://review.openstack.org/#/c/598366/ passed with https://review.openstack.org/#/c/598365/ but i'm not sure if that tells us much
18:21:48 sean-k-mooney cfriesen: ill fix this two ways. first ill cache te result of _build_regex_range wit a decorator and second ill probably convert the file level vars to propertys so they are lazy evaulted
18:22:04 cfriesen sean-k-mooney: sounds good
18:48:52 openstackgerrit Adam Harwell proposed openstack/nova stable/pike: Add apply_cells to nova-manage https://review.openstack.org/599050
18:51:48 openstackgerrit Merged openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049
18:51:57 openstackgerrit Merged openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236
18:52:07 openstackgerrit Merged openstack/nova master: Do test_reshape with an actual startup https://review.openstack.org/597218
18:52:16 openstackgerrit Merged openstack/nova master: Fix reshaper report client functonal test nits https://review.openstack.org/598330
18:55:28 cfriesen any chance of a second core looking at https://review.openstack.org/#/c/588657/ ? It's a fairly straightforward change related to image properties, claims, and evacuate.
18:58:53 mriedem that is an excellent patch

Earlier   Later