| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-31 | |||
| 17:51:48 | sean-k-mooney | cfriesen: ocuch 6 seconds for a singel function call for paramater validation is nuts | |
| 17:52:23 | sean-k-mooney | hamzy: yes they will be active regardless of if dhcp is enable on the subnet | |
| 17:52:28 | cfriesen | sean-k-mooney: that was in vbox, but it's still multiple seconds on bare metal. internally we moved the import down into the actual code so it's only done if we actually do the flavor creation | |
| 17:52:31 | sean-k-mooney | they also will have ips assigned | |
| 17:53:15 | sean-k-mooney | cfriesen: wait the import is executing code? is it building the regex on import or something if so that is terible | |
| 17:53:38 | hamzy | sean-k-mooney, http://paste.openstack.org/show/729253/ | |
| 17:54:02 | hamzy | is the DHCP agent only run on the controller? | |
| 17:54:56 | sean-k-mooney | cfriesen: sigh... https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L144-L492 its all at file scope we do this on every import | |
| 17:55:08 | cfriesen | sean-k-mooney: precisely | |
| 17:56:24 | sean-k-mooney | cfriesen: can you open a bug for that. i would have to check what this actully used for but personally i think we should lazy initallies the values or replace them with caching factoryFunctions | |
| 17:56:44 | cfriesen | sean-k-mooney: sure | |
| 17:57:06 | sean-k-mooney | the simplest thing to do is proably make them all properties but not sure that is leagal at file/module scope | |
| 17:57:38 | sean-k-mooney | hamzy: yes it should only be running on the contoller/networking nodes | |
| 17:58:33 | hamzy | sean-k-mooney, I would think that "sudo ip addr add ..." and ping .254 (router) should work as a test for network connectivity, right? | |
| 17:58:51 | sean-k-mooney | hamzy: can you do an "openstack subnet list" and then "openstack subnet show" for the subnet the vm is connected too | |
| 17:58:56 | leakypipes | cfriesen, sean-k-mooney: I'm actually only seeing a single re.compile() in that module... | |
| 17:59:22 | leakypipes | cfriesen, sean-k-mooney: It would seem like it would be more efficient to be re.compile()'ing all the regexes that are in module scope in that module. | |
| 18:00:28 | sean-k-mooney | leakypipes: actully i had assumed the ValidationRegex object was compiling them but i see now its not | |
| 18:01:10 | leakypipes | sean-k-mooney: no it is not... | |
| 18:01:33 | leakypipes | sean-k-mooney: or at least, it isn't AFAICT | |
| 18:02:02 | sean-k-mooney | ya looking over the file level stuff its not really doing anything expecive | |
| 18:02:16 | cfriesen | try running python and then doing "from nova.api.validation import parameter_types" | |
| 18:02:45 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/api/validation/__init__.py however might be | |
| 18:03:28 | sean-k-mooney | also not really be i hate when people hide stuff in __init__.py files | |
| 18:04:54 | cfriesen | just retested and it's 3 secs on pretty beefy hardware | |
| 18:05:25 | sean-k-mooney | cfriesen: and if you move the import to the fuction where its used what happens? | |
| 18:05:45 | cfriesen | sean-k-mooney: then you don't hit the delay at process startup, only when you actually run the function | |
| 18:06:11 | sean-k-mooney | cfriesen: yes but what is the delta in start up | |
| 18:06:22 | sean-k-mooney | if its 3 second to 2.9 then do we care | |
| 18:06:43 | cfriesen | no, it's 3 secs just for "from nova.api.validation import parameter_types" | |
| 18:06:56 | sean-k-mooney | oh that is different | |
| 18:07:25 | leakypipes | sean-k-mooney: well, there's no state in the dunderinit file at least... just functions. | |
| 18:07:44 | sean-k-mooney | cfriesen: can you temporally comment out the file level suff just to see if it changes | |
| 18:09:07 | leakypipes | holy fucksticks, Batman. | |
| 18:09:08 | leakypipes | [jaypipes@uberbox nova]$ source .tox/py27/bin/activate | |
| 18:09:09 | leakypipes | sys0m0.201s | |
| 18:09:09 | leakypipes | user0m3.331s | |
| 18:09:09 | leakypipes | real0m5.653s | |
| 18:09:09 | leakypipes | (py27) [jaypipes@uberbox nova]$ time python -m nova.api.validation.parameter_types | |
| 18:09:34 | sean-k-mooney | cfriesen: the only thin that looks even remotely expensive is maybe _build_regex_range and even then the most expecive thing i see there is re.escape | |
| 18:10:08 | sean-k-mooney | i dont really like this https://github.com/openstack/nova/blob/master/nova/api/validation/parameter_types.py#L128-L142 | |
| 18:11:08 | sean-k-mooney | def _get_all_chars(): | |
| 18:11:10 | sean-k-mooney | for i in range(0xFFFF): | |
| 18:11:12 | sean-k-mooney | yield six.unichr(i) | |
| 18:11:50 | sean-k-mooney | so that is got to loop 65535 times | |
| 18:12:01 | sean-k-mooney | *going too | |
| 18:12:44 | sean-k-mooney | and we call the function 17 times | |
| 18:14:08 | sean-k-mooney | so that 1.1 million callse to re.escape every time we load that module | |
| 18:14:45 | sean-k-mooney | leakypipes: cfriesen shall i fix this or do one of ye wnat to do it | |
| 18:15:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Default AZ for instance if cross_az_attach=False and checking from API https://review.openstack.org/469675 | |
| 18:15:40 | cfriesen | sean-k-mooney: go for it. :) | |
| 18:16:00 | sean-k-mooney | cfriesen: good find by the way that is nuts | |
| 18:17:25 | openstack | Launchpad bug 1790195 in OpenStack Compute (nova) "performance problems starting up nova process due to regex code" [Undecided,New] | |
| 18:17:25 | cfriesen | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1790195 | |
| 18:18:17 | leakypipes | cfriesen: ++ | |
| 18:21:06 | cfriesen | sean-k-mooney: I think that's likely the culprit. Changing it to "for i in range(0x1):" made "python -m nova.api.validation.parameter_types" take 1.5 sec instead of 7. | |
| 18:21:19 | mriedem | hmm, so https://review.openstack.org/#/c/598366/ passed with https://review.openstack.org/#/c/598365/ but i'm not sure if that tells us much | |
| 18:21:48 | sean-k-mooney | cfriesen: ill fix this two ways. first ill cache te result of _build_regex_range wit a decorator and second ill probably convert the file level vars to propertys so they are lazy evaulted | |
| 18:22:04 | cfriesen | sean-k-mooney: sounds good | |
| 18:48:52 | openstackgerrit | Adam Harwell proposed openstack/nova stable/pike: Add apply_cells to nova-manage https://review.openstack.org/599050 | |
| 18:51:48 | openstackgerrit | Merged openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049 | |
| 18:51:57 | openstackgerrit | Merged openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 18:52:07 | openstackgerrit | Merged openstack/nova master: Do test_reshape with an actual startup https://review.openstack.org/597218 | |
| 18:52:16 | openstackgerrit | Merged openstack/nova master: Fix reshaper report client functonal test nits https://review.openstack.org/598330 | |
| 18:55:28 | cfriesen | any chance of a second core looking at https://review.openstack.org/#/c/588657/ ? It's a fairly straightforward change related to image properties, claims, and evacuate. | |
| 18:58:53 | mriedem | that is an excellent patch | |
| 19:01:23 | leakypipes | cfriesen: ack, I'll review shortly. | |
| 19:07:30 | cfriesen | leakypipes: much appreciated | |
| 19:08:07 | leakypipes | cfriesen: you KNOW how much I love the instance migration/evacuate/shelve/funkychicken code paths. | |
| 19:08:42 | cfriesen | just the thing for a friday | |
| 19:11:15 | sean-k-mooney | cfriesen: so the decorator alone went from 0m3.535s to 1.128s | |
| 19:11:43 | cfriesen | cool | |
| 19:33:54 | leakypipes | cfriesen: +W | |
| 19:34:22 | cfriesen | leakypipes: sweet, thx | |
| 19:43:29 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [Medium,Triaged] | |
| 19:43:29 | mriedem | fried_rice: you wanted to know when i reported this https://bugs.launchpad.net/nova/+bug/1790204 | |
| 19:43:40 | mriedem | cfriesen: ^ fyi | |
| 19:44:05 | dansmith | mriedem: shall we? https://review.openstack.org/#/c/598353 | |
| 19:44:26 | mriedem | dansmith: see my comment on the test? | |
| 19:44:42 | dansmith | oh, sure | |
| 19:44:48 | mriedem | iow, if i remove the fix the test would still pass i think | |
| 19:44:49 | fried_rice | mriedem: ack, thx | |
| 19:44:59 | dansmith | mriedem: yep probably | |
| 19:45:15 | dansmith | mriedem: the way the gate is that makes this probably tuesday before it's in | |
| 19:45:24 | dansmith | I dunno what the fail rate is | |
| 19:45:50 | mriedem | my gut says the pass rate is not great | |
| 19:46:09 | dansmith | fail rate? | |
| 19:46:22 | mriedem | but this has also been this way since queens i think when i changed devstack to use [upgrade_levels]compute=auto so what's a few more days | |
| 19:47:00 | mriedem | i don't actually know what the pass/fail rate is right now | |
| 19:47:32 | mriedem | heh http://grafana.openstack.org/d/QBHIN5Smk/tempest-failure-rate?orgId=1 | |
| 19:48:27 | mriedem | these dashboards are all using old job names | |
| 19:49:55 | dansmith | okay i thought this was spiking in the gate based on the discussion | |
| 19:50:07 | mriedem | not it's definitely not our worst failure | |
| 19:50:19 | dansmith | ack | |
| 19:50:25 | mriedem | http://status.openstack.org/elastic-recheck/#1789484 | |
| 19:50:27 | openstackgerrit | Dan Smith proposed openstack/nova master: Move conductor wait_until_ready() delay before manager init https://review.openstack.org/598353 | |
| 19:50:31 | mriedem | it's not nothing either | |
| 20:05:47 | mriedem | dansmith: +2 | |
| 20:13:31 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [Medium,Triaged] | |
| 20:13:31 | mriedem | fried_rice: more details https://bugs.launchpad.net/nova/+bug/1790204/comments/1 - i expect reschedules would make any solution for that bug even messier | |