| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-26 | |||
| 17:19:48 | sean-k-mooney | ya looking at load | |
| 17:20:15 | sean-k-mooney | 15 min avgerage is like 0.76 | |
| 17:20:30 | sean-k-mooney | it was using 1-2 cores max while running before | |
| 17:20:45 | sean-k-mooney | it spiked a littel when doing the db migrations | |
| 17:21:04 | sean-k-mooney | so that is where the extra cores definetly help | |
| 17:21:15 | dansmith | yeah, on one of my machines it went from about 0.5 to 2.0 | |
| 17:21:41 | dansmith | not for the whole thing but for good portions of it | |
| 17:21:56 | sean-k-mooney | ok its stacking again ill let you know in 15 mins or so | |
| 17:22:50 | sean-k-mooney | 13-15 mins is what i normally expect for this vm so if its better or the same then the patch is proably an improvment | |
| 17:22:59 | dansmith | cool | |
| 17:31:18 | sean-k-mooney | and done | |
| 17:31:25 | sean-k-mooney | it hit 3.78 there for a bit | |
| 17:32:09 | sean-k-mooney | http://paste.openstack.org/show/802007/ | |
| 17:32:48 | sean-k-mooney | so that is much better | |
| 17:33:14 | dansmith | 30% | |
| 17:33:17 | dansmith | you're welcome :) | |
| 17:33:43 | sean-k-mooney | what me to test this on clean vms to see what impact it has then | |
| 17:33:51 | sean-k-mooney | this is how i normallyuse devstack however | |
| 17:34:05 | sean-k-mooney | so 30% on my normal uscase it very good | |
| 17:34:14 | dansmith | well, the gate preloads the project git dirs, so it's closer to what you just tested | |
| 17:34:27 | sean-k-mooney | ya it is | |
| 17:34:30 | dansmith | there is more fat to be cut, by the way | |
| 17:34:37 | dansmith | I've optimized some of stack.sh and nova, | |
| 17:34:43 | sean-k-mooney | and they cache the pip and apt stuff close to the vms too | |
| 17:34:46 | dansmith | but there are things neutron does that we can parallellize I think | |
| 17:34:54 | dansmith | oh I did a bunch of keystone things too | |
| 17:35:03 | sean-k-mooney | did you look at the plugins | |
| 17:35:11 | dansmith | being able to stack in 6 minutes is a major life improvement to me | |
| 17:35:31 | sean-k-mooney | i would have to run in offlinemode to do that | |
| 17:35:45 | dansmith | only tempest, which I do some of in parallel, but we regen the venv several times for reasons I don't understand, but we could probably improve there too | |
| 17:36:03 | sean-k-mooney | ya i have seen that | |
| 17:36:20 | sean-k-mooney | its not clear to me why either | |
| 17:36:33 | sean-k-mooney | the local pip cache helps but its not perfect | |
| 17:37:45 | sean-k-mooney | dansmith: what was your normal stack time before out of interest | |
| 17:38:27 | dansmith | sean-k-mooney: on my usual stripped-down config, it's 513s serialized, 402s parallell | |
| 17:39:18 | sean-k-mooney | ya not bad still a nice improvment | |
| 17:39:59 | sean-k-mooney | that said even a 15min run is better then ooo | |
| 17:40:57 | dansmith | mm yeah :) | |
| 17:42:58 | sean-k-mooney | dansmith: apparently our downstreeam upgrade jobs take 11hours currently. | |
| 17:43:15 | dansmith | yeah that's pretty sadface | |
| 17:43:20 | sean-k-mooney | i just can even comprehend debuging those if they fail | |
| 17:43:39 | dansmith | yeah :/ | |
| 17:43:45 | dansmith | sean-k-mooney: if you could comment on that async patch with your results and environment, I'd appreciate it | |
| 17:44:44 | sean-k-mooney | yep i need to prep a fedra 32 vm for other things anyway today so ill do a fresh install run on that and comment when its done with the details | |
| 17:44:58 | dansmith | okay thanks | |
| 17:45:10 | dansmith | gmann: results from sean-k-mooney btw :) ^ | |
| 17:45:34 | dansmith | tl;dr his VMs are closer to upstream gate and he got 30% improvement | |
| 17:46:44 | sean-k-mooney | i can test this in my ci later in the week proably too need to do some maintance on it before i do but i have some other things to rebase and backport first so wont get to it for a while | |
| 17:47:21 | dansmith | sean-k-mooney: was that a fairly fat devstack config? like all the normal services? | |
| 17:48:54 | sean-k-mooney | yes an no ill past bin it | |
| 17:49:08 | sean-k-mooney | http://paste.openstack.org/show/802010/ | |
| 17:49:19 | sean-k-mooney | no swift or heat but i hav ecinder and horizon | |
| 17:49:58 | gmann | dansmith: sean-k-mooney nice. are you running it on fresh machine with/wihtout devstack-parallel or with/without unstack/stack ? | |
| 17:50:16 | sean-k-mooney | so nova,neutron,placement,cinder,glance,horizon,tempest with ml2/ovs | |
| 17:50:39 | dansmith | gmann: when I test, I run it with/without parallel on a "warmed up" machine, i.e. where /opt/stack already has the projects cloned, like gate | |
| 17:50:55 | dansmith | we could definitely parallelize the cloning of all the projects too | |
| 17:51:01 | sean-k-mooney | gmann: basically the same for my test | |
| 17:51:07 | sean-k-mooney | gmann: i stacked then unstack | |
| 17:51:09 | dansmith | sean-k-mooney: ack, okay mine is no cinder or horizin | |
| 17:51:18 | dansmith | right, stack/unstack/stack | |
| 17:51:28 | sean-k-mooney | then stacked again to get the baseline time unstackted. and stacked in parallel mode | |
| 17:51:49 | sean-k-mooney | so all the pacakge and repos should be cached or clonned | |
| 17:51:52 | gmann | yeah clone is not something to count in this | |
| 17:54:31 | sean-k-mooney | i did not have OFFLINE=True which i can do that would disable all package installs and clones but it should not really be a factor in my current testing | |
| 17:54:42 | sean-k-mooney | ill let ye both know how the fresh install goes | |
| 17:55:15 | dansmith | once the system is warmed up I don't think that would do much anyway, right? | |
| 17:56:11 | sean-k-mooney | it will prevent it even trying to do apt/dnf install i dont think it really affect pip however | |
| 17:56:18 | sean-k-mooney | so when its primmed no not really | |
| 17:56:33 | dansmith | ack, but apt should mostly just say "yeah already got that" | |
| 17:56:39 | dansmith | minor improvement maybe but nothing major | |
| 17:56:43 | dansmith | in the noise I expect | |
| 17:56:47 | sean-k-mooney | yep it just has to check the index is up to date | |
| 17:57:41 | sean-k-mooney | also i have gigabit networking at home so its going to be pretty fast considing i sit less then 2 miles form where the irish henet mirrors are hosted | |
| 17:58:04 | sean-k-mooney | there hosted in the university i live beside | |
| 17:59:42 | sean-k-mooney | 64 bytes from ftp-node-prod-bl-01.heanet.ie (193.1.193.10): icmp_seq=1 ttl=58 time=7.50 ms | |
| 18:01:13 | sean-k-mooney | vs 64 bytes from 172.20.4.244: icmp_seq=15 ttl=62 time=3.41 ms to my vm :) | |
| 18:58:12 | sean-k-mooney | looks like my clean fedora run is going to fail because the cirros download is hanging | |
| 18:58:29 | sean-k-mooney | so ill just post my old results and node details to the devstack review | |
| 19:12:21 | openstackgerrit | Merged openstack/nova master: Move query param generation to RequestGroup o.vo https://review.opendev.org/c/openstack/nova/+/756894 | |
| 19:33:01 | gmann | stephenfin: lbragstad should not placement_policy.reset() take care of cleaning all default or registered rules? https://review.opendev.org/c/openstack/placement/+/772508/1/placement/tests/unit/policy_fixture.py#36 | |
| 19:33:39 | gmann | I am not completely understanding how it is causing the issue with placement_policy.reset() in test. though i agree on fixing it on oslo policy side | |
| 19:34:05 | gmann | or something i am missing here | |
| 19:58:59 | dansmith | sean-k-mooney: thanks | |
| 20:15:19 | gmann | lbragstad: before I proceed on other patches in that series, one comment about SYSTEM_ADMIN https://review.opendev.org/c/openstack/placement/+/760240/21/placement/policies/base.py#20 | |
| 20:15:56 | gmann | I think we do not need SYSTEM_ADMIN in placement as such | |
| 20:16:40 | gmann | and this way is better and avoid rule deprecation - https://review.opendev.org/c/openstack/placement/+/760235/10/placement/policies/aggregate.py#52 | |
| 20:45:07 | lbragstad | gmann i think that's true with the exception of the usage api since it supports project and system scope? | |
| 20:46:37 | lbragstad | actually - not if enforce_scope=True | |
| 20:47:21 | gmann | lbragstad: yeah usage API use only reader right | |
| 20:48:06 | gmann | if enforce_scope=True then scope_type will care as existing scope is system for admin API | |
| 20:51:37 | gmann | only use case i see is if enforce_scope=False , enforce_new_defaults=True but technically there is no new default for admin APIs its admin only right | |
| 21:12:39 | openstackgerrit | melanie witt proposed openstack/nova stable/ussuri: compute: Lock by instance.uuid lock during swap_volume https://review.opendev.org/c/openstack/nova/+/758732 | |
| 23:16:07 | openstackgerrit | Merged openstack/nova master: db: Compact Juno database migration https://review.opendev.org/c/openstack/nova/+/758395 | |
| 23:56:40 | sean-k-mooney | lbragstad: gmann im going to stop reviewing the palcemnt series until ye have a chance ot responed regarding defineing SYTEM_ADMIN as a dict and preferably in oslo policy eventually | |
| 23:58:00 | sean-k-mooney | i really think that SYSTEM_ADMIN should be a common shared defintion across all poject and not something that can vary between cloud or services. | |
| 23:59:19 | gmann | sean-k-mooney: lbragstad stephenfin replied on https://review.opendev.org/c/openstack/placement/+/760240/21/placement/policies/base.py#20 | |
| 23:59:41 | gmann | sean-k-mooney: we can move to common place but there are few challenge in doing that | |
| #openstack-nova - 2021-01-27 | |||
| 00:00:44 | gmann | https://review.opendev.org/c/openstack/oslo.policy/+/766536 | |
| 00:02:03 | gmann | if we want to support 4th scenario I mentioned in that review then yes we can have otherwise there is no change with SYSTEM_ADMIN and what we have currently | |
| 00:03:14 | sean-k-mooney | well i was suggesting that at a minium we define SYSTEM_ADMIN as | |