Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-26
16:25:47 lyarwood dansmith: snap
16:25:57 dansmith okay, cool, now I'm a believer :)
16:26:19 lyarwood wonderful
16:50:40 bauzas dansmith: man, you mean a belieber ? ;)
16:52:09 dansmith bauzas: in this *one* thing :)
16:52:42 dansmith unless that's a justin bieber joke, in which case.. no.
16:53:00 bauzas yeah, was just a bad joke
16:54:20 stephenfin bauzas: bad jokes mean you have to go outside?
16:54:27 stephenfin She has you well trained ;)
16:54:43 bauzas stephenfin: indeed :)
17:11:38 sean-k-mooney dansmith: so when the system is up to date and the repos are already there it looks like the async task change does not make much of a difference its better but only slightly http://paste.openstack.org/show/802006/
17:11:54 openstackgerrit Balazs Gibizer proposed openstack/nova master: libvirt: add AsyncDeviceDetachEventsHandler https://review.opendev.org/c/openstack/nova/+/772381
17:11:55 openstackgerrit Balazs Gibizer proposed openstack/nova master: libvirt: allow querying devices from the persistent domain https://review.opendev.org/c/openstack/nova/+/772383
17:11:55 openstackgerrit Balazs Gibizer proposed openstack/nova master: libvirt: parse alias out from device config https://review.opendev.org/c/openstack/nova/+/772384
17:12:17 sean-k-mooney dansmith: i can however try it again with a clean vm and see what teh delta is then
17:13:01 sean-k-mooney looking at the console output too i did not see any real change either so it does not seam to affect the debugablity of things
17:14:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246
17:16:03 dansmith sean-k-mooney: did you enable async? doesn't look like it
17:16:12 dansmith DEVSTACK_PARALLEL=True
17:16:15 sean-k-mooney dansmith: it looks like its saveing between 5 and 16 second over a 830 seconds
17:16:20 sean-k-mooney oh hehe no
17:16:34 sean-k-mooney well in that case i have a better base line
17:16:39 sean-k-mooney ill let it restack again
17:16:43 dansmith sean-k-mooney: you should see a timing compotent for async_wait once you do
17:17:07 sean-k-mooney ok i was seeign async task prinnted but i guess it was blocking
17:17:29 dansmith yeah it'll still print those things but wait instead of spawn
17:18:33 dansmith sean-k-mooney: how many vcpus on the vm you're using?
17:18:41 sean-k-mooney 8
17:18:45 dansmith ack
17:19:00 dansmith there are some things that are actually cpu bound, but if you only had one it wouldn't help
17:19:05 dansmith the db_syncs for example
17:19:08 sean-k-mooney this is like a upstream node so 8 cores and 8GB of ram although it has 2 numa nodes and nested vert
17:19:48 sean-k-mooney ya looking at load
17:20:15 sean-k-mooney 15 min avgerage is like 0.76
17:20:30 sean-k-mooney it was using 1-2 cores max while running before
17:20:45 sean-k-mooney it spiked a littel when doing the db migrations
17:21:04 sean-k-mooney so that is where the extra cores definetly help
17:21:15 dansmith yeah, on one of my machines it went from about 0.5 to 2.0
17:21:41 dansmith not for the whole thing but for good portions of it
17:21:56 sean-k-mooney ok its stacking again ill let you know in 15 mins or so
17:22:50 sean-k-mooney 13-15 mins is what i normally expect for this vm so if its better or the same then the patch is proably an improvment
17:22:59 dansmith cool
17:31:18 sean-k-mooney and done
17:31:25 sean-k-mooney it hit 3.78 there for a bit
17:32:09 sean-k-mooney http://paste.openstack.org/show/802007/
17:32:48 sean-k-mooney so that is much better
17:33:14 dansmith 30%
17:33:17 dansmith you're welcome :)
17:33:43 sean-k-mooney what me to test this on clean vms to see what impact it has then
17:33:51 sean-k-mooney this is how i normallyuse devstack however
17:34:05 sean-k-mooney so 30% on my normal uscase it very good
17:34:14 dansmith well, the gate preloads the project git dirs, so it's closer to what you just tested
17:34:27 sean-k-mooney ya it is
17:34:30 dansmith there is more fat to be cut, by the way
17:34:37 dansmith I've optimized some of stack.sh and nova,
17:34:43 sean-k-mooney and they cache the pip and apt stuff close to the vms too
17:34:46 dansmith but there are things neutron does that we can parallellize I think
17:34:54 dansmith oh I did a bunch of keystone things too
17:35:03 sean-k-mooney did you look at the plugins
17:35:11 dansmith being able to stack in 6 minutes is a major life improvement to me
17:35:31 sean-k-mooney i would have to run in offlinemode to do that
17:35:45 dansmith only tempest, which I do some of in parallel, but we regen the venv several times for reasons I don't understand, but we could probably improve there too
17:36:03 sean-k-mooney ya i have seen that
17:36:20 sean-k-mooney its not clear to me why either
17:36:33 sean-k-mooney the local pip cache helps but its not perfect
17:37:45 sean-k-mooney dansmith: what was your normal stack time before out of interest
17:38:27 dansmith sean-k-mooney: on my usual stripped-down config, it's 513s serialized, 402s parallell
17:39:18 sean-k-mooney ya not bad still a nice improvment
17:39:59 sean-k-mooney that said even a 15min run is better then ooo
17:40:57 dansmith mm yeah :)
17:42:58 sean-k-mooney dansmith: apparently our downstreeam upgrade jobs take 11hours currently.
17:43:15 dansmith yeah that's pretty sadface
17:43:20 sean-k-mooney i just can even comprehend debuging those if they fail
17:43:39 dansmith yeah :/
17:43:45 dansmith sean-k-mooney: if you could comment on that async patch with your results and environment, I'd appreciate it
17:44:44 sean-k-mooney yep i need to prep a fedra 32 vm for other things anyway today so ill do a fresh install run on that and comment when its done with the details
17:44:58 dansmith okay thanks
17:45:10 dansmith gmann: results from sean-k-mooney btw :) ^
17:45:34 dansmith tl;dr his VMs are closer to upstream gate and he got 30% improvement
17:46:44 sean-k-mooney i can test this in my ci later in the week proably too need to do some maintance on it before i do but i have some other things to rebase and backport first so wont get to it for a while
17:47:21 dansmith sean-k-mooney: was that a fairly fat devstack config? like all the normal services?
17:48:54 sean-k-mooney yes an no ill past bin it
17:49:08 sean-k-mooney http://paste.openstack.org/show/802010/
17:49:19 sean-k-mooney no swift or heat but i hav ecinder and horizon
17:49:58 gmann dansmith: sean-k-mooney nice. are you running it on fresh machine with/wihtout devstack-parallel or with/without unstack/stack ?
17:50:16 sean-k-mooney so nova,neutron,placement,cinder,glance,horizon,tempest with ml2/ovs
17:50:39 dansmith gmann: when I test, I run it with/without parallel on a "warmed up" machine, i.e. where /opt/stack already has the projects cloned, like gate
17:50:55 dansmith we could definitely parallelize the cloning of all the projects too
17:51:01 sean-k-mooney gmann: basically the same for my test
17:51:07 sean-k-mooney gmann: i stacked then unstack
17:51:09 dansmith sean-k-mooney: ack, okay mine is no cinder or horizin
17:51:18 dansmith right, stack/unstack/stack
17:51:28 sean-k-mooney then stacked again to get the baseline time unstackted. and stacked in parallel mode
17:51:49 sean-k-mooney so all the pacakge and repos should be cached or clonned
17:51:52 gmann yeah clone is not something to count in this
17:54:31 sean-k-mooney i did not have OFFLINE=True which i can do that would disable all package installs and clones but it should not really be a factor in my current testing
17:54:42 sean-k-mooney ill let ye both know how the fresh install goes
17:55:15 dansmith once the system is warmed up I don't think that would do much anyway, right?
17:56:11 sean-k-mooney it will prevent it even trying to do apt/dnf install i dont think it really affect pip however
17:56:18 sean-k-mooney so when its primmed no not really

Earlier   Later