| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-01-26 | |||
| 16:18:34 | lyarwood | upload.: nova.exception.ImageUnaccep table: Image 0d4ef007-9ffd-4b8f-9bd2-1d9e3fca5f1b is unacceptable: Image is not raw format | |
| 16:18:48 | lyarwood | ^ that's the error with the original image | |
| 16:18:53 | dansmith | yeah | |
| 16:19:23 | dansmith | is that in a boot or something else? | |
| 16:19:46 | lyarwood | that's on boot | |
| 16:20:10 | ysirndjuro | sean-k-mooney: thx for the sample, i've seen that config option already, but was looking to avoid enabling it on every vm on that node, just specific use case where it's needed for CI/CD to run packer builds, but with on-demand capability versus having an openstack admin configure every vm CI spins up | |
| 16:20:12 | sean-k-mooney | dansmith: by the way i havent forgoten about testing your devstack patch. im going to test it slightly differently then the gate would in that i did a stack on the node, now im going to unstack and stack again use that as my base time and then unstace and stack with your patch | |
| 16:20:15 | dansmith | lyarwood: so nova should ask glance to copy that to the ceph store first, just like it initially did for the image_ref one | |
| 16:20:22 | dansmith | sean-k-mooney: thanks | |
| 16:20:26 | sean-k-mooney | dansmith: that should elimiate any network overhead | |
| 16:21:43 | lyarwood | dansmith: would that be rejected if the disk format isn't RAW? | |
| 16:21:50 | dansmith | lyarwood: ...if it's a normal boot all the same as the original.. that's why I'm wondering what else might be different | |
| 16:21:52 | sean-k-mooney | ysirndjuro: you can kind fo do what you want if you define your own cpu model in the libvirt directorys | |
| 16:22:07 | dansmith | lyarwood: mmmmaybe, what format is the image they created? | |
| 16:22:43 | dansmith | lemme look at the conditions for that again | |
| 16:22:48 | sean-k-mooney | ysirndjuro: if you use cpu_mode=custom then cpu_model=<default model you want>,<nested-virt-modele> then have a special flavor that request vmx i belive that willl work | |
| 16:23:08 | lyarwood | dansmith: I see qcow2 in the request to glance | |
| 16:23:37 | dansmith | ah, hmm | |
| 16:25:24 | sean-k-mooney | ysirndjuro: you should be able to use https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/cpu-model-selection.html to achive your usecase | |
| 16:25:40 | ysirndjuro | literally just dmd u to verify that link haha | |
| 16:25:42 | sean-k-mooney | ysirndjuro: but you will need to define a new cpu model with vmx | |
| 16:25:43 | dansmith | lyarwood: ah, okay, yeah.. we don't even get to the download code because of this in clone: if image_meta.get('disk_format') not in ['raw', 'iso']: | |
| 16:25:43 | lyarwood | dansmith: right it's raising before we get to the copy | |
| 16:25:45 | ysirndjuro | ty, will play with that | |
| 16:25:47 | lyarwood | dansmith: snap | |
| 16:25:57 | dansmith | okay, cool, now I'm a believer :) | |
| 16:26:19 | lyarwood | wonderful | |
| 16:50:40 | bauzas | dansmith: man, you mean a belieber ? ;) | |
| 16:52:09 | dansmith | bauzas: in this *one* thing :) | |
| 16:52:42 | dansmith | unless that's a justin bieber joke, in which case.. no. | |
| 16:53:00 | bauzas | yeah, was just a bad joke | |
| 16:54:20 | stephenfin | bauzas: bad jokes mean you have to go outside? | |
| 16:54:27 | stephenfin | She has you well trained ;) | |
| 16:54:43 | bauzas | stephenfin: indeed :) | |
| 17:11:38 | sean-k-mooney | dansmith: so when the system is up to date and the repos are already there it looks like the async task change does not make much of a difference its better but only slightly http://paste.openstack.org/show/802006/ | |
| 17:11:54 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: add AsyncDeviceDetachEventsHandler https://review.opendev.org/c/openstack/nova/+/772381 | |
| 17:11:55 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: allow querying devices from the persistent domain https://review.opendev.org/c/openstack/nova/+/772383 | |
| 17:11:55 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: libvirt: parse alias out from device config https://review.opendev.org/c/openstack/nova/+/772384 | |
| 17:12:17 | sean-k-mooney | dansmith: i can however try it again with a clean vm and see what teh delta is then | |
| 17:13:01 | sean-k-mooney | looking at the console output too i did not see any real change either so it does not seam to affect the debugablity of things | |
| 17:14:38 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246 | |
| 17:16:03 | dansmith | sean-k-mooney: did you enable async? doesn't look like it | |
| 17:16:12 | dansmith | DEVSTACK_PARALLEL=True | |
| 17:16:15 | sean-k-mooney | dansmith: it looks like its saveing between 5 and 16 second over a 830 seconds | |
| 17:16:20 | sean-k-mooney | oh hehe no | |
| 17:16:34 | sean-k-mooney | well in that case i have a better base line | |
| 17:16:39 | sean-k-mooney | ill let it restack again | |
| 17:16:43 | dansmith | sean-k-mooney: you should see a timing compotent for async_wait once you do | |
| 17:17:07 | sean-k-mooney | ok i was seeign async task prinnted but i guess it was blocking | |
| 17:17:29 | dansmith | yeah it'll still print those things but wait instead of spawn | |
| 17:18:33 | dansmith | sean-k-mooney: how many vcpus on the vm you're using? | |
| 17:18:41 | sean-k-mooney | 8 | |
| 17:18:45 | dansmith | ack | |
| 17:19:00 | dansmith | there are some things that are actually cpu bound, but if you only had one it wouldn't help | |
| 17:19:05 | dansmith | the db_syncs for example | |
| 17:19:08 | sean-k-mooney | this is like a upstream node so 8 cores and 8GB of ram although it has 2 numa nodes and nested vert | |
| 17:19:48 | sean-k-mooney | ya looking at load | |
| 17:20:15 | sean-k-mooney | 15 min avgerage is like 0.76 | |
| 17:20:30 | sean-k-mooney | it was using 1-2 cores max while running before | |
| 17:20:45 | sean-k-mooney | it spiked a littel when doing the db migrations | |
| 17:21:04 | sean-k-mooney | so that is where the extra cores definetly help | |
| 17:21:15 | dansmith | yeah, on one of my machines it went from about 0.5 to 2.0 | |
| 17:21:41 | dansmith | not for the whole thing but for good portions of it | |
| 17:21:56 | sean-k-mooney | ok its stacking again ill let you know in 15 mins or so | |
| 17:22:50 | sean-k-mooney | 13-15 mins is what i normally expect for this vm so if its better or the same then the patch is proably an improvment | |
| 17:22:59 | dansmith | cool | |
| 17:31:18 | sean-k-mooney | and done | |
| 17:31:25 | sean-k-mooney | it hit 3.78 there for a bit | |
| 17:32:09 | sean-k-mooney | http://paste.openstack.org/show/802007/ | |
| 17:32:48 | sean-k-mooney | so that is much better | |
| 17:33:14 | dansmith | 30% | |
| 17:33:17 | dansmith | you're welcome :) | |
| 17:33:43 | sean-k-mooney | what me to test this on clean vms to see what impact it has then | |
| 17:33:51 | sean-k-mooney | this is how i normallyuse devstack however | |
| 17:34:05 | sean-k-mooney | so 30% on my normal uscase it very good | |
| 17:34:14 | dansmith | well, the gate preloads the project git dirs, so it's closer to what you just tested | |
| 17:34:27 | sean-k-mooney | ya it is | |
| 17:34:30 | dansmith | there is more fat to be cut, by the way | |
| 17:34:37 | dansmith | I've optimized some of stack.sh and nova, | |
| 17:34:43 | sean-k-mooney | and they cache the pip and apt stuff close to the vms too | |
| 17:34:46 | dansmith | but there are things neutron does that we can parallellize I think | |
| 17:34:54 | dansmith | oh I did a bunch of keystone things too | |
| 17:35:03 | sean-k-mooney | did you look at the plugins | |
| 17:35:11 | dansmith | being able to stack in 6 minutes is a major life improvement to me | |
| 17:35:31 | sean-k-mooney | i would have to run in offlinemode to do that | |
| 17:35:45 | dansmith | only tempest, which I do some of in parallel, but we regen the venv several times for reasons I don't understand, but we could probably improve there too | |
| 17:36:03 | sean-k-mooney | ya i have seen that | |
| 17:36:20 | sean-k-mooney | its not clear to me why either | |
| 17:36:33 | sean-k-mooney | the local pip cache helps but its not perfect | |
| 17:37:45 | sean-k-mooney | dansmith: what was your normal stack time before out of interest | |
| 17:38:27 | dansmith | sean-k-mooney: on my usual stripped-down config, it's 513s serialized, 402s parallell | |
| 17:39:18 | sean-k-mooney | ya not bad still a nice improvment | |
| 17:39:59 | sean-k-mooney | that said even a 15min run is better then ooo | |
| 17:40:57 | dansmith | mm yeah :) | |
| 17:42:58 | sean-k-mooney | dansmith: apparently our downstreeam upgrade jobs take 11hours currently. | |
| 17:43:15 | dansmith | yeah that's pretty sadface | |
| 17:43:20 | sean-k-mooney | i just can even comprehend debuging those if they fail | |
| 17:43:39 | dansmith | yeah :/ | |
| 17:43:45 | dansmith | sean-k-mooney: if you could comment on that async patch with your results and environment, I'd appreciate it | |
| 17:44:44 | sean-k-mooney | yep i need to prep a fedra 32 vm for other things anyway today so ill do a fresh install run on that and comment when its done with the details | |