Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-26
16:00:15 stephenfin agreed
16:00:25 stephenfin we'll see what gmann thinks
16:05:52 dansmith lyarwood: question for you here: https://review.opendev.org/c/openstack/tempest/+/771958
16:06:00 dansmith not sure I understand what actually changed
16:07:41 gmann stephenfin: lbragstad ack, I will check config things on policy init locally as config setting come into pic when test run and start the policy enforcement instead of policy__init__. but I am ok with the current approach.
16:07:50 gmann will review those today
16:08:49 gmann config things are similar to what we do in nova. init policy first and then switch the config flag for enforce scope
16:10:47 ysirndjuro Is there a way to bake "cpu mode='host-passthrough'" flag inside an image or flavor for nested virt support in an on demand VM, so far the only docs i've found have been from devstack and involve direct virsh edit to an existing VM.
16:12:46 ysirndjuro or do I just slap 'cpu_mode=host-passthrough' in nova.conf and call it a day
16:13:04 sean-k-mooney ysirndjuro:no that is intetniolly not supported
16:13:22 lyarwood dansmith: the tests were just creating a fresh image instead of reusing CONF.compute.image_ref, I assumed that borked caching in the job and resulted in us hitting the never_download_image_if_on_rbd check
16:13:42 sean-k-mooney users cannot set the cpu_mode
16:13:49 sean-k-mooney so we dont support it in the image
16:13:51 dansmith lyarwood: I don't think so
16:14:20 dansmith lyarwood: in that job, we upload the image initially to the file backend in glance, and make nova auto copy it to ceph on the first boot
16:14:37 dansmith lyarwood: so just creating another image in the file repo should do the same thing
16:14:42 sean-k-mooney and we dont support setting the mode in the flavor since its specific to libvirt and we dont want to encurage a different flavor per cpu model
16:15:31 sean-k-mooney ysirndjuro: so either set it in your nova.conf via your local.conf
16:15:40 dansmith lyarwood: I'm thinking there's some cinder interaction that tries to do something with that image before it has been moved to ceph, but then your change won't actually make any difference if that test happens to run before any other job that just boots an instance from the normal image
16:15:46 sean-k-mooney or modify it and restart n-cpu
16:15:56 ysirndjuro sean-k-mooney: thx, yeah figured as much there, was leaning towards a flavor config, feelsbad
16:16:55 sean-k-mooney ysirndjuro: this is more or less my default local.conf http://paste.openstack.org/show/801996/
16:16:59 lyarwood dansmith: pretty sure the volume is empty
16:17:02 lyarwood dansmith: let me check
16:17:19 sean-k-mooney ysirndjuro: it has it set so you can copy that
16:17:57 lyarwood dansmith: yeah the volume shouldn't reference any image ref
16:18:33 lyarwood 49942 Jan 20 22:28:53.068420 ubuntu-focal-rax-iad-0022637570 nova-compute[94729]: WARNING nova.virt.libvirt.driver [None req-3a8ebd8e-b22a-446f-ad35-16b12e7a1182 tempest-TestEncryptedCinderVolumes-88193216 tempest-TestEncryptedCinderVolumes-88193216] Image 0d4ef007-9ffd-4b8f-9bd2-1d9e3fca5f1b is not on my ceph and [workarounds]/never_download_image_if_on_rbd=True; refusing to fetch and
16:18:33 dansmith lyarwood: okay well then I don't see what your change would actually do differently (in the case where that test happens to run first
16:18:34 lyarwood upload.: nova.exception.ImageUnaccep table: Image 0d4ef007-9ffd-4b8f-9bd2-1d9e3fca5f1b is unacceptable: Image is not raw format
16:18:48 lyarwood ^ that's the error with the original image
16:18:53 dansmith yeah
16:19:23 dansmith is that in a boot or something else?
16:19:46 lyarwood that's on boot
16:20:10 ysirndjuro sean-k-mooney: thx for the sample, i've seen that config option already, but was looking to avoid enabling it on every vm on that node, just specific use case where it's needed for CI/CD to run packer builds, but with on-demand capability versus having an openstack admin configure every vm CI spins up
16:20:12 sean-k-mooney dansmith: by the way i havent forgoten about testing your devstack patch. im going to test it slightly differently then the gate would in that i did a stack on the node, now im going to unstack and stack again use that as my base time and then unstace and stack with your patch
16:20:15 dansmith lyarwood: so nova should ask glance to copy that to the ceph store first, just like it initially did for the image_ref one
16:20:22 dansmith sean-k-mooney: thanks
16:20:26 sean-k-mooney dansmith: that should elimiate any network overhead
16:21:43 lyarwood dansmith: would that be rejected if the disk format isn't RAW?
16:21:50 dansmith lyarwood: ...if it's a normal boot all the same as the original.. that's why I'm wondering what else might be different
16:21:52 sean-k-mooney ysirndjuro: you can kind fo do what you want if you define your own cpu model in the libvirt directorys
16:22:07 dansmith lyarwood: mmmmaybe, what format is the image they created?
16:22:43 dansmith lemme look at the conditions for that again
16:22:48 sean-k-mooney ysirndjuro: if you use cpu_mode=custom then cpu_model=<default model you want>,<nested-virt-modele> then have a special flavor that request vmx i belive that willl work
16:23:08 lyarwood dansmith: I see qcow2 in the request to glance
16:23:37 dansmith ah, hmm
16:25:24 sean-k-mooney ysirndjuro: you should be able to use https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/cpu-model-selection.html to achive your usecase
16:25:40 ysirndjuro literally just dmd u to verify that link haha
16:25:42 sean-k-mooney ysirndjuro: but you will need to define a new cpu model with vmx
16:25:43 lyarwood dansmith: right it's raising before we get to the copy
16:25:43 dansmith lyarwood: ah, okay, yeah.. we don't even get to the download code because of this in clone: if image_meta.get('disk_format') not in ['raw', 'iso']:
16:25:45 ysirndjuro ty, will play with that
16:25:47 lyarwood dansmith: snap
16:25:57 dansmith okay, cool, now I'm a believer :)
16:26:19 lyarwood wonderful
16:50:40 bauzas dansmith: man, you mean a belieber ? ;)
16:52:09 dansmith bauzas: in this *one* thing :)
16:52:42 dansmith unless that's a justin bieber joke, in which case.. no.
16:53:00 bauzas yeah, was just a bad joke
16:54:20 stephenfin bauzas: bad jokes mean you have to go outside?
16:54:27 stephenfin She has you well trained ;)
16:54:43 bauzas stephenfin: indeed :)
17:11:38 sean-k-mooney dansmith: so when the system is up to date and the repos are already there it looks like the async task change does not make much of a difference its better but only slightly http://paste.openstack.org/show/802006/
17:11:54 openstackgerrit Balazs Gibizer proposed openstack/nova master: libvirt: add AsyncDeviceDetachEventsHandler https://review.opendev.org/c/openstack/nova/+/772381
17:11:55 openstackgerrit Balazs Gibizer proposed openstack/nova master: libvirt: parse alias out from device config https://review.opendev.org/c/openstack/nova/+/772384
17:11:55 openstackgerrit Balazs Gibizer proposed openstack/nova master: libvirt: allow querying devices from the persistent domain https://review.opendev.org/c/openstack/nova/+/772383
17:12:17 sean-k-mooney dansmith: i can however try it again with a clean vm and see what teh delta is then
17:13:01 sean-k-mooney looking at the console output too i did not see any real change either so it does not seam to affect the debugablity of things
17:14:38 openstackgerrit Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246
17:16:03 dansmith sean-k-mooney: did you enable async? doesn't look like it
17:16:12 dansmith DEVSTACK_PARALLEL=True
17:16:15 sean-k-mooney dansmith: it looks like its saveing between 5 and 16 second over a 830 seconds
17:16:20 sean-k-mooney oh hehe no
17:16:34 sean-k-mooney well in that case i have a better base line
17:16:39 sean-k-mooney ill let it restack again
17:16:43 dansmith sean-k-mooney: you should see a timing compotent for async_wait once you do
17:17:07 sean-k-mooney ok i was seeign async task prinnted but i guess it was blocking
17:17:29 dansmith yeah it'll still print those things but wait instead of spawn
17:18:33 dansmith sean-k-mooney: how many vcpus on the vm you're using?
17:18:41 sean-k-mooney 8
17:18:45 dansmith ack
17:19:00 dansmith there are some things that are actually cpu bound, but if you only had one it wouldn't help
17:19:05 dansmith the db_syncs for example
17:19:08 sean-k-mooney this is like a upstream node so 8 cores and 8GB of ram although it has 2 numa nodes and nested vert
17:19:48 sean-k-mooney ya looking at load
17:20:15 sean-k-mooney 15 min avgerage is like 0.76
17:20:30 sean-k-mooney it was using 1-2 cores max while running before
17:20:45 sean-k-mooney it spiked a littel when doing the db migrations
17:21:04 sean-k-mooney so that is where the extra cores definetly help
17:21:15 dansmith yeah, on one of my machines it went from about 0.5 to 2.0
17:21:41 dansmith not for the whole thing but for good portions of it
17:21:56 sean-k-mooney ok its stacking again ill let you know in 15 mins or so
17:22:50 sean-k-mooney 13-15 mins is what i normally expect for this vm so if its better or the same then the patch is proably an improvment
17:22:59 dansmith cool
17:31:18 sean-k-mooney and done
17:31:25 sean-k-mooney it hit 3.78 there for a bit
17:32:09 sean-k-mooney http://paste.openstack.org/show/802007/
17:32:48 sean-k-mooney so that is much better
17:33:14 dansmith 30%
17:33:17 dansmith you're welcome :)

Earlier   Later