Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-10-18
15:43:13 bauzas netcalol
16:13:26 bauzas sean-k-mooney: fwiw, read_sys only reads files :) https://github.com/openstack/nova/blob/master/nova/filesystem.py#L28
16:13:42 bauzas sean-k-mooney: and /sys/devices/virtual/mtty is a dir :)
16:14:27 bauzas I can provide some helper method that would wrap os.path.isdir() but would you nack if I was just calling os.path.isdir() straight in the driver ?
16:14:39 bauzas or .exists()
16:21:36 colby__ noonedeadpunk: Thanks Ill check on that. Did not realize the tunnel was dropped. Was this a change on xena to yoga? Everything worked fine on xena.
16:23:05 sean-k-mooney bauzas: good point on it being a dir i guess you can just add a new helper ya in the same module
16:23:29 sean-k-mooney or just call os.path.isdir() but it would be ok ot put in the filesystem module
16:23:45 sean-k-mooney an exist check is enough
16:23:53 bauzas sean-k-mooney: well, if the helper method only wraps isdir, that sounds to me a bit unnecessary
16:24:10 sean-k-mooney so i would go with exist because it might actully be a simlink
16:24:13 colby__ I have another question. It appears that the amount of local disk available changed in yoga. It appears to be only considering the actual local disk now (without ceph). But instances that use a ceph backed volume to boot are still counting as local disk usage. Was this intentional? Its causing no resources available issues for me. Im working on changing the disk allocation ratio to get around it.
16:24:27 bauzas yeah, agreed on exists(), semantically better
16:24:41 bauzas but if I'm just returning the boolean, meh
16:25:00 bauzas I can pretend to return a FileNotFound exception if exists says no
16:25:12 bauzas at least that would justify the helper
16:25:46 sean-k-mooney colby__: boot form volume instance shoudl use flavor with disk_gb = 0
16:26:26 sean-k-mooney we allow boot form volue instnace with flavor that dont set it to 0 but it is only for legacy/upgrade reasons
16:26:53 sean-k-mooney if you mena ceph volume via image_type=rbd
16:26:59 sean-k-mooney then use flavor as normal
16:27:08 sean-k-mooney with disk_gb set to the correct size
16:27:52 sean-k-mooney in yoga and master for that mater we report local_gb based on the rbd pool size for every instnace as far as i remember
16:31:06 colby__ yea unfortunately our flavors have disk size set even though they all use ceph boot volumes. Everythign we use is in ceph (images/ephemeral/volumes). But seems there is no way around the already set flavor disk size then. From the hypervisor screen its seems to be only showing the hypervisor boot disk volume for local disk
16:31:51 colby__ Im in the process of upgrading yoga and trying to live migrate instances to the already upgraded machines and running into this issue
16:32:07 colby__ is there a way to override the disk size of a running instance flavor?
16:32:34 sean-k-mooney colby__: are you using images_type=rbd
16:32:38 sean-k-mooney or cinder volumes
16:33:24 colby__ cinder volumes
16:33:47 sean-k-mooney adn your using images_type=qcow
16:33:52 sean-k-mooney or raw or flat
16:34:14 colby__ we did not set it so its currently default
16:34:32 sean-k-mooney ok that woudl be defcault which for libvirt is qcow
16:35:09 sean-k-mooney imporanly in that config then the expect bevioru is to report local_disk_gb as the free space on the compute node partion where /var/lib/nova is
16:35:20 sean-k-mooney and that was the ame behvior in older release
16:35:33 colby__ ok so set images_type=rbd then?
16:35:37 sean-k-mooney no
16:35:51 sean-k-mooney that means use cpeh for root disk
16:35:56 sean-k-mooney even with out ceinder
16:36:14 colby__ ah sorry I misunderstood. so raw then?
16:36:21 sean-k-mooney no dont change it
16:36:35 sean-k-mooney nova is working as it should form a compute config point of view
16:36:47 colby__ ok
16:37:01 sean-k-mooney i need to check somehting but i alo need to join a 1:1 with my manager
16:37:12 colby__ ok no problem. Thanks for the help
16:37:42 sean-k-mooney did you used to set allcoation raitio by aggreates prehaps?
16:39:56 dansmith ugh, feels like we're back to gate suckage.. some trivial patches are in recheck hell
16:41:03 colby__ sean-k-mooney: we never set anything for disk allocation
17:03:28 colby__ sean-k-mooney: I was able to increase the disk allocation ratio but now seeing this: nova.exception.InvalidSharedStorage: <host> is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks. This is a boot from volume ceph instance
17:04:04 colby__ is this all due to the disk_gb of the flavors?
17:08:19 colby__ Im migrating from a xena hypervisor to a yoga and seeing this issue
17:21:38 sean-k-mooney colby__: so partly. but you shoudl have had the same behavior on xena and yoga i think
17:22:01 sean-k-mooney colby__: so you should not be passing --shared-srotage wehn live migrating
17:22:32 colby__ I was just using the GUI for the live migration but can switch to the cli
17:22:36 sean-k-mooney you should be using a newer microvstion where that defautl to auto
17:23:10 sean-k-mooney so for boot form volume guest you should not tick shared storage
17:23:43 sean-k-mooney shared storage for that option really means is /var/lib/nova on nfs
17:23:58 sean-k-mooney so are you using images_type=rbd
17:24:17 colby__ no
17:24:23 colby__ just set to default
17:24:30 sean-k-mooney i missed typed
17:25:11 sean-k-mooney what i ment is you only say "shared_stroage=true" if you the vm is booted on a shared file cycle without using cinder volumes
17:25:21 sean-k-mooney so that is not your case
17:25:35 colby__ also there is no check box for shared storage just "block migration" and "disk overcommit" on gui live migration
17:25:56 sean-k-mooney right so your using horizon yes?
17:26:22 sean-k-mooney it likely would be better to do live migration form the openstack client
17:26:32 colby__ yep will try that
17:28:38 sean-k-mooney so in mitaka 2.25 we added auto
17:28:40 sean-k-mooney https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-mitaka
17:32:14 melwitt stephenfin: I wondered if you might be able to shed some light on this bug report regarding an alembic migration https://bugs.launchpad.net/nova/+bug/2008716 they report a failure to drop columns in the build_requests table going from xena -> yoga. yet this doesn't fail in the grenade job for xena -> yoga, so I'm confused if there is really a bug?
17:40:35 sean-k-mooney melwitt: i think gibi ^ looked at that before
17:40:38 colby__ sean-k-mooney: that seemed to work. Im able to migrate from xena to yoga hypervisor. But Im not able to migrate to the original hypervisor that I fully patched (updated qemu/libvirt). On that machine Im seeing the following: qemu-kvm: load of migration failed: Invalid argument: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-10-18T17:33:57.637682Z qemu-kvm: Missing section footer for
17:41:22 sean-k-mooney i that sound liek a qemu crash
17:44:54 melwitt sean-k-mooney: yeah, he did and I agree it seems like a bug in the migration. but if it is, why doesn't it fail in grenade? I looked for awhile and couldn't figure it out. I thought stephenfin might have ideas since he worked on alembic stuff
17:46:21 sean-k-mooney i think we get diffent behavior in fressh install vs upgraded case but i dont exactly rememebr why
17:46:54 sean-k-mooney i assume the base scemea we use in once ase either has or does not have the column
17:48:23 melwitt yeah.. if a fresh install of xena doesn't have the columns and the yoga migration would fail to drop them bc they don't exist, I thought it should have happened in grenade since grenade does a fresh install of xena and then upgrades it to yoga
17:48:34 sean-k-mooney https://github.com/openstack/nova/blob/stable/xena/nova/db/api/migrations/versions/d67eeaabee36_initial_version.py#L175-L208
17:48:46 sean-k-mooney so that is how the build request tabel is created on a clena install
17:49:21 sean-k-mooney https://github.com/openstack/nova/blob/stable/xena/nova/db/api/migrations/versions/d67eeaabee36_initial_version.py#L186
17:49:28 sean-k-mooney so it has vm_states
17:49:36 sean-k-mooney so a clean install on xena woudl have it
17:49:54 sean-k-mooney but a cloud that was upgraded form wallaby perhaps might not
17:50:49 sean-k-mooney hum it should be there in wallaby https://github.com/openstack/nova/blob/stable/wallaby/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/067_train.py#L178
17:51:33 melwitt older versions do the initial migration too. I am just not seeing how it's possible to get into a case where you don't have the columns in xena if the yoga migration is the only thing that removes them
17:51:46 sean-k-mooney its also there if you use legacy migrations for green filed in xena https://github.com/openstack/nova/blob/stable/xena/nova/db/api/legacy_migrations/versions/067_train.py#L166
17:52:14 sean-k-mooney melwitt: perhaps we need to look at the code before we collapsed the migrations
17:54:26 sean-k-mooney if the bug was intoduced anywhere it was in the db compaction series
17:57:10 sean-k-mooney so in the patch before the dbcompaction
17:57:12 sean-k-mooney https://github.com/openstack/nova/blob/2d4eff0ef789957fe5498640015a6cccff43a2c3/nova/db/sqlalchemy/api_models.py#L241
17:57:27 sean-k-mooney the build request tabel did not have the offending colum in the model
17:59:11 sean-k-mooney ah hwere so even then it was created in migration 6 with the field
17:59:13 sean-k-mooney https://github.com/openstack/nova/blob/2d4eff0ef789957fe5498640015a6cccff43a2c3/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/006_build_request.py#L48
18:00:21 sean-k-mooney i dont see the removal migation however at this point in the repo
18:01:31 melwitt is that yoga? bc the removal migration was added in yoga I think
18:01:46 sean-k-mooney that is xena
18:01:58 melwitt oh ok
18:02:00 sean-k-mooney so in xena the table was created with the column and i dont see the removal
18:02:15 sean-k-mooney an pre xena the colume also existed
18:03:19 sean-k-mooney do you knwo which commit added the drop
18:04:17 sean-k-mooney https://github.com/openstack/nova/commit/9657297dd6c63e7a1e0c84c3e943b26f1795d388

Earlier   Later