Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-10-18
16:31:51 colby__ Im in the process of upgrading yoga and trying to live migrate instances to the already upgraded machines and running into this issue
16:32:07 colby__ is there a way to override the disk size of a running instance flavor?
16:32:34 sean-k-mooney colby__: are you using images_type=rbd
16:32:38 sean-k-mooney or cinder volumes
16:33:24 colby__ cinder volumes
16:33:47 sean-k-mooney adn your using images_type=qcow
16:33:52 sean-k-mooney or raw or flat
16:34:14 colby__ we did not set it so its currently default
16:34:32 sean-k-mooney ok that woudl be defcault which for libvirt is qcow
16:35:09 sean-k-mooney imporanly in that config then the expect bevioru is to report local_disk_gb as the free space on the compute node partion where /var/lib/nova is
16:35:20 sean-k-mooney and that was the ame behvior in older release
16:35:33 colby__ ok so set images_type=rbd then?
16:35:37 sean-k-mooney no
16:35:51 sean-k-mooney that means use cpeh for root disk
16:35:56 sean-k-mooney even with out ceinder
16:36:14 colby__ ah sorry I misunderstood. so raw then?
16:36:21 sean-k-mooney no dont change it
16:36:35 sean-k-mooney nova is working as it should form a compute config point of view
16:36:47 colby__ ok
16:37:01 sean-k-mooney i need to check somehting but i alo need to join a 1:1 with my manager
16:37:12 colby__ ok no problem. Thanks for the help
16:37:42 sean-k-mooney did you used to set allcoation raitio by aggreates prehaps?
16:39:56 dansmith ugh, feels like we're back to gate suckage.. some trivial patches are in recheck hell
16:41:03 colby__ sean-k-mooney: we never set anything for disk allocation
17:03:28 colby__ sean-k-mooney: I was able to increase the disk allocation ratio but now seeing this: nova.exception.InvalidSharedStorage: <host> is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks. This is a boot from volume ceph instance
17:04:04 colby__ is this all due to the disk_gb of the flavors?
17:08:19 colby__ Im migrating from a xena hypervisor to a yoga and seeing this issue
17:21:38 sean-k-mooney colby__: so partly. but you shoudl have had the same behavior on xena and yoga i think
17:22:01 sean-k-mooney colby__: so you should not be passing --shared-srotage wehn live migrating
17:22:32 colby__ I was just using the GUI for the live migration but can switch to the cli
17:22:36 sean-k-mooney you should be using a newer microvstion where that defautl to auto
17:23:10 sean-k-mooney so for boot form volume guest you should not tick shared storage
17:23:43 sean-k-mooney shared storage for that option really means is /var/lib/nova on nfs
17:23:58 sean-k-mooney so are you using images_type=rbd
17:24:17 colby__ no
17:24:23 colby__ just set to default
17:24:30 sean-k-mooney i missed typed
17:25:11 sean-k-mooney what i ment is you only say "shared_stroage=true" if you the vm is booted on a shared file cycle without using cinder volumes
17:25:21 sean-k-mooney so that is not your case
17:25:35 colby__ also there is no check box for shared storage just "block migration" and "disk overcommit" on gui live migration
17:25:56 sean-k-mooney right so your using horizon yes?
17:26:22 sean-k-mooney it likely would be better to do live migration form the openstack client
17:26:32 colby__ yep will try that
17:28:38 sean-k-mooney so in mitaka 2.25 we added auto
17:28:40 sean-k-mooney https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-mitaka
17:32:14 melwitt stephenfin: I wondered if you might be able to shed some light on this bug report regarding an alembic migration https://bugs.launchpad.net/nova/+bug/2008716 they report a failure to drop columns in the build_requests table going from xena -> yoga. yet this doesn't fail in the grenade job for xena -> yoga, so I'm confused if there is really a bug?
17:40:35 sean-k-mooney melwitt: i think gibi ^ looked at that before
17:40:38 colby__ sean-k-mooney: that seemed to work. Im able to migrate from xena to yoga hypervisor. But Im not able to migrate to the original hypervisor that I fully patched (updated qemu/libvirt). On that machine Im seeing the following: qemu-kvm: load of migration failed: Invalid argument: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-10-18T17:33:57.637682Z qemu-kvm: Missing section footer for
17:41:22 sean-k-mooney i that sound liek a qemu crash
17:44:54 melwitt sean-k-mooney: yeah, he did and I agree it seems like a bug in the migration. but if it is, why doesn't it fail in grenade? I looked for awhile and couldn't figure it out. I thought stephenfin might have ideas since he worked on alembic stuff
17:46:21 sean-k-mooney i think we get diffent behavior in fressh install vs upgraded case but i dont exactly rememebr why
17:46:54 sean-k-mooney i assume the base scemea we use in once ase either has or does not have the column
17:48:23 melwitt yeah.. if a fresh install of xena doesn't have the columns and the yoga migration would fail to drop them bc they don't exist, I thought it should have happened in grenade since grenade does a fresh install of xena and then upgrades it to yoga
17:48:34 sean-k-mooney https://github.com/openstack/nova/blob/stable/xena/nova/db/api/migrations/versions/d67eeaabee36_initial_version.py#L175-L208
17:48:46 sean-k-mooney so that is how the build request tabel is created on a clena install
17:49:21 sean-k-mooney https://github.com/openstack/nova/blob/stable/xena/nova/db/api/migrations/versions/d67eeaabee36_initial_version.py#L186
17:49:28 sean-k-mooney so it has vm_states
17:49:36 sean-k-mooney so a clean install on xena woudl have it
17:49:54 sean-k-mooney but a cloud that was upgraded form wallaby perhaps might not
17:50:49 sean-k-mooney hum it should be there in wallaby https://github.com/openstack/nova/blob/stable/wallaby/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/067_train.py#L178
17:51:33 melwitt older versions do the initial migration too. I am just not seeing how it's possible to get into a case where you don't have the columns in xena if the yoga migration is the only thing that removes them
17:51:46 sean-k-mooney its also there if you use legacy migrations for green filed in xena https://github.com/openstack/nova/blob/stable/xena/nova/db/api/legacy_migrations/versions/067_train.py#L166
17:52:14 sean-k-mooney melwitt: perhaps we need to look at the code before we collapsed the migrations
17:54:26 sean-k-mooney if the bug was intoduced anywhere it was in the db compaction series
17:57:10 sean-k-mooney so in the patch before the dbcompaction
17:57:12 sean-k-mooney https://github.com/openstack/nova/blob/2d4eff0ef789957fe5498640015a6cccff43a2c3/nova/db/sqlalchemy/api_models.py#L241
17:57:27 sean-k-mooney the build request tabel did not have the offending colum in the model
17:59:11 sean-k-mooney ah hwere so even then it was created in migration 6 with the field
17:59:13 sean-k-mooney https://github.com/openstack/nova/blob/2d4eff0ef789957fe5498640015a6cccff43a2c3/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/006_build_request.py#L48
18:00:21 sean-k-mooney i dont see the removal migation however at this point in the repo
18:01:31 melwitt is that yoga? bc the removal migration was added in yoga I think
18:01:46 sean-k-mooney that is xena
18:01:58 melwitt oh ok
18:02:00 sean-k-mooney so in xena the table was created with the column and i dont see the removal
18:02:15 sean-k-mooney an pre xena the colume also existed
18:03:19 sean-k-mooney do you knwo which commit added the drop
18:04:17 sean-k-mooney https://github.com/openstack/nova/commit/9657297dd6c63e7a1e0c84c3e943b26f1795d388
18:06:07 sean-k-mooney so honestly i don tsee the bug
18:06:35 sean-k-mooney xena had the colume and both ways of createating a clena db have the column
18:32:50 melwitt yeah, same
18:42:53 opendevreview Alexey Stupnikov proposed openstack/nova master: Preserve cached base images for failed resize ops https://review.opendev.org/c/openstack/nova/+/877410
22:02:55 opendevreview melanie witt proposed openstack/nova master: docs: Fix unified limits code block text alignment https://review.opendev.org/c/openstack/nova/+/898759
23:05:48 opendevreview Merged openstack/nova master: Improve logging at '_numa_cells_support_network_metadata' https://review.opendev.org/c/openstack/nova/+/860930
23:16:21 opendevreview Merged openstack/nova master: Clean up service_get_all() https://review.opendev.org/c/openstack/nova/+/897674
#openstack-nova - 2023-10-19
11:01:54 opendevreview Pavlo Shchelokovskyy proposed openstack/placement master: Use new db session when retrying https://review.opendev.org/c/openstack/placement/+/898807
11:02:39 opendevreview Pavlo Shchelokovskyy proposed openstack/placement master: Use separate transaction in capacity check https://review.opendev.org/c/openstack/placement/+/898808
13:36:07 opendevreview Merged openstack/nova master: docs: Fix unified limits code block text alignment https://review.opendev.org/c/openstack/nova/+/898759
14:12:12 dansmith sean-k-mooney: can you hit this? https://review.opendev.org/c/openstack/nova/+/898741
14:17:41 sean-k-mooney sorry was doing something else but sure i can look at it now
14:17:52 sean-k-mooney ah the revert sure
14:18:18 sean-k-mooney done
14:18:47 dansmith sean-k-mooney: yeah didn't mean to distract, just figured it was an easy +W
14:19:20 sean-k-mooney no its fine i was just commiting something had litrally just finished
14:20:11 sean-k-mooney so before i started somethign else is the perfect time to ping
14:54:21 bauzas guys, I have a conflicting personal appointment next Friday at 4pm UTC
14:54:28 bauzas next Thur*
14:54:51 bauzas do people prefer we unbook the slot or do we leave it without me ?
14:56:21 dansmith I think we can do things without you, but maybe try to avoid scheduling anything important there and we can use it as open discussion if and as appropriate?
14:56:35 bauzas cool, I just noted it then in the etherpad
14:58:02 bauzas ralonsoh: do you have any nova related topics you'd like to be discussed at the PTG ?

Earlier   Later