| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-18 | |||
| 16:26:53 | sean-k-mooney | if you mena ceph volume via image_type=rbd | |
| 16:26:59 | sean-k-mooney | then use flavor as normal | |
| 16:27:08 | sean-k-mooney | with disk_gb set to the correct size | |
| 16:27:52 | sean-k-mooney | in yoga and master for that mater we report local_gb based on the rbd pool size for every instnace as far as i remember | |
| 16:31:06 | colby__ | yea unfortunately our flavors have disk size set even though they all use ceph boot volumes. Everythign we use is in ceph (images/ephemeral/volumes). But seems there is no way around the already set flavor disk size then. From the hypervisor screen its seems to be only showing the hypervisor boot disk volume for local disk | |
| 16:31:51 | colby__ | Im in the process of upgrading yoga and trying to live migrate instances to the already upgraded machines and running into this issue | |
| 16:32:07 | colby__ | is there a way to override the disk size of a running instance flavor? | |
| 16:32:34 | sean-k-mooney | colby__: are you using images_type=rbd | |
| 16:32:38 | sean-k-mooney | or cinder volumes | |
| 16:33:24 | colby__ | cinder volumes | |
| 16:33:47 | sean-k-mooney | adn your using images_type=qcow | |
| 16:33:52 | sean-k-mooney | or raw or flat | |
| 16:34:14 | colby__ | we did not set it so its currently default | |
| 16:34:32 | sean-k-mooney | ok that woudl be defcault which for libvirt is qcow | |
| 16:35:09 | sean-k-mooney | imporanly in that config then the expect bevioru is to report local_disk_gb as the free space on the compute node partion where /var/lib/nova is | |
| 16:35:20 | sean-k-mooney | and that was the ame behvior in older release | |
| 16:35:33 | colby__ | ok so set images_type=rbd then? | |
| 16:35:37 | sean-k-mooney | no | |
| 16:35:51 | sean-k-mooney | that means use cpeh for root disk | |
| 16:35:56 | sean-k-mooney | even with out ceinder | |
| 16:36:14 | colby__ | ah sorry I misunderstood. so raw then? | |
| 16:36:21 | sean-k-mooney | no dont change it | |
| 16:36:35 | sean-k-mooney | nova is working as it should form a compute config point of view | |
| 16:36:47 | colby__ | ok | |
| 16:37:01 | sean-k-mooney | i need to check somehting but i alo need to join a 1:1 with my manager | |
| 16:37:12 | colby__ | ok no problem. Thanks for the help | |
| 16:37:42 | sean-k-mooney | did you used to set allcoation raitio by aggreates prehaps? | |
| 16:39:56 | dansmith | ugh, feels like we're back to gate suckage.. some trivial patches are in recheck hell | |
| 16:41:03 | colby__ | sean-k-mooney: we never set anything for disk allocation | |
| 17:03:28 | colby__ | sean-k-mooney: I was able to increase the disk allocation ratio but now seeing this: nova.exception.InvalidSharedStorage: <host> is not on shared storage: Shared storage live-migration requires either shared storage or boot-from-volume with no local disks. This is a boot from volume ceph instance | |
| 17:04:04 | colby__ | is this all due to the disk_gb of the flavors? | |
| 17:08:19 | colby__ | Im migrating from a xena hypervisor to a yoga and seeing this issue | |
| 17:21:38 | sean-k-mooney | colby__: so partly. but you shoudl have had the same behavior on xena and yoga i think | |
| 17:22:01 | sean-k-mooney | colby__: so you should not be passing --shared-srotage wehn live migrating | |
| 17:22:32 | colby__ | I was just using the GUI for the live migration but can switch to the cli | |
| 17:22:36 | sean-k-mooney | you should be using a newer microvstion where that defautl to auto | |
| 17:23:10 | sean-k-mooney | so for boot form volume guest you should not tick shared storage | |
| 17:23:43 | sean-k-mooney | shared storage for that option really means is /var/lib/nova on nfs | |
| 17:23:58 | sean-k-mooney | so are you using images_type=rbd | |
| 17:24:17 | colby__ | no | |
| 17:24:23 | colby__ | just set to default | |
| 17:24:30 | sean-k-mooney | i missed typed | |
| 17:25:11 | sean-k-mooney | what i ment is you only say "shared_stroage=true" if you the vm is booted on a shared file cycle without using cinder volumes | |
| 17:25:21 | sean-k-mooney | so that is not your case | |
| 17:25:35 | colby__ | also there is no check box for shared storage just "block migration" and "disk overcommit" on gui live migration | |
| 17:25:56 | sean-k-mooney | right so your using horizon yes? | |
| 17:26:22 | sean-k-mooney | it likely would be better to do live migration form the openstack client | |
| 17:26:32 | colby__ | yep will try that | |
| 17:28:38 | sean-k-mooney | so in mitaka 2.25 we added auto | |
| 17:28:40 | sean-k-mooney | https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#maximum-in-mitaka | |
| 17:32:14 | melwitt | stephenfin: I wondered if you might be able to shed some light on this bug report regarding an alembic migration https://bugs.launchpad.net/nova/+bug/2008716 they report a failure to drop columns in the build_requests table going from xena -> yoga. yet this doesn't fail in the grenade job for xena -> yoga, so I'm confused if there is really a bug? | |
| 17:40:35 | sean-k-mooney | melwitt: i think gibi ^ looked at that before | |
| 17:40:38 | colby__ | sean-k-mooney: that seemed to work. Im able to migrate from xena to yoga hypervisor. But Im not able to migrate to the original hypervisor that I fully patched (updated qemu/libvirt). On that machine Im seeing the following: qemu-kvm: load of migration failed: Invalid argument: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-10-18T17:33:57.637682Z qemu-kvm: Missing section footer for | |
| 17:41:22 | sean-k-mooney | i that sound liek a qemu crash | |
| 17:44:54 | melwitt | sean-k-mooney: yeah, he did and I agree it seems like a bug in the migration. but if it is, why doesn't it fail in grenade? I looked for awhile and couldn't figure it out. I thought stephenfin might have ideas since he worked on alembic stuff | |
| 17:46:21 | sean-k-mooney | i think we get diffent behavior in fressh install vs upgraded case but i dont exactly rememebr why | |
| 17:46:54 | sean-k-mooney | i assume the base scemea we use in once ase either has or does not have the column | |
| 17:48:23 | melwitt | yeah.. if a fresh install of xena doesn't have the columns and the yoga migration would fail to drop them bc they don't exist, I thought it should have happened in grenade since grenade does a fresh install of xena and then upgrades it to yoga | |
| 17:48:34 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/xena/nova/db/api/migrations/versions/d67eeaabee36_initial_version.py#L175-L208 | |
| 17:48:46 | sean-k-mooney | so that is how the build request tabel is created on a clena install | |
| 17:49:21 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/xena/nova/db/api/migrations/versions/d67eeaabee36_initial_version.py#L186 | |
| 17:49:28 | sean-k-mooney | so it has vm_states | |
| 17:49:36 | sean-k-mooney | so a clean install on xena woudl have it | |
| 17:49:54 | sean-k-mooney | but a cloud that was upgraded form wallaby perhaps might not | |
| 17:50:49 | sean-k-mooney | hum it should be there in wallaby https://github.com/openstack/nova/blob/stable/wallaby/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/067_train.py#L178 | |
| 17:51:33 | melwitt | older versions do the initial migration too. I am just not seeing how it's possible to get into a case where you don't have the columns in xena if the yoga migration is the only thing that removes them | |
| 17:51:46 | sean-k-mooney | its also there if you use legacy migrations for green filed in xena https://github.com/openstack/nova/blob/stable/xena/nova/db/api/legacy_migrations/versions/067_train.py#L166 | |
| 17:52:14 | sean-k-mooney | melwitt: perhaps we need to look at the code before we collapsed the migrations | |
| 17:54:26 | sean-k-mooney | if the bug was intoduced anywhere it was in the db compaction series | |
| 17:57:10 | sean-k-mooney | so in the patch before the dbcompaction | |
| 17:57:12 | sean-k-mooney | https://github.com/openstack/nova/blob/2d4eff0ef789957fe5498640015a6cccff43a2c3/nova/db/sqlalchemy/api_models.py#L241 | |
| 17:57:27 | sean-k-mooney | the build request tabel did not have the offending colum in the model | |
| 17:59:11 | sean-k-mooney | ah hwere so even then it was created in migration 6 with the field | |
| 17:59:13 | sean-k-mooney | https://github.com/openstack/nova/blob/2d4eff0ef789957fe5498640015a6cccff43a2c3/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/006_build_request.py#L48 | |
| 18:00:21 | sean-k-mooney | i dont see the removal migation however at this point in the repo | |
| 18:01:31 | melwitt | is that yoga? bc the removal migration was added in yoga I think | |
| 18:01:46 | sean-k-mooney | that is xena | |
| 18:01:58 | melwitt | oh ok | |
| 18:02:00 | sean-k-mooney | so in xena the table was created with the column and i dont see the removal | |
| 18:02:15 | sean-k-mooney | an pre xena the colume also existed | |
| 18:03:19 | sean-k-mooney | do you knwo which commit added the drop | |
| 18:04:17 | sean-k-mooney | https://github.com/openstack/nova/commit/9657297dd6c63e7a1e0c84c3e943b26f1795d388 | |
| 18:06:07 | sean-k-mooney | so honestly i don tsee the bug | |
| 18:06:35 | sean-k-mooney | xena had the colume and both ways of createating a clena db have the column | |
| 18:32:50 | melwitt | yeah, same | |
| 18:42:53 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Preserve cached base images for failed resize ops https://review.opendev.org/c/openstack/nova/+/877410 | |
| 22:02:55 | opendevreview | melanie witt proposed openstack/nova master: docs: Fix unified limits code block text alignment https://review.opendev.org/c/openstack/nova/+/898759 | |
| 23:05:48 | opendevreview | Merged openstack/nova master: Improve logging at '_numa_cells_support_network_metadata' https://review.opendev.org/c/openstack/nova/+/860930 | |
| 23:16:21 | opendevreview | Merged openstack/nova master: Clean up service_get_all() https://review.opendev.org/c/openstack/nova/+/897674 | |
| #openstack-nova - 2023-10-19 | |||
| 11:01:54 | opendevreview | Pavlo Shchelokovskyy proposed openstack/placement master: Use new db session when retrying https://review.opendev.org/c/openstack/placement/+/898807 | |
| 11:02:39 | opendevreview | Pavlo Shchelokovskyy proposed openstack/placement master: Use separate transaction in capacity check https://review.opendev.org/c/openstack/placement/+/898808 | |
| 13:36:07 | opendevreview | Merged openstack/nova master: docs: Fix unified limits code block text alignment https://review.opendev.org/c/openstack/nova/+/898759 | |
| 14:12:12 | dansmith | sean-k-mooney: can you hit this? https://review.opendev.org/c/openstack/nova/+/898741 | |
| 14:17:41 | sean-k-mooney | sorry was doing something else but sure i can look at it now | |
| 14:17:52 | sean-k-mooney | ah the revert sure | |
| 14:18:18 | sean-k-mooney | done | |
| 14:18:47 | dansmith | sean-k-mooney: yeah didn't mean to distract, just figured it was an easy +W | |
| 14:19:20 | sean-k-mooney | no its fine i was just commiting something had litrally just finished | |
| 14:20:11 | sean-k-mooney | so before i started somethign else is the perfect time to ping | |
| 14:54:21 | bauzas | guys, I have a conflicting personal appointment next Friday at 4pm UTC | |