| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-09-12 | |||
| 16:36:39 | sean-k-mooney | then stub the function for the shared storage check to return true | |
| 16:36:57 | sean-k-mooney | https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/tests/functional/regressions/test_bug_1939545.py | |
| 16:36:59 | bauzas | we have a couple of functional tests for live-migration, that's not a problem | |
| 16:37:05 | sean-k-mooney | i think has all the ground work | |
| 16:37:16 | bauzas | oh cool, thanks for the digging but I was about to do it :) | |
| 17:20:08 | bauzas | okay, I'm out of steam and I don't have ideas on how to correctly assert we delete the local disks, except checking we call the delete | |
| 17:20:17 | bauzas | see you tomorrow | |
| 20:09:17 | opendevreview | sean mooney proposed openstack/nova master: repoduce post liberty pre vicoria instance numa db issue https://review.opendev.org/c/openstack/nova/+/929025 | |
| 20:09:18 | opendevreview | sean mooney proposed openstack/nova master: allow upgrade of pre-victoria InstanceNUMACells https://review.opendev.org/c/openstack/nova/+/929187 | |
| 20:14:55 | opendevreview | sean mooney proposed openstack/nova master: repoduce post liberty pre vicoria instance numa db issue https://review.opendev.org/c/openstack/nova/+/929025 | |
| 20:14:56 | opendevreview | sean mooney proposed openstack/nova master: allow upgrade of pre-victoria InstanceNUMACells https://review.opendev.org/c/openstack/nova/+/929187 | |
| 22:11:45 | sean-k-mooney | looks like openstack clinet is broken | |
| 22:11:48 | sean-k-mooney | 024-09-12 21:04:32.171696 | controller | + /opt/stack/nova/roles/run-evacuate-hook/files/test_negative_evacuate.sh:evacuate_and_wait_for_error:19 : openstack --os-compute-api-version 2.67 server evacuate --host np0038485592 evacuate-test | |
| 22:11:51 | sean-k-mooney | 2024-09-12 21:04:33.305045 | controller | compute version 2.67 is not in supported versions: 2, 2.1 | |
| 22:11:53 | sean-k-mooney | 2024-09-12 21:04:34.474915 | controller | Proxy.evacuate_server() got an unexpected keyword argument 'password' | |
| 22:12:49 | sean-k-mooney | stephenfin: ^ fyi for the morning | |
| #openstack-nova - 2024-09-13 | |||
| 03:59:41 | opendevreview | Sam Morrison proposed openstack/nova master: Filter out deleted instances when looking for build timouts https://review.opendev.org/c/openstack/nova/+/880125 | |
| 05:59:13 | frickler | stephenfin: sean-k-mooney: note that we release osc 7.1.0 yesterday, so likely related to that. 7.0.0 was also broken and never made it into u-c. I can also reproduce the above issue locally fwiw | |
| 06:01:22 | frickler | maybe let's move this to sdks channel | |
| 07:56:08 | gibi | sean-k-mooney: I have small things in https://review.opendev.org/q/topic:%22bug/2080556%22 but I'm overall positive. | |
| 08:02:26 | gibi | bauzas: I have a nit in the prelude https://review.opendev.org/c/openstack/nova/+/928995 | |
| 08:03:17 | gibi | bauzas: also I think we can go with the RC1 for placement even if we need to wait with the nova RC1 | |
| 08:04:54 | gibi | bauzas: also we have negative answer for our proposal of fixing the live migration issue https://bugs.launchpad.net/nova/+bug/2080436 | |
| 08:05:16 | bauzas | cool, I'm just busy with another task | |
| 08:05:27 | bauzas | you're next in my waiting queue :) | |
| 08:05:40 | gibi | I'm not waiting :) | |
| 08:14:10 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add Dalmatian prelude section https://review.opendev.org/c/openstack/nova/+/928995 | |
| 08:26:38 | bauzas | gibi: I have full attention to the regression bug now | |
| 08:26:54 | bauzas | so, that sounds the conditional clause isn't enough | |
| 08:27:18 | bauzas | something in the cleanup really deletes the instance path | |
| 08:29:58 | gibi | I just published a set of observations in https://review.opendev.org/c/openstack/nova/+/928970 | |
| 08:32:12 | gibi | If I have to guess the complication is around: > NUMA pinned VMs and an NFS mount for the VMs, but we also use cinder boot volumes | |
| 08:33:07 | gibi | I guess we won't be able to avoid setting up a multinode devstack.. | |
| 08:36:28 | bauzas | I tried yesterday to write a functional test but that was hard | |
| 08:37:00 | bauzas | I would have to create a specific tempfs for the instance paths in order to correctly reproduce it | |
| 08:37:19 | bauzas | but I failed finding where we mock the paths | |
| 08:37:31 | bauzas | ++ | |
| 08:37:43 | bauzas | (oops, wrong window) | |
| 08:40:15 | gibi | I will try to set up multinode devstack with NFS but I cannot promise it will work today | |
| 08:41:18 | bauzas | I have two nodes, I can also try to modify them | |
| 08:41:42 | bauzas | either way, we'll probably need to come to a decision today or monday | |
| 08:41:58 | bauzas | we can't delay RC1 more | |
| 08:42:24 | bauzas | but I'd like some core quorum | |
| 08:43:18 | gibi | an alternative is to revert the power mgmt live migration fix. Declare that that power mgmt does not work with live migration and fix the whole thing later | |
| 08:44:07 | bauzas | that's what I had in mind | |
| 08:44:26 | bauzas | or a third alternative is ship power management but let the call be optional | |
| 08:44:46 | bauzas | if you don't setup power management, you shouldn't be impacted | |
| 08:44:49 | gibi | we could hack a fix together where shared storage live migration works if power mgmt is not used, and power mgmt live migration works if no shared storage is used (basically the original proposal from the reported) but that is uggly | |
| 08:45:14 | bauzas | that's the concern I have with the current master, which inconditionnally says you're power manageable | |
| 08:45:14 | gibi | and power mgmt with shared storage will still not work | |
| 08:45:44 | bauzas | I wish we had a flag on the migrate_data object that would signal that info | |
| 08:45:53 | bauzas | like we did for mdevs or vpmems | |
| 08:46:27 | gibi | that would not remove the problem that shared storage live migration does not work if you have power mgmt, mdevs, or vpmems. | |
| 08:46:37 | gibi | as those would trigger calling clenaup | |
| 08:46:38 | bauzas | yup, but that's something we can doc | |
| 08:47:04 | bauzas | here the problem is that shared storage is broken *anyway* | |
| 08:47:20 | bauzas | and you can't opt out from it | |
| 08:48:28 | gibi | yepp it seem shared storage live migration only works we we don't call cleanup | |
| 08:48:48 | gibi | but I'm not 100% sure whw | |
| 08:48:50 | gibi | why | |
| 08:49:27 | gibi | do you have any comment on my las comment https://review.opendev.org/c/openstack/nova/+/928970/comment/c28b9ea3_30ed15a5/ | |
| 08:49:58 | gibi | especially around deleting the instance dir when the VM is volume backed? | |
| 08:51:50 | bauzas | even if the instance is volume backed, we still have an instance dir, right? | |
| 08:51:55 | bauzas | we just don't have disks | |
| 08:53:42 | bauzas | I could imagine that they have NFS on /var/lib/instances | |
| 08:53:57 | bauzas | so the BFV instances are present on that shared path | |
| 08:54:22 | bauzas | in that case, | |
| 08:54:25 | bauzas | cleanup_instance_dir = migrate_data.is_shared_block_storage | |
| 08:54:36 | bauzas | it will cleanup the dir, right ? | |
| 08:55:01 | bauzas | oh wait no | |
| 08:55:17 | bauzas | that's not shared storage, that's shared RBD image backend | |
| 08:55:44 | gibi | the instance dir for a volume booted instance (without config drive) only holds the console.log fiel nothing else | |
| 08:56:22 | bauzas | I don't disagree, but that's still an instance dir | |
| 08:56:23 | gibi | is_shared_block_storage will be true if the instance is booted from volume | |
| 08:56:44 | gibi | (and in other cases too) | |
| 08:56:54 | bauzas | okay, so that's why this is deleting the instance dir | |
| 08:56:59 | gibi | Is it a problem if we delete hte instance dir | |
| 08:57:03 | gibi | ? | |
| 08:57:08 | bauzas | good question | |
| 08:57:26 | bauzas | as you said, the instance dir is quite empty | |
| 08:57:38 | bauzas | where is stored the domain definition ? | |
| 09:01:51 | gibi | that is nova does not store the xml | |
| 09:02:21 | gibi | * nova does not store the xml | |
| 09:02:30 | gibi | I'm not sure libvirt stores it by default | |
| 09:03:21 | gibi | it is under /run/libvirt/qemu/instance-00000002.xml | |
| 09:03:31 | gibi | in devstack | |
| 09:04:36 | bauzas | okay | |
| 09:05:44 | bauzas | your question is actually good : does it harm if we delete an instance dir from a BFV instance that's shared ? | |
| 09:06:09 | bauzas | https://github.com/openstack/nova/blob/16d815990b53b9afa35fc4da38609f87820fa690/nova/virt/libvirt/driver.py#L1708-L1711 | |
| 09:06:24 | bauzas | I enjoy mdbooth's comment : "I'm pretty sure this is wrong" | |
| 09:07:24 | bauzas | from what I read from both comments is that this is EXPECTED that we delete the instance dir if the instance is BFV | |
| 09:07:44 | bauzas | not sure we do support shared storage with BFV | |
| 09:07:50 | bauzas | see ? | |
| 09:12:34 | gibi | I see | |
| 09:13:14 | gibi | I'm not sure wht mdbooth's comment really mean. I.e. does the comment really implies that the logic is wrong and we should change it? | |
| 09:16:19 | gibi | we need to figure out if live migration on shared storage with a volume backed VM actually fails if the instance dir is deleted | |
| 09:16:34 | gibi | we never seen the real failure from the reporter | |
| 09:16:45 | gibi | they just stated that the dir is deleted | |
| 09:17:06 | gibi | the only thing they said is | |
| 09:17:11 | gibi | "making the VM unusable for future operations." | |