| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-02 | |||
| 21:52:06 | admin1 | openstack compute service delete 56 => Unable to delete compute service that has in-progress migrations. .. How do I check these migrations ? | |
| 22:08:34 | melwitt | admin1: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-migration.html#server-migration-list | |
| 22:10:16 | admin1 | melwitt, this cluster is still in rocky .. so that command is not there | |
| 22:10:36 | admin1 | we are in the process of upgrading it .. so migrating, delete compute, reinstall and upgrade, add it back | |
| 22:11:02 | opendevreview | melanie witt proposed openstack/nova master: Raise InstanceNotFound on fkey constraint fail saving info cache https://review.opendev.org/c/openstack/nova/+/826942 | |
| 22:11:02 | melwitt | oh I see | |
| 22:18:09 | admin1 | nova migration-list lists the migraitons, but there is nothing pending or ongoing | |
| 22:23:41 | melwitt | you could try the force or abort commands on the migrations if they are leftover https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-force-complete | |
| 22:23:43 | melwitt | https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort | |
| 22:34:12 | admin1 | i found the instance.. the instance is in pre-migrating status .... when i enter the command nova live-migration-abort $UUID $ID, it says Instance $UUID is in an invalid state for 'abort live migration' | |
| 22:34:32 | admin1 | how do I abort a pre-migration status ? | |
| 22:35:09 | admin1 | server show status is Running .. so instance does not have a pre-migrating status | |
| 22:38:14 | melwitt | did you try the force complete too? | |
| 22:40:15 | admin1 | yeah .. it gave Instance $UUID is in an invalid state for 'force_complete' | |
| 22:41:02 | admin1 | maybe i can just delete the compute service from the db ? | |
| 22:41:06 | admin1 | and add it again .. | |
| 22:41:20 | admin1 | want to do it properly though .. | |
| 22:44:32 | admin1 | melwitt, https://gist.github.com/a1git/9a975b96cd91da4683084c7df3220530 | |
| 22:45:43 | admin1 | baiscally the way i have been upgrading from rocky (xenial ) -> bionic is .. for example, migrate all instances from A - B and empty A, then compute service delete A , reinstall A with bionic, same hostname, install nova .. , and then repeat the process again .. | |
| 22:45:52 | admin1 | but for some reasons, this one is locked/blocked .. | |
| 22:46:01 | melwitt | +1 going into the db is the last resort if the proper tools don't work | |
| 22:48:21 | admin1 | i think 'pre-migrating' is blocking the deletion .. and I have found nothing to force this to either error or completed | |
| 22:51:41 | admin1 | i did a mysqldump and i found the pre-migrating in only 1 table .. nova.migrations .. | |
| 22:53:28 | melwitt | I think you are right that it's the migration(s) blocking the service delete | |
| 22:54:01 | admin1 | update migrations set status='error' where instance_uuid=$UUID and status='pre-migrating'; | |
| 22:54:15 | admin1 | and then the delet worked fine :) .. openstack compute service delete 56 | |
| 22:56:29 | melwitt | I was just about to suggest that, change the status rather than deleting the migration record :) | |
| 23:03:27 | admin1 | how to delete a host from placement service ? | |
| 23:04:49 | melwitt | hm, the service delete should have done that | |
| 23:05:03 | melwitt | it's the host you deleted the service for right? | |
| 23:06:18 | admin1 | ResourceProviderCreationFailed: Failed to create resource provider h1 | |
| 23:07:24 | admin1 | openstack compute service list -- it gets added there .. | |
| 23:08:09 | admin1 | i tried via "openstack compute service delete $id" and in 2nd attempt nova service-delete $uuid | |
| 23:08:25 | admin1 | 4 other servers, no issues .. this one hypervisor = this and that errors :) | |
| 23:08:54 | melwitt | this is the cli for the placement service https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-delete | |
| 23:10:16 | melwitt | just be careful and make sure it's not associated with any instances allocations | |
| 23:10:21 | melwitt | before deleting it | |
| 23:10:26 | admin1 | there are no instances | |
| 23:10:38 | melwitt | k | |
| 23:10:43 | admin1 | all instances were migrated, this is a new install which just came up | |
| 23:10:50 | melwitt | ok cool | |
| 23:10:52 | admin1 | no such command in rocky | |
| 23:11:20 | admin1 | this upgrade is necessary to upgrade from rocky =.. as stein needs 18.04 .. | |
| 23:11:48 | admin1 | if i need to delete a hypervisor, isn't it just deleting compute service ? | |
| 23:12:19 | melwitt | yeah but it does a few things: deletes the nova service record, the nova compute_nodes record, and then the placement resource provider | |
| 23:12:45 | melwitt | and the last step has (probably) failed in your case from the previous delete | |
| 23:13:15 | melwitt | so it sees a duplicate placement resource provider and refuses to create it. so you are correct you need to delete it so it can recreate it again | |
| 23:13:38 | melwitt | as for the command, you could upgrade openstackclient to newer than rocky and it will still work with rocky | |
| 23:18:23 | admin1 | sudo apt install python3-openstackclient python3-openstackclient is already the newest version (5.6.0-0ubuntu1). ... no such command | |
| 23:19:07 | melwitt | yeah I'd try pip installing it in a venv or something | |
| 23:23:16 | admin1 | 5.6.0 is the latest version, even via pip install | |
| 23:23:18 | admin1 | in a vnv | |
| 23:23:25 | admin1 | and it does not have placement | |
| 23:23:44 | melwitt | oh ugh sorry, you need the osc-placement package | |
| 23:23:51 | melwitt | it's a openstackclient plugin | |
| 23:24:01 | admin1 | hmm.. how to install it ? | |
| 23:24:11 | melwitt | I think you can just apt install it | |
| 23:24:34 | admin1 | found it | |
| 23:27:28 | admin1 | i will try tomorrow on this .. | |
| 23:27:36 | admin1 | melwitt, many thanks .. will report back tomorrow | |
| 23:28:51 | melwitt | ok yw o/ | |
| #openstack-nova - 2022-02-03 | |||
| 10:05:35 | opendevreview | yuval proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606 | |
| 10:15:24 | opendevreview | yuval proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606 | |
| 11:02:42 | opendevreview | Balazs Gibizer proposed openstack/placement master: Refactor trait normalization https://review.opendev.org/c/openstack/placement/+/825847 | |
| 11:02:43 | opendevreview | Balazs Gibizer proposed openstack/placement master: DB layer should only depend on trait id not names https://review.opendev.org/c/openstack/placement/+/826490 | |
| 11:02:43 | opendevreview | Balazs Gibizer proposed openstack/placement master: Extend the RP db query to support any-traits https://review.opendev.org/c/openstack/placement/+/825848 | |
| 11:02:44 | opendevreview | Balazs Gibizer proposed openstack/placement master: Extend the RP tree DB query to support any-traits https://review.opendev.org/c/openstack/placement/+/825849 | |
| 11:02:44 | opendevreview | Balazs Gibizer proposed openstack/placement master: Enhance doc of _get_trees_with_traits https://review.opendev.org/c/openstack/placement/+/825780 | |
| 11:03:07 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add any-traits support for listing resource providers https://review.opendev.org/c/openstack/placement/+/826491 | |
| 11:03:08 | opendevreview | Balazs Gibizer proposed openstack/placement master: Remove unused compatibility code https://review.opendev.org/c/openstack/placement/+/826493 | |
| 11:03:08 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add any-traits support for allocation candidates https://review.opendev.org/c/openstack/placement/+/826492 | |
| 11:03:22 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add microversion 1.39 to support any-trait queries https://review.opendev.org/c/openstack/placement/+/826719 | |
| 11:42:27 | opendevreview | Stephen Finucane proposed openstack/nova master: docs: Follow-ups for cells v2, architecture docs https://review.opendev.org/c/openstack/nova/+/827336 | |
| 11:56:54 | artom | I guess we'll need a tracker for the test_tagged_attachment failures :( | |
| 12:27:02 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Testing change to test_tagged_attachment in tempest https://review.opendev.org/c/openstack/nova/+/827549 | |
| 12:46:04 | gibi | artom: yes please :) | |
| 12:46:55 | gibi | artom: and thanks for trying out if switching away from q35 helps | |
| 12:47:13 | artom | gibi, yeah, just trying to gather data points | |
| 12:57:18 | gibi | artom: I will push a tempest change up that will add more logs around that curl command that is used to verify the metadata | |
| 12:57:45 | gibi | as the current code simply drops the exception and returns false so we don't see what fails | |
| 12:58:17 | artom | gibi, yeah, I think the most useful piece of information would be what the metadata API returns | |
| 12:58:21 | artom | Also, https://bugs.launchpad.net/nova/+bug/1959899 is the tracker | |
| 12:58:28 | artom | And I sent an email to the ML | |
| 12:58:55 | gibi | artom: thanks! | |
| 13:10:07 | opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: troubleshoot nova-next tagged attach test https://review.opendev.org/c/openstack/nova/+/827661 | |
| 13:25:49 | bauzas | dumb question here, but I can't find any docs about how to test our working Tempest branches with it, I guess we need to just use a venv with pip -e tempest ? | |
| 13:26:04 | bauzas | and then running tempest against our new test classes ? | |
| 13:27:19 | gibi | bauzas: you want to run tempest against a nova stable branch? | |
| 13:27:49 | bauzas | gibi: no I'm working on adding a new tempest test for https://specs.openstack.org/openstack/nova-specs/specs/yoga/approved/boot-vm-with-unaddressed-port.html#id11 | |
| 13:28:03 | bauzas | gibi: so I added a new tempest scenario and I want to test it locally | |
| 13:28:11 | bauzas | against some devstack env | |
| 13:28:31 | bauzas | so I guess I'll just create a build subdir and pip -e the source directly | |
| 13:28:40 | bauzas | with a venv | |
| 13:28:53 | gibi | so you have a devstack with the new nova code and with the new tempest code. Then | |
| 13:28:59 | gibi | tox -evenv -- tempest run --regex <new test case> | |
| 13:28:59 | bauzas | yup | |
| 13:29:19 | gibi | in the tempest source tree | |
| 13:29:20 | bauzas | oh man, I haven't thought about it | |
| 13:29:31 | bauzas | I was about to create the venv by hand | |
| 13:29:36 | gibi | :) | |