Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-02-02
18:57:33 sean-k-mooney its doing it in a loop
18:57:49 sean-k-mooney but by the time the test complete the data is not in the metadata servie
18:58:00 sean-k-mooney but that could be related to the attach taking a long time
19:00:37 artom It's not 100% though, so whatever we try will have to be rechecked at least a few times
19:01:42 sean-k-mooney ya i dont know it just gotten flaky recently
19:01:52 sean-k-mooney not clear reason why
19:02:09 sean-k-mooney and as you say its no 100% so its hard to tell why
19:11:21 artom I wonder if we should wait for volume and interface attach before carrying on
19:11:37 artom Like, we're somehow "confusing" the guest by issuing two device attach commands in quick succession
19:11:46 artom I realize how non-engineery that sounds
19:22:32 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: Testing change to test_tagged_attachment in tempest https://review.opendev.org/c/openstack/nova/+/827549
19:22:38 artom ^^ we'll see
19:51:28 sean-k-mooney artom: we proably should although i think we have an instace level lock at the comptue manager so only one attachment can happen at a time
19:51:51 sean-k-mooney we do for 2 interfaces or volumes but not sure about one of each
19:52:33 sean-k-mooney so ya lets see if that helps
19:54:14 sean-k-mooney o/ all chat to ye tomrrow
19:54:20 artom It's a shot in the dark to serve as a data poin
19:54:21 artom t
21:19:49 admin1 hi .. is there a nova command to see all ongoing migrations ?
21:23:09 mloza hello, need help figure out what would cause nova to delete a port on an instance
21:23:17 mloza i see this in the logs `Creating event network-vif-deleted`
21:23:32 mloza here's the full logs https://paste.openstack.org/raw/bRTqxJGwy3WxAPGVEPo9/
21:52:06 admin1 openstack compute service delete 56 => Unable to delete compute service that has in-progress migrations. .. How do I check these migrations ?
22:08:34 melwitt admin1: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-migration.html#server-migration-list
22:10:16 admin1 melwitt, this cluster is still in rocky .. so that command is not there
22:10:36 admin1 we are in the process of upgrading it .. so migrating, delete compute, reinstall and upgrade, add it back
22:11:02 opendevreview melanie witt proposed openstack/nova master: Raise InstanceNotFound on fkey constraint fail saving info cache https://review.opendev.org/c/openstack/nova/+/826942
22:11:02 melwitt oh I see
22:18:09 admin1 nova migration-list lists the migraitons, but there is nothing pending or ongoing
22:23:41 melwitt you could try the force or abort commands on the migrations if they are leftover https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-force-complete
22:23:43 melwitt https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort
22:34:12 admin1 i found the instance.. the instance is in pre-migrating status .... when i enter the command nova live-migration-abort $UUID $ID, it says Instance $UUID is in an invalid state for 'abort live migration'
22:34:32 admin1 how do I abort a pre-migration status ?
22:35:09 admin1 server show status is Running .. so instance does not have a pre-migrating status
22:38:14 melwitt did you try the force complete too?
22:40:15 admin1 yeah .. it gave Instance $UUID is in an invalid state for 'force_complete'
22:41:02 admin1 maybe i can just delete the compute service from the db ?
22:41:06 admin1 and add it again ..
22:41:20 admin1 want to do it properly though ..
22:44:32 admin1 melwitt, https://gist.github.com/a1git/9a975b96cd91da4683084c7df3220530
22:45:43 admin1 baiscally the way i have been upgrading from rocky (xenial ) -> bionic is .. for example, migrate all instances from A - B and empty A, then compute service delete A , reinstall A with bionic, same hostname, install nova .. , and then repeat the process again ..
22:45:52 admin1 but for some reasons, this one is locked/blocked ..
22:46:01 melwitt +1 going into the db is the last resort if the proper tools don't work
22:48:21 admin1 i think 'pre-migrating' is blocking the deletion .. and I have found nothing to force this to either error or completed
22:51:41 admin1 i did a mysqldump and i found the pre-migrating in only 1 table .. nova.migrations ..
22:53:28 melwitt I think you are right that it's the migration(s) blocking the service delete
22:54:01 admin1 update migrations set status='error' where instance_uuid=$UUID and status='pre-migrating';
22:54:15 admin1 and then the delet worked fine :) .. openstack compute service delete 56
22:56:29 melwitt I was just about to suggest that, change the status rather than deleting the migration record :)
23:03:27 admin1 how to delete a host from placement service ?
23:04:49 melwitt hm, the service delete should have done that
23:05:03 melwitt it's the host you deleted the service for right?
23:06:18 admin1 ResourceProviderCreationFailed: Failed to create resource provider h1
23:07:24 admin1 openstack compute service list -- it gets added there ..
23:08:09 admin1 i tried via "openstack compute service delete $id" and in 2nd attempt nova service-delete $uuid
23:08:25 admin1 4 other servers, no issues .. this one hypervisor = this and that errors :)
23:08:54 melwitt this is the cli for the placement service https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-delete
23:10:16 melwitt just be careful and make sure it's not associated with any instances allocations
23:10:21 melwitt before deleting it
23:10:26 admin1 there are no instances
23:10:38 melwitt k
23:10:43 admin1 all instances were migrated, this is a new install which just came up
23:10:50 melwitt ok cool
23:10:52 admin1 no such command in rocky
23:11:20 admin1 this upgrade is necessary to upgrade from rocky =.. as stein needs 18.04 ..
23:11:48 admin1 if i need to delete a hypervisor, isn't it just deleting compute service ?
23:12:19 melwitt yeah but it does a few things: deletes the nova service record, the nova compute_nodes record, and then the placement resource provider
23:12:45 melwitt and the last step has (probably) failed in your case from the previous delete
23:13:15 melwitt so it sees a duplicate placement resource provider and refuses to create it. so you are correct you need to delete it so it can recreate it again
23:13:38 melwitt as for the command, you could upgrade openstackclient to newer than rocky and it will still work with rocky
23:18:23 admin1 sudo apt install python3-openstackclient python3-openstackclient is already the newest version (5.6.0-0ubuntu1). ... no such command
23:19:07 melwitt yeah I'd try pip installing it in a venv or something
23:23:16 admin1 5.6.0 is the latest version, even via pip install
23:23:18 admin1 in a vnv
23:23:25 admin1 and it does not have placement
23:23:44 melwitt oh ugh sorry, you need the osc-placement package
23:23:51 melwitt it's a openstackclient plugin
23:24:01 admin1 hmm.. how to install it ?
23:24:11 melwitt I think you can just apt install it
23:24:34 admin1 found it
23:27:28 admin1 i will try tomorrow on this ..
23:27:36 admin1 melwitt, many thanks .. will report back tomorrow
23:28:51 melwitt ok yw o/
#openstack-nova - 2022-02-03
10:05:35 opendevreview yuval proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606
10:15:24 opendevreview yuval proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606
11:02:42 opendevreview Balazs Gibizer proposed openstack/placement master: Refactor trait normalization https://review.opendev.org/c/openstack/placement/+/825847
11:02:43 opendevreview Balazs Gibizer proposed openstack/placement master: DB layer should only depend on trait id not names https://review.opendev.org/c/openstack/placement/+/826490
11:02:43 opendevreview Balazs Gibizer proposed openstack/placement master: Extend the RP db query to support any-traits https://review.opendev.org/c/openstack/placement/+/825848
11:02:44 opendevreview Balazs Gibizer proposed openstack/placement master: Extend the RP tree DB query to support any-traits https://review.opendev.org/c/openstack/placement/+/825849
11:02:44 opendevreview Balazs Gibizer proposed openstack/placement master: Enhance doc of _get_trees_with_traits https://review.opendev.org/c/openstack/placement/+/825780
11:03:07 opendevreview Balazs Gibizer proposed openstack/placement master: Add any-traits support for listing resource providers https://review.opendev.org/c/openstack/placement/+/826491
11:03:08 opendevreview Balazs Gibizer proposed openstack/placement master: Remove unused compatibility code https://review.opendev.org/c/openstack/placement/+/826493
11:03:08 opendevreview Balazs Gibizer proposed openstack/placement master: Add any-traits support for allocation candidates https://review.opendev.org/c/openstack/placement/+/826492
11:03:22 opendevreview Balazs Gibizer proposed openstack/placement master: Add microversion 1.39 to support any-trait queries https://review.opendev.org/c/openstack/placement/+/826719
11:42:27 opendevreview Stephen Finucane proposed openstack/nova master: docs: Follow-ups for cells v2, architecture docs https://review.opendev.org/c/openstack/nova/+/827336
11:56:54 artom I guess we'll need a tracker for the test_tagged_attachment failures :(
12:27:02 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: Testing change to test_tagged_attachment in tempest https://review.opendev.org/c/openstack/nova/+/827549
12:46:04 gibi artom: yes please :)
12:46:55 gibi artom: and thanks for trying out if switching away from q35 helps
12:47:13 artom gibi, yeah, just trying to gather data points

Earlier   Later