| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-02 | |||
| 13:12:10 | gibi | ralonsoh: I'm not sure I understand the reason of the wontfix on https://bugs.launchpad.net/neutron/+bug/1959749 | |
| 13:16:06 | gibi | ralonsoh: you #1) point is what I would need to work. So that the rule type list returns all rule types not just rule types that are supported by every configured driver | |
| 13:32:09 | ralonsoh | gibi, sorry, I don't know why I set this flag | |
| 13:32:25 | ralonsoh | confirmed, this is the correct value | |
| 13:32:37 | gibi | ralonsoh: that is better, thanks :) | |
| 13:48:09 | admin1 | hi guys .. is there a way to "transfer ownership" of an instance from one project to another ? | |
| 14:04:57 | opendevreview | yuval proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606 | |
| 14:58:51 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Document remote-managed port usage considerations https://review.opendev.org/c/openstack/nova/+/827513 | |
| 16:45:59 | gibi | if somebody wants a change of (code) scenery then I can suggest looking at the placement code review series to support any-traits queries in microvarsion 1.39. The series starts here https://review.opendev.org/c/openstack/placement/+/825846/3 :) | |
| 17:01:15 | opendevreview | Merged openstack/nova master: docs: Add a new cells v2 document https://review.opendev.org/c/openstack/nova/+/814562 | |
| 17:15:10 | melwitt | gibi: I will look at some new code scenery :) | |
| 17:27:53 | dmitriis | gibi, sean-k-mooney: mostly been getting unrelated gate failures so I am waiting for some rechecks to complete. I made a functional change to the patch that introduces the remote_managed tag here https://review.opendev.org/c/openstack/nova/+/824834/8/nova/pci/devspec.py#322 to include a check for the presence of a serial number when a device is | |
| 17:27:53 | dmitriis | tagged as remote_managed which is something I overlooked in the previous iteration and updated testing to reflect that. | |
| 17:28:38 | dmitriis | I started working on the docs and started a doc review but most of the docs will be in Neutron under the OVN driver guide similar to how it's done today with OVS hardware offload. | |
| 17:42:44 | gibi | melwitt: thanks! :) | |
| 17:43:12 | gibi | dmitriis: I will read back tomorrow I have to go now | |
| 17:43:32 | dmitriis | gibi: np, thanks a lot for the help so far | |
| 18:13:35 | sean-k-mooney | dmitriis: ack | |
| 18:14:07 | sean-k-mooney | dmitriis: im looking at some downstream stuff currently but ill try to take a look proably tomorow at this point | |
| 18:14:48 | sean-k-mooney | dmitriis: most of the doc make sense for the neutorn guide but we shoudl detail how to use the remote managed flag ectra in nova | |
| 18:16:56 | dmitriis | sean-k-mooney: ack, ta. | |
| 18:16:57 | dmitriis | sean-k-mooney: I currently describe some of it in the latest doc change and reference the option docstring but I can expand the description in the docs themselves too. | |
| 18:18:36 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/805649 | |
| 18:34:13 | opendevreview | melanie witt proposed openstack/nova master: Raise InstanceNotFound on fkey constraint fail saving info cache https://review.opendev.org/c/openstack/nova/+/826942 | |
| 18:51:41 | sean-k-mooney | o/ are we tracking the failure of tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment | |
| 18:51:49 | sean-k-mooney | as a potential gate issue anywhere | |
| 18:52:05 | sean-k-mooney | im seeing that fail more and more on reviews over the last 2 weeks | |
| 18:52:32 | sean-k-mooney | its like it knew lyarwood was starting on kubvirt this week :) | |
| 18:54:03 | sean-k-mooney | so this could be q35 related | |
| 18:54:11 | artom | Yeah, repeating what I said downstream... it's not even a Tempest race or whatever, it's the guest itself. Is this the q35 problem again? Surely we'd see other tests fail in that job, unless nova-next doesn't do any other device attachment tests, which would be... weird | |
| 18:54:12 | sean-k-mooney | but im not sure about that | |
| 18:54:30 | sean-k-mooney | it could be that the volume is not attched fully yet | |
| 18:54:48 | sean-k-mooney | i.e. the series from lee to wait fothe vm to be pingable might help | |
| 18:54:57 | sean-k-mooney | but in this case the test is sshing into the vm | |
| 18:55:04 | sean-k-mooney | to check the tag is there right | |
| 18:55:34 | sean-k-mooney | the failure message at the top level is Details: Timeout while verifying metadata on server. | |
| 18:55:55 | artom | No, there are definitely other tests that attach stuff that pass | |
| 18:56:29 | sean-k-mooney | so the test is doing Remote command: set -eu -o pipefail; PATH=$PATH:/sbin:/usr/sbin; curl http://169.254.169.254/openstack/latest/meta_data.json | |
| 18:56:29 | artom | Mind you, they may not be SSH'ing into the guest? | |
| 18:56:42 | sean-k-mooney | by sshing into the guest | |
| 18:56:54 | sean-k-mooney | and that ssh connection is well connecting | |
| 18:57:13 | artom | So it's curl that's timing out? | |
| 18:57:17 | sean-k-mooney | im looking at https://zuul.opendev.org/t/openstack/build/d836724c364843e98bf893ac71574828 | |
| 18:57:27 | sean-k-mooney | no i think the curl command is working | |
| 18:57:33 | sean-k-mooney | its doing it in a loop | |
| 18:57:49 | sean-k-mooney | but by the time the test complete the data is not in the metadata servie | |
| 18:58:00 | sean-k-mooney | but that could be related to the attach taking a long time | |
| 19:00:37 | artom | It's not 100% though, so whatever we try will have to be rechecked at least a few times | |
| 19:01:42 | sean-k-mooney | ya i dont know it just gotten flaky recently | |
| 19:01:52 | sean-k-mooney | not clear reason why | |
| 19:02:09 | sean-k-mooney | and as you say its no 100% so its hard to tell why | |
| 19:11:21 | artom | I wonder if we should wait for volume and interface attach before carrying on | |
| 19:11:37 | artom | Like, we're somehow "confusing" the guest by issuing two device attach commands in quick succession | |
| 19:11:46 | artom | I realize how non-engineery that sounds | |
| 19:22:32 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Testing change to test_tagged_attachment in tempest https://review.opendev.org/c/openstack/nova/+/827549 | |
| 19:22:38 | artom | ^^ we'll see | |
| 19:51:28 | sean-k-mooney | artom: we proably should although i think we have an instace level lock at the comptue manager so only one attachment can happen at a time | |
| 19:51:51 | sean-k-mooney | we do for 2 interfaces or volumes but not sure about one of each | |
| 19:52:33 | sean-k-mooney | so ya lets see if that helps | |
| 19:54:14 | sean-k-mooney | o/ all chat to ye tomrrow | |
| 19:54:20 | artom | It's a shot in the dark to serve as a data poin | |
| 19:54:21 | artom | t | |
| 21:19:49 | admin1 | hi .. is there a nova command to see all ongoing migrations ? | |
| 21:23:09 | mloza | hello, need help figure out what would cause nova to delete a port on an instance | |
| 21:23:17 | mloza | i see this in the logs `Creating event network-vif-deleted` | |
| 21:23:32 | mloza | here's the full logs https://paste.openstack.org/raw/bRTqxJGwy3WxAPGVEPo9/ | |
| 21:52:06 | admin1 | openstack compute service delete 56 => Unable to delete compute service that has in-progress migrations. .. How do I check these migrations ? | |
| 22:08:34 | melwitt | admin1: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-migration.html#server-migration-list | |
| 22:10:16 | admin1 | melwitt, this cluster is still in rocky .. so that command is not there | |
| 22:10:36 | admin1 | we are in the process of upgrading it .. so migrating, delete compute, reinstall and upgrade, add it back | |
| 22:11:02 | melwitt | oh I see | |
| 22:11:02 | opendevreview | melanie witt proposed openstack/nova master: Raise InstanceNotFound on fkey constraint fail saving info cache https://review.opendev.org/c/openstack/nova/+/826942 | |
| 22:18:09 | admin1 | nova migration-list lists the migraitons, but there is nothing pending or ongoing | |
| 22:23:41 | melwitt | you could try the force or abort commands on the migrations if they are leftover https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-force-complete | |
| 22:23:43 | melwitt | https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-live-migration-abort | |
| 22:34:12 | admin1 | i found the instance.. the instance is in pre-migrating status .... when i enter the command nova live-migration-abort $UUID $ID, it says Instance $UUID is in an invalid state for 'abort live migration' | |
| 22:34:32 | admin1 | how do I abort a pre-migration status ? | |
| 22:35:09 | admin1 | server show status is Running .. so instance does not have a pre-migrating status | |
| 22:38:14 | melwitt | did you try the force complete too? | |
| 22:40:15 | admin1 | yeah .. it gave Instance $UUID is in an invalid state for 'force_complete' | |
| 22:41:02 | admin1 | maybe i can just delete the compute service from the db ? | |
| 22:41:06 | admin1 | and add it again .. | |
| 22:41:20 | admin1 | want to do it properly though .. | |
| 22:44:32 | admin1 | melwitt, https://gist.github.com/a1git/9a975b96cd91da4683084c7df3220530 | |
| 22:45:43 | admin1 | baiscally the way i have been upgrading from rocky (xenial ) -> bionic is .. for example, migrate all instances from A - B and empty A, then compute service delete A , reinstall A with bionic, same hostname, install nova .. , and then repeat the process again .. | |
| 22:45:52 | admin1 | but for some reasons, this one is locked/blocked .. | |
| 22:46:01 | melwitt | +1 going into the db is the last resort if the proper tools don't work | |
| 22:48:21 | admin1 | i think 'pre-migrating' is blocking the deletion .. and I have found nothing to force this to either error or completed | |
| 22:51:41 | admin1 | i did a mysqldump and i found the pre-migrating in only 1 table .. nova.migrations .. | |
| 22:53:28 | melwitt | I think you are right that it's the migration(s) blocking the service delete | |
| 22:54:01 | admin1 | update migrations set status='error' where instance_uuid=$UUID and status='pre-migrating'; | |
| 22:54:15 | admin1 | and then the delet worked fine :) .. openstack compute service delete 56 | |
| 22:56:29 | melwitt | I was just about to suggest that, change the status rather than deleting the migration record :) | |
| 23:03:27 | admin1 | how to delete a host from placement service ? | |
| 23:04:49 | melwitt | hm, the service delete should have done that | |
| 23:05:03 | melwitt | it's the host you deleted the service for right? | |
| 23:06:18 | admin1 | ResourceProviderCreationFailed: Failed to create resource provider h1 | |
| 23:07:24 | admin1 | openstack compute service list -- it gets added there .. | |
| 23:08:09 | admin1 | i tried via "openstack compute service delete $id" and in 2nd attempt nova service-delete $uuid | |
| 23:08:25 | admin1 | 4 other servers, no issues .. this one hypervisor = this and that errors :) | |