| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-18 | |||
| 18:29:49 | mriedem | and if you have deleted those instances, than simply running the archive/prune commands will remove them | |
| 18:30:38 | sean-k-mooney | it depend i gues on how we specified the forin key constratint | |
| 18:31:10 | sean-k-mooney | e.g. is it set to cascase on vm deletion or is it a restict key that prevents deleteion untill the instance is deleted | |
| 18:31:24 | mriedem | we don't have any fkeys that do cascading deletes | |
| 18:31:49 | sean-k-mooney | this would be specifid in the modles.py right? | |
| 18:32:12 | sean-k-mooney | we had a customer delete migration recently so i think you can do it | |
| 18:32:45 | mriedem | it's in the models yes | |
| 18:32:50 | mriedem | sean-k-mooney: are you fixing that config drive functional test or what? | |
| 18:33:08 | mloza | sorry, I meant only status error. Just found the confirmed status are for migration and resize action | |
| 18:33:09 | sean-k-mooney | i pushed a version | |
| 18:33:48 | mloza | mriedem: ok, i'll just the migration records in the db for now | |
| 18:33:49 | sean-k-mooney | https://review.opendev.org/#/c/669738/9/nova/tests/functional/regressions/test_bug_1835822.py@66 | |
| 18:33:54 | mloza | leave* | |
| 18:34:06 | sean-k-mooney | i added the wait for notification and kept the state change wait | |
| 18:34:15 | sean-k-mooney | so it cant race on still being active | |
| 18:34:24 | sean-k-mooney | when we start to wait | |
| 18:36:15 | sean-k-mooney | mloza: ok the foreign_key constratin just ensure the instance_uuid exists in the isntace table and the migration instance uuid matche and the instance is not deleted | |
| 18:36:17 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py#L805-L809 | |
| 18:36:44 | sean-k-mooney | so it wont prevent you deleteing the old migration recored while the vm still exists | |
| 18:39:09 | mloza | sean-k-mooney: any sides effect the old migration record that are status error while the vm still exist? | |
| 18:40:30 | sean-k-mooney | the only one im aware of is that you will nolonger be able to retivie the migration info since you deleted it. | |
| 18:41:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.opendev.org/641453 | |
| 18:41:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Mark "block_migration" arg deprecation on pre_live_migration method https://review.opendev.org/682963 | |
| 18:41:43 | mloza | understand. I don't need it for auditing | |
| 18:41:45 | mloza | Thanks | |
| 18:41:58 | mriedem | don't take that as a recommendation that it's ok | |
| 18:42:08 | mriedem | messing with the db directly is not really a supported thing, so beware | |
| 18:42:39 | mriedem | especially with migration-based allocation consumers since queens i could see us (nova) relying on migration records existing to determine a type of consumer of resource allocatoins in placement | |
| 18:42:47 | mriedem | like in the audit command that bauzas is working on | |
| 18:42:53 | mriedem | mloza: ^ | |
| 18:43:09 | sean-k-mooney | right from a downstream perspecit i think we would normally ask you to file a support exception first and assess if its safe | |
| 18:43:52 | mriedem | for example, let's say a migration failed and we failed to cleanup allocatoins in placement properly, then placement might be saying there are more consumers of resources than there actually are, and without some kind of audit tooling to determine that - which would use migration records - you might have a hard time sorting that all out | |
| 18:44:56 | mriedem | that kind of orphan issue can lead to a situation where you expect to be able to land new VMs on a host but placement filters them out saying there is no room | |
| 18:45:02 | sean-k-mooney | yes. our customer issue was related to FFU and failed evauctations leave evaucate migration recored in pre-migrating | |
| 18:45:31 | sean-k-mooney | in that case in queens we deleted there vms when the comptue agent start up | |
| 18:46:48 | sean-k-mooney | that said they didnt porperly check the evaucations before unfenceign the node | |
| 18:47:33 | mriedem | that said you didn't provide them a tool to check when it was safe to unfence the ndoe | |
| 18:47:34 | mriedem | *node | |
| 18:47:41 | mriedem | especially for the $$$ they are giving you | |
| 18:48:16 | sean-k-mooney | true but the db recoreds showed they used --force --host to do a nova evac to the same host | |
| 18:48:24 | sean-k-mooney | which is why the evacuation failed | |
| 18:49:16 | sean-k-mooney | you cant evacuate to the same host that you are evacuating form. we should however have an api check for that | |
| 18:49:42 | sean-k-mooney | this came up last week so we havent really gotten around to figuring out what needs to be done upstream yet | |
| 18:50:00 | mriedem | well, w/o --force the scheduler would have kicked it out | |
| 18:50:06 | sean-k-mooney | yep | |
| 18:50:07 | mriedem | as long as the compute was still disabled | |
| 18:50:19 | mriedem | we should probably deprecate the --force option on the evacuate cli https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-evacuate | |
| 18:50:21 | sean-k-mooney | if the compute was active it would have kicked it out too | |
| 18:50:39 | sean-k-mooney | mriedem: didnt you already do that? | |
| 18:50:53 | mriedem | the force parameter was deprecated in the api | |
| 18:50:58 | mriedem | but it's still there on earlier microversions | |
| 18:51:28 | mriedem | added in 2.29 and removed in 267 | |
| 18:51:36 | sean-k-mooney | ah ok i remember talking about not supporting it in the osc version | |
| 18:51:49 | mloza | mriedem: I guess I hitting that orphan issue. I possibly cleared the migration records before. Now wonder I everytime I launch a new VM it goes always goes to a specific host. | |
| 18:51:49 | sean-k-mooney | speaking of which i should update that patch. | |
| 18:52:15 | mloza | What would be a remedy to this issue? | |
| 18:52:35 | sean-k-mooney | mloza: what release are you running | |
| 18:52:41 | mloza | stable/stein | |
| 18:52:57 | mriedem | sean-k-mooney: please tell me you ran pep8 on this https://review.opendev.org/#/c/669738/8.. | |
| 18:52:58 | sean-k-mooney | ok i was wondering if you had the old failed builds behavior. | |
| 18:52:59 | mriedem | https://review.opendev.org/#/c/669738/ | |
| 18:54:11 | sean-k-mooney | i think i did | |
| 18:54:18 | sean-k-mooney | i know i had to fix the odering of the imports | |
| 18:54:34 | mriedem | mloza: this is a work in progress but you might be able to use this https://review.opendev.org/#/c/670112/ - that would also provide valuable feedback on what it reports | |
| 18:55:38 | mriedem | mloza: you can use https://docs.openstack.org/osc-placement/latest/ to see what is consuming resource allocations on your nodes (resource providers) | |
| 18:55:55 | mriedem | i would start by investigating a particular host that you think should be available but placement is saying it's not | |
| 18:56:37 | mloza | will take a look | |
| 18:56:40 | mriedem | so using this https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-show with the --allocations option | |
| 18:57:02 | mriedem | the allocations are keyed by consumer uuid, which in the case of nova since queens can be migrations or instances | |
| 18:57:23 | mriedem | migrations for the source host during a migration, instances for the dest host | |
| 18:57:57 | mriedem | so let's say you evacuated all instances from a down host but placement is still reporting allocations against that host - those allocation consumers are likely migration records | |
| 18:57:59 | mriedem | and were orphaned | |
| 18:58:33 | mriedem | there are a couple of related bugs in this patch https://review.opendev.org/#/c/678100/ | |
| 19:18:35 | mriedem | melwitt: how are you feeling about this? https://review.opendev.org/#/c/541420/ i split the refactor out and without the compat code for old cinder it's quite a bit simpler than the last time you looked. | |
| 20:03:55 | KeithMnemonic1 | I have a customer running into an odd issue with multipath during migration (Pike). To make a long story short the LUNS assignments are getting mixed up somewhere. geguileo mentioned it possibly could be this https://review.opendev.org/#/c/551302/ I did a quick look at cherry-picking and it seems there are a ton of dependencies. Does that seems accurate or does someone have some magic to cherry-pick it to | |
| 20:03:55 | KeithMnemonic1 | pike? | |
| 20:15:29 | KeithMnemonic1 | mriedem, melwitt any thoughts on my question above | |
| 20:16:27 | mriedem | coincidentally that came up yesterday on pike https://review.opendev.org/#/c/670016/ | |
| 20:16:44 | mriedem | ^ will not fix your issue though | |
| 20:17:33 | mriedem | looking at the conflicts on the queens backport i'm not surprised that there would be conflicts going to pike, but w/o doing it myself i do'nt know how bad it is, | |
| 20:17:45 | mriedem | red hat probably doesn't care to backport that since i don't think they are supporting pike | |
| 20:23:47 | mriedem | efried_pto: are you going to be running the meeting tomorrow morning or should someone else? | |
| 20:39:26 | KeithMnemonic1 | mriedem i was using aspiers git-deps and it was a long list. I am not experienced enough to know if there are any shortcuts | |
| 20:48:30 | KeithMnemonic1 | I was looking to confirm with someone if my inital look was correct (that it is too cumbersome) or if it is not that bad | |
| 20:50:10 | mriedem | i can peek in a bit | |
| 20:51:07 | melwitt | mriedem: I dunno, haven't looked at it in a long time. will go through it today | |
| 20:56:37 | melwitt | KeithMnemonic1, mriedem: I can confirm that we haven't backported https://review.opendev.org/#/c/551302/ beyond queens downstream either, so I also don't know what to expect as far as how gnarly the backport will be without actually trying to do it | |
| 20:58:12 | KeithMnemonic1 | I took a look and it seemed gnarly but was hoping for a second opinion | |
| 21:00:13 | mriedem | is this a new failure in building docs locally? | |
| 21:00:13 | mriedem | WARNING: RSVG converter command 'rsvg-convert' cannot be run. Check the rsvg_converter_bin setting | |
| 21:00:59 | KeithMnemonic1 | melwitt a plug for aspiers this is what I used to gauge the effort https://github.com/aspiers/git-deps | |
| 21:04:31 | KeithMnemonic1 | but that pulls in every dependency and i am not experienced enough to know if any could be skipped | |
| 21:05:43 | mriedem | ah i see https://github.com/openstack/nova/commit/16b9486bf7e91bfd5dc48297cee9f54b49156c93 | |
| 21:15:19 | mriedem | we need librsvg2-bin in bindep.txt | |
| 21:19:33 | mriedem | KeithMnemonic1: do you have a paste of the dependencies? i wouldn't be surprised if a bunch of the conflicts are due to like mox removal patches | |
| 21:28:11 | mriedem | melwitt: if you have a fedora or centos system available can you confirm that librsvg2-tools provides the rsvg-convert command? | |
| 21:28:53 | mriedem | which is i think just: yum provides rsvg-convert | |
| 21:28:53 | mriedem | right? | |
| 21:29:13 | melwitt | let me check | |
| 21:31:23 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Re-propose cross-cell-resize spec for Ussuri https://review.opendev.org/683002 | |