| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-18 | |||
| 18:09:20 | artom | As opposed to from nova import context | |
| 18:10:01 | mriedem | i mean the latter | |
| 18:10:07 | mriedem | from nova import context | |
| 18:10:08 | mriedem | ... | |
| 18:10:23 | mriedem | instances = objects.Instance.get_by_instance_uuid(context, instance_uuid) | |
| 18:10:40 | artom | Yep, that would suck | |
| 18:11:10 | mriedem | we do it all over the place, but i think it's only bit us in the ass once that i know of | |
| 18:11:34 | sean-k-mooney | passing the module instead of a context object | |
| 18:12:09 | sean-k-mooney | that would have intersting sideffect if it did not explode | |
| 18:17:04 | sean-k-mooney | if your really unlucky and only do the assignment to the module in unit test you can end up with the tests passing or fialing dependign on the order that they are run | |
| 18:18:20 | sean-k-mooney | i remember helping to debug an intermitent failture in the neutron gate which was casuse by acidetally assign to the module in a test | |
| 18:20:32 | mloza | hello, is it safe to cleanup failed migrations(those status that are in confirmed or error) in the migration table of the nova database? | |
| 18:22:13 | sean-k-mooney | well those that are in confimed are not failed | |
| 18:23:04 | sean-k-mooney | if you mean can you clean up old migration recored in general i belive so although im not sure if they are needed for teh audit logs. | |
| 18:23:59 | sean-k-mooney | the main sideffect of cleaning up old migration that are confirm or error should be just they are nolonger available for quitying via the api. but maybe mriedem can double check that is correct? | |
| 18:24:27 | sean-k-mooney | i dont know if we have a nova manage command for that but i can see that they would build up over time | |
| 18:24:46 | sean-k-mooney | at least until the vms are delested in any case | |
| 18:27:53 | mriedem | there is no audit log for migrations, in case you're thinking of instance actions | |
| 18:27:56 | mriedem | which are different | |
| 18:28:21 | mriedem | and yeah the downside to removing complete migratoin records is losing any history on those and yeah they won't be archived/pruned until the instance is deleted, | |
| 18:28:40 | sean-k-mooney | yes i was trying to think of what side effect it could have | |
| 18:28:44 | mriedem | we don't have any nova-manage command to archive old migration records - i don't think you can even do that b/c of a foreign key reference to the instance | |
| 18:29:33 | mriedem | mloza: so to answer your questoin i don't think you can do that until the instances for those migrations are also deleted, | |
| 18:29:49 | mriedem | and if you have deleted those instances, than simply running the archive/prune commands will remove them | |
| 18:30:38 | sean-k-mooney | it depend i gues on how we specified the forin key constratint | |
| 18:31:10 | sean-k-mooney | e.g. is it set to cascase on vm deletion or is it a restict key that prevents deleteion untill the instance is deleted | |
| 18:31:24 | mriedem | we don't have any fkeys that do cascading deletes | |
| 18:31:49 | sean-k-mooney | this would be specifid in the modles.py right? | |
| 18:32:12 | sean-k-mooney | we had a customer delete migration recently so i think you can do it | |
| 18:32:45 | mriedem | it's in the models yes | |
| 18:32:50 | mriedem | sean-k-mooney: are you fixing that config drive functional test or what? | |
| 18:33:08 | mloza | sorry, I meant only status error. Just found the confirmed status are for migration and resize action | |
| 18:33:09 | sean-k-mooney | i pushed a version | |
| 18:33:48 | mloza | mriedem: ok, i'll just the migration records in the db for now | |
| 18:33:49 | sean-k-mooney | https://review.opendev.org/#/c/669738/9/nova/tests/functional/regressions/test_bug_1835822.py@66 | |
| 18:33:54 | mloza | leave* | |
| 18:34:06 | sean-k-mooney | i added the wait for notification and kept the state change wait | |
| 18:34:15 | sean-k-mooney | so it cant race on still being active | |
| 18:34:24 | sean-k-mooney | when we start to wait | |
| 18:36:15 | sean-k-mooney | mloza: ok the foreign_key constratin just ensure the instance_uuid exists in the isntace table and the migration instance uuid matche and the instance is not deleted | |
| 18:36:17 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py#L805-L809 | |
| 18:36:44 | sean-k-mooney | so it wont prevent you deleteing the old migration recored while the vm still exists | |
| 18:39:09 | mloza | sean-k-mooney: any sides effect the old migration record that are status error while the vm still exist? | |
| 18:40:30 | sean-k-mooney | the only one im aware of is that you will nolonger be able to retivie the migration info since you deleted it. | |
| 18:41:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Mark "block_migration" arg deprecation on pre_live_migration method https://review.opendev.org/682963 | |
| 18:41:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.opendev.org/641453 | |
| 18:41:43 | mloza | understand. I don't need it for auditing | |
| 18:41:45 | mloza | Thanks | |
| 18:41:58 | mriedem | don't take that as a recommendation that it's ok | |
| 18:42:08 | mriedem | messing with the db directly is not really a supported thing, so beware | |
| 18:42:39 | mriedem | especially with migration-based allocation consumers since queens i could see us (nova) relying on migration records existing to determine a type of consumer of resource allocatoins in placement | |
| 18:42:47 | mriedem | like in the audit command that bauzas is working on | |
| 18:42:53 | mriedem | mloza: ^ | |
| 18:43:09 | sean-k-mooney | right from a downstream perspecit i think we would normally ask you to file a support exception first and assess if its safe | |
| 18:43:52 | mriedem | for example, let's say a migration failed and we failed to cleanup allocatoins in placement properly, then placement might be saying there are more consumers of resources than there actually are, and without some kind of audit tooling to determine that - which would use migration records - you might have a hard time sorting that all out | |
| 18:44:56 | mriedem | that kind of orphan issue can lead to a situation where you expect to be able to land new VMs on a host but placement filters them out saying there is no room | |
| 18:45:02 | sean-k-mooney | yes. our customer issue was related to FFU and failed evauctations leave evaucate migration recored in pre-migrating | |
| 18:45:31 | sean-k-mooney | in that case in queens we deleted there vms when the comptue agent start up | |
| 18:46:48 | sean-k-mooney | that said they didnt porperly check the evaucations before unfenceign the node | |
| 18:47:33 | mriedem | that said you didn't provide them a tool to check when it was safe to unfence the ndoe | |
| 18:47:34 | mriedem | *node | |
| 18:47:41 | mriedem | especially for the $$$ they are giving you | |
| 18:48:16 | sean-k-mooney | true but the db recoreds showed they used --force --host to do a nova evac to the same host | |
| 18:48:24 | sean-k-mooney | which is why the evacuation failed | |
| 18:49:16 | sean-k-mooney | you cant evacuate to the same host that you are evacuating form. we should however have an api check for that | |
| 18:49:42 | sean-k-mooney | this came up last week so we havent really gotten around to figuring out what needs to be done upstream yet | |
| 18:50:00 | mriedem | well, w/o --force the scheduler would have kicked it out | |
| 18:50:06 | sean-k-mooney | yep | |
| 18:50:07 | mriedem | as long as the compute was still disabled | |
| 18:50:19 | mriedem | we should probably deprecate the --force option on the evacuate cli https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-evacuate | |
| 18:50:21 | sean-k-mooney | if the compute was active it would have kicked it out too | |
| 18:50:39 | sean-k-mooney | mriedem: didnt you already do that? | |
| 18:50:53 | mriedem | the force parameter was deprecated in the api | |
| 18:50:58 | mriedem | but it's still there on earlier microversions | |
| 18:51:28 | mriedem | added in 2.29 and removed in 267 | |
| 18:51:36 | sean-k-mooney | ah ok i remember talking about not supporting it in the osc version | |
| 18:51:49 | sean-k-mooney | speaking of which i should update that patch. | |
| 18:51:49 | mloza | mriedem: I guess I hitting that orphan issue. I possibly cleared the migration records before. Now wonder I everytime I launch a new VM it goes always goes to a specific host. | |
| 18:52:15 | mloza | What would be a remedy to this issue? | |
| 18:52:35 | sean-k-mooney | mloza: what release are you running | |
| 18:52:41 | mloza | stable/stein | |
| 18:52:57 | mriedem | sean-k-mooney: please tell me you ran pep8 on this https://review.opendev.org/#/c/669738/8.. | |
| 18:52:58 | sean-k-mooney | ok i was wondering if you had the old failed builds behavior. | |
| 18:52:59 | mriedem | https://review.opendev.org/#/c/669738/ | |
| 18:54:11 | sean-k-mooney | i think i did | |
| 18:54:18 | sean-k-mooney | i know i had to fix the odering of the imports | |
| 18:54:34 | mriedem | mloza: this is a work in progress but you might be able to use this https://review.opendev.org/#/c/670112/ - that would also provide valuable feedback on what it reports | |
| 18:55:38 | mriedem | mloza: you can use https://docs.openstack.org/osc-placement/latest/ to see what is consuming resource allocations on your nodes (resource providers) | |
| 18:55:55 | mriedem | i would start by investigating a particular host that you think should be available but placement is saying it's not | |
| 18:56:37 | mloza | will take a look | |
| 18:56:40 | mriedem | so using this https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-show with the --allocations option | |
| 18:57:02 | mriedem | the allocations are keyed by consumer uuid, which in the case of nova since queens can be migrations or instances | |
| 18:57:23 | mriedem | migrations for the source host during a migration, instances for the dest host | |
| 18:57:57 | mriedem | so let's say you evacuated all instances from a down host but placement is still reporting allocations against that host - those allocation consumers are likely migration records | |
| 18:57:59 | mriedem | and were orphaned | |
| 18:58:33 | mriedem | there are a couple of related bugs in this patch https://review.opendev.org/#/c/678100/ | |
| 19:18:35 | mriedem | melwitt: how are you feeling about this? https://review.opendev.org/#/c/541420/ i split the refactor out and without the compat code for old cinder it's quite a bit simpler than the last time you looked. | |
| 20:03:55 | KeithMnemonic1 | pike? | |
| 20:03:55 | KeithMnemonic1 | I have a customer running into an odd issue with multipath during migration (Pike). To make a long story short the LUNS assignments are getting mixed up somewhere. geguileo mentioned it possibly could be this https://review.opendev.org/#/c/551302/ I did a quick look at cherry-picking and it seems there are a ton of dependencies. Does that seems accurate or does someone have some magic to cherry-pick it to | |
| 20:15:29 | KeithMnemonic1 | mriedem, melwitt any thoughts on my question above | |