Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-18
18:08:18 mriedem i know it does if you shadow imports
18:08:32 mriedem which is helpful in cases where we pass around a thing called 'context' and have a module import of nova.context
18:09:09 artom IIRC importing nova.context you still to access it with nova.context
18:09:20 artom As opposed to from nova import context
18:10:01 mriedem i mean the latter
18:10:07 mriedem from nova import context
18:10:08 mriedem ...
18:10:23 mriedem instances = objects.Instance.get_by_instance_uuid(context, instance_uuid)
18:10:40 artom Yep, that would suck
18:11:10 mriedem we do it all over the place, but i think it's only bit us in the ass once that i know of
18:11:34 sean-k-mooney passing the module instead of a context object
18:12:09 sean-k-mooney that would have intersting sideffect if it did not explode
18:17:04 sean-k-mooney if your really unlucky and only do the assignment to the module in unit test you can end up with the tests passing or fialing dependign on the order that they are run
18:18:20 sean-k-mooney i remember helping to debug an intermitent failture in the neutron gate which was casuse by acidetally assign to the module in a test
18:20:32 mloza hello, is it safe to cleanup failed migrations(those status that are in confirmed or error) in the migration table of the nova database?
18:22:13 sean-k-mooney well those that are in confimed are not failed
18:23:04 sean-k-mooney if you mean can you clean up old migration recored in general i belive so although im not sure if they are needed for teh audit logs.
18:23:59 sean-k-mooney the main sideffect of cleaning up old migration that are confirm or error should be just they are nolonger available for quitying via the api. but maybe mriedem can double check that is correct?
18:24:27 sean-k-mooney i dont know if we have a nova manage command for that but i can see that they would build up over time
18:24:46 sean-k-mooney at least until the vms are delested in any case
18:27:53 mriedem there is no audit log for migrations, in case you're thinking of instance actions
18:27:56 mriedem which are different
18:28:21 mriedem and yeah the downside to removing complete migratoin records is losing any history on those and yeah they won't be archived/pruned until the instance is deleted,
18:28:40 sean-k-mooney yes i was trying to think of what side effect it could have
18:28:44 mriedem we don't have any nova-manage command to archive old migration records - i don't think you can even do that b/c of a foreign key reference to the instance
18:29:33 mriedem mloza: so to answer your questoin i don't think you can do that until the instances for those migrations are also deleted,
18:29:49 mriedem and if you have deleted those instances, than simply running the archive/prune commands will remove them
18:30:38 sean-k-mooney it depend i gues on how we specified the forin key constratint
18:31:10 sean-k-mooney e.g. is it set to cascase on vm deletion or is it a restict key that prevents deleteion untill the instance is deleted
18:31:24 mriedem we don't have any fkeys that do cascading deletes
18:31:49 sean-k-mooney this would be specifid in the modles.py right?
18:32:12 sean-k-mooney we had a customer delete migration recently so i think you can do it
18:32:45 mriedem it's in the models yes
18:32:50 mriedem sean-k-mooney: are you fixing that config drive functional test or what?
18:33:08 mloza sorry, I meant only status error. Just found the confirmed status are for migration and resize action
18:33:09 sean-k-mooney i pushed a version
18:33:48 mloza mriedem: ok, i'll just the migration records in the db for now
18:33:49 sean-k-mooney https://review.opendev.org/#/c/669738/9/nova/tests/functional/regressions/test_bug_1835822.py@66
18:33:54 mloza leave*
18:34:06 sean-k-mooney i added the wait for notification and kept the state change wait
18:34:15 sean-k-mooney so it cant race on still being active
18:34:24 sean-k-mooney when we start to wait
18:36:15 sean-k-mooney mloza: ok the foreign_key constratin just ensure the instance_uuid exists in the isntace table and the migration instance uuid matche and the instance is not deleted
18:36:17 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py#L805-L809
18:36:44 sean-k-mooney so it wont prevent you deleteing the old migration recored while the vm still exists
18:39:09 mloza sean-k-mooney: any sides effect the old migration record that are status error while the vm still exist?
18:40:30 sean-k-mooney the only one im aware of is that you will nolonger be able to retivie the migration info since you deleted it.
18:41:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.opendev.org/641453
18:41:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Mark "block_migration" arg deprecation on pre_live_migration method https://review.opendev.org/682963
18:41:43 mloza understand. I don't need it for auditing
18:41:45 mloza Thanks
18:41:58 mriedem don't take that as a recommendation that it's ok
18:42:08 mriedem messing with the db directly is not really a supported thing, so beware
18:42:39 mriedem especially with migration-based allocation consumers since queens i could see us (nova) relying on migration records existing to determine a type of consumer of resource allocatoins in placement
18:42:47 mriedem like in the audit command that bauzas is working on
18:42:53 mriedem mloza: ^
18:43:09 sean-k-mooney right from a downstream perspecit i think we would normally ask you to file a support exception first and assess if its safe
18:43:52 mriedem for example, let's say a migration failed and we failed to cleanup allocatoins in placement properly, then placement might be saying there are more consumers of resources than there actually are, and without some kind of audit tooling to determine that - which would use migration records - you might have a hard time sorting that all out
18:44:56 mriedem that kind of orphan issue can lead to a situation where you expect to be able to land new VMs on a host but placement filters them out saying there is no room
18:45:02 sean-k-mooney yes. our customer issue was related to FFU and failed evauctations leave evaucate migration recored in pre-migrating
18:45:31 sean-k-mooney in that case in queens we deleted there vms when the comptue agent start up
18:46:48 sean-k-mooney that said they didnt porperly check the evaucations before unfenceign the node
18:47:33 mriedem that said you didn't provide them a tool to check when it was safe to unfence the ndoe
18:47:34 mriedem *node
18:47:41 mriedem especially for the $$$ they are giving you
18:48:16 sean-k-mooney true but the db recoreds showed they used --force --host to do a nova evac to the same host
18:48:24 sean-k-mooney which is why the evacuation failed
18:49:16 sean-k-mooney you cant evacuate to the same host that you are evacuating form. we should however have an api check for that
18:49:42 sean-k-mooney this came up last week so we havent really gotten around to figuring out what needs to be done upstream yet
18:50:00 mriedem well, w/o --force the scheduler would have kicked it out
18:50:06 sean-k-mooney yep
18:50:07 mriedem as long as the compute was still disabled
18:50:19 mriedem we should probably deprecate the --force option on the evacuate cli https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-evacuate
18:50:21 sean-k-mooney if the compute was active it would have kicked it out too
18:50:39 sean-k-mooney mriedem: didnt you already do that?
18:50:53 mriedem the force parameter was deprecated in the api
18:50:58 mriedem but it's still there on earlier microversions
18:51:28 mriedem added in 2.29 and removed in 267
18:51:36 sean-k-mooney ah ok i remember talking about not supporting it in the osc version
18:51:49 mloza mriedem: I guess I hitting that orphan issue. I possibly cleared the migration records before. Now wonder I everytime I launch a new VM it goes always goes to a specific host.
18:51:49 sean-k-mooney speaking of which i should update that patch.
18:52:15 mloza What would be a remedy to this issue?
18:52:35 sean-k-mooney mloza: what release are you running
18:52:41 mloza stable/stein
18:52:57 mriedem sean-k-mooney: please tell me you ran pep8 on this https://review.opendev.org/#/c/669738/8..
18:52:58 sean-k-mooney ok i was wondering if you had the old failed builds behavior.
18:52:59 mriedem https://review.opendev.org/#/c/669738/
18:54:11 sean-k-mooney i think i did
18:54:18 sean-k-mooney i know i had to fix the odering of the imports
18:54:34 mriedem mloza: this is a work in progress but you might be able to use this https://review.opendev.org/#/c/670112/ - that would also provide valuable feedback on what it reports
18:55:38 mriedem mloza: you can use https://docs.openstack.org/osc-placement/latest/ to see what is consuming resource allocations on your nodes (resource providers)
18:55:55 mriedem i would start by investigating a particular host that you think should be available but placement is saying it's not
18:56:37 mloza will take a look
18:56:40 mriedem so using this https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-show with the --allocations option
18:57:02 mriedem the allocations are keyed by consumer uuid, which in the case of nova since queens can be migrations or instances
18:57:23 mriedem migrations for the source host during a migration, instances for the dest host
18:57:57 mriedem so let's say you evacuated all instances from a down host but placement is still reporting allocations against that host - those allocation consumers are likely migration records
18:57:59 mriedem and were orphaned
18:58:33 mriedem there are a couple of related bugs in this patch https://review.opendev.org/#/c/678100/
19:18:35 mriedem melwitt: how are you feeling about this? https://review.opendev.org/#/c/541420/ i split the refactor out and without the compat code for old cinder it's quite a bit simpler than the last time you looked.

Earlier   Later