Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-10-11
22:04:45 mriedem so i think we're ok
22:05:37 mriedem and i think we should probably add 'done' to https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L4863 in some other unrelated change, probably just because someone forgot about it before, but 'done' is definitely not in-progres
22:05:40 dansmith mriedem: so, we _are_ passing the functional tests that cover evac.. the only thing that might be missing is source compute coming back or something
22:06:07 mriedem i know we have some functional tests for ^ because they are asserting the source removes the allocations from the source after it's moved
22:06:36 dansmith I'm not really sure why accepted should be in there, tbh
22:06:42 dansmith since it's very much in progress
22:06:54 mriedem https://github.com/openstack/nova/blob/master/nova/tests/functional/test_servers.py#L1658
22:07:15 mriedem yeah not sure about accepted being in there either
22:07:33 mriedem so ^ is the test that once you have evacuated to the dest and start up the source, the allocations are removed from the source
22:07:44 dansmith ah yeah okay
22:07:51 mriedem which is actually currently broken :)
22:08:05 mriedem https://review.openstack.org/#/c/510176/
22:08:13 mriedem mayhap you'd like to +W that
22:08:28 mriedem our code to remove the source allocations on startup after evac is not working
22:09:17 dansmith so explain to me again how you can evacuate from a source, go to error, and then start back up on the source successfully?
22:09:41 dansmith we should be caught by the vm_state==ERROR skip clause no?
22:10:26 SamYaple if its in an errored state and you reset it to state active then do a hard-reboot on it (assuming shared storage) it should come back
22:11:46 dansmith SamYaple: yeah, reset-state means you're cheating and other things are likely going to be out of sync
22:12:26 SamYaple dansmith: fair enough. im just saying what would typically get done in ops
22:12:32 SamYaple im not advocating for this
22:12:45 dansmith SamYaple: ack, yeah, I was thinking mriedem was saying there's a path out of the forest for this by a user
22:12:50 mriedem dansmith: do we set the instance to error state on failed evac?
22:13:00 dansmith mriedem: if we didn't you couldn't call evac on it again
22:13:09 dansmith and you said we could
22:13:28 mriedem i don't see that we change the vm_state on evac
22:13:36 dansmith so we leave it as rebuilding?
22:13:47 mriedem no
22:13:48 mriedem def evacuate(self, context, instance, host, on_shared_storage,
22:13:48 mriedem vm_states.ERROR])
22:13:48 mriedem @check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.STOPPED,
22:13:56 mriedem i think it would be active
22:14:05 mriedem well, on a successful evac it would be active
22:14:18 mriedem if it started as error and evac failed, it'd stay in error
22:14:26 dansmith oh task_state=rebuilding
22:14:29 mriedem yar
22:14:29 dansmith so um
22:14:40 dansmith that means I can re-evac while an evac is going?
22:14:45 dansmith that'd be fun
22:14:49 mriedem no
22:15:04 mriedem i think @check_instance_state defaults to not allow doing things if the task_state is set
22:15:18 mriedem and we set the task_state to rebuilding in the api once you've started an evac
22:15:21 dansmith does that decorator imply no task state if not specified?
22:15:22 dansmith okay
22:15:53 mriedem yeah it's weird and confusing to read
22:15:57 mriedem task_state=(None,)
22:16:02 mriedem instance.task_state not in task_state):
22:16:02 mriedem if (task_state is not None and
22:16:17 dansmith okay
22:16:57 mriedem so unrelated,
22:17:12 mriedem but now i realize that if your instance was stopped and it was successfully evacuated, it's now running
22:17:31 mriedem we don't stop it again like we do for resizing/cold migrating a stopped instance
22:17:57 mriedem which, as it turns out, was my first major bug fix in nova :)
22:18:00 mriedem remember?!
22:18:38 dansmith oh I remember
22:20:00 dansmith okay so nothing to change on this?
22:22:36 dansmith also I'd like to note that this migration unfuckification has been the least fun thing to work on since, oh I dunno, the great flavor migration of 2015
22:22:50 mriedem yeah i just +Wed
22:23:37 mriedem appears CI just exploded
22:25:48 dansmith we _can_ do a migration-holding source allocation if you want,
22:26:23 dansmith but it's work for no good reason, IMHO,
22:26:33 mriedem yeah, that's why i said if/when we need to
22:26:43 dansmith okay
22:26:55 mriedem because that would mean the cleanup if source comes back up thing would have to deal with whether or not the allocation is on the migration or not
22:27:03 dansmith right
22:27:20 mriedem and that would definitely be
22:27:23 mriedem say it with me now
22:27:30 dansmith I'm a little concerned that we have this reset-state-hard-reboot path to keep these on the source even after an evac has been triggered once
22:28:15 mriedem wonky
22:28:19 dansmith heh
22:28:20 mriedem it would be wonky
22:35:52 mriedem wtf, why do we joinedload instance.system_metadata when pulling migration records out of the db? https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L4867
22:40:25 dansmith because migration.instance.system_metadata
22:40:32 dansmith although I'm sure we don't need that anymore
22:41:24 mriedem you know what else is super confusing,
22:41:29 mriedem we have a 'finished' migration status,
22:41:47 mriedem but it doesn't mean the migration is actually done, it means the instance is done being resized, but it's in verify_resize state
22:41:52 mriedem so it's not an end state :/
22:42:00 dansmith yup
22:43:57 mriedem oh, an
22:43:58 mriedem *and
22:44:03 mriedem when reverting a resize,
22:44:17 mriedem we set the migration status to 'reverted' on the dest host before casting back to the source host to complete the revert
22:44:27 mriedem so technically the migration is still in progress at that point
22:44:49 mriedem i sure hope nothing like resource tracking depends on us not fucking that all up :)
22:50:19 mriedem check out this fun stuff http://paste.openstack.org/show/623387/
22:50:39 mriedem creating rp inventory on the fly like a mad dog
23:05:27 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix migration_get_in_progress_by_host_and_node status list https://review.openstack.org/511351
23:10:46 openstackgerrit Eric Fried proposed openstack/nova-specs master: Granular Resource Request Syntax https://review.openstack.org/510244
23:12:38 efried Man Jenkins is a mess.
23:35:05 mriedem dansmith: i guess the 'accepted' filter is intentional https://review.openstack.org/#/c/244486/
23:38:46 openstackgerrit melanie witt proposed openstack/nova master: Save updated libvirt domain XML after swapping volume https://review.openstack.org/498983
23:43:20 openstackgerrit melanie witt proposed openstack/nova master: Save updated libvirt domain XML after swapping volume https://review.openstack.org/498983
23:47:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Add 'done' to migration_get_in_progress_by_host_and_node filter https://review.openstack.org/511351
23:52:32 openstackgerrit Ed Leafe proposed openstack/nova master: Add Selection objects https://review.openstack.org/499239
23:52:32 openstackgerrit Ed Leafe proposed openstack/nova master: Add alternate hosts https://review.openstack.org/486215
23:52:33 openstackgerrit Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/510159
23:52:33 openstackgerrit Ed Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver https://review.openstack.org/495854
23:52:34 openstackgerrit Ed Leafe proposed openstack/nova master: WIP - Make conductor pass and use host_lists https://review.openstack.org/511358
23:52:34 openstackgerrit Ed Leafe proposed openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357
#openstack-nova - 2017-10-12
01:06:08 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949

Earlier   Later