| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-20 | |||
| 14:40:05 | mdbooth | Because that would be a nova fix, and possibly useful in its own right | |
| 14:40:42 | sdague | mdbooth: no, because that problem is equally theoretically a problem for all the other services as well | |
| 14:40:47 | sdague | except nova-compute | |
| 14:40:47 | dansmith | I definitely don't want separate logs for conductor workers | |
| 14:40:51 | dansmith | because..holy crap | |
| 14:40:54 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646 | |
| 14:40:57 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708 | |
| 14:41:08 | mdbooth | dansmith: Merged logs, ftw! ;) | |
| 14:41:09 | sdague | and it totally would wreck all the log injest systems people have | |
| 14:41:29 | mdbooth | True. | |
| 14:41:34 | mdbooth | oslo.log it is | |
| 14:43:30 | jamespage | mriedem: working a fix now | |
| 14:43:38 | cfriesen | cdent: that does seem to be the most likely suspect. points to a gap in our testing. | |
| 14:45:25 | openstackgerrit | Merged openstack/nova master: Update docs for _destroy_evacuated_instances https://review.openstack.org/500144 | |
| 14:46:00 | mriedem | jamespage: i'm testing sdague's idea too | |
| 14:46:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: check qemu version when calling qemu-img info https://review.openstack.org/505673 | |
| 14:46:34 | openstackgerrit | Elod Illes proposed openstack/nova master: Add instance.interface_attach notification https://review.openstack.org/503089 | |
| 14:46:58 | cdent | cfriesen: gaps in testing is a bit of a trend, but gibi is fixing it ;) | |
| 14:47:51 | openstackgerrit | Chris Dent proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.openstack.org/452006 | |
| 14:47:58 | mriedem | jamespage: testing here https://review.openstack.org/#/c/505674/ | |
| 14:48:01 | sdague | mriedem: you got a devstack patch depends on that? | |
| 14:48:16 | mriedem | yes ^ | |
| 14:48:47 | sdague | ah cool | |
| 14:49:27 | gibi | cdent: do you mean I should add a test case which boots multiple instances with a single boot command then try to migrate them? ;) | |
| 14:49:56 | cdent | I meant you were fixing the trend more generally but if you’re feeling motivated :) | |
| 14:50:24 | dansmith | mriedem: if you're okay with it I'm just going to fast approve all these unregister patches as they're just mechanical search/replace: https://review.openstack.org/#/c/502157/5 | |
| 14:51:24 | openstackgerrit | Merged openstack/nova master: Add @targets_cell for live_migrate_instance method in conductor https://review.openstack.org/503601 | |
| 14:51:56 | mriedem | dansmith: i can go through them quick | |
| 14:52:14 | mriedem | cfriesen: re that ML thread, he's disabling a host and live migrating off the source host - does he mean evacuating off the source host? | |
| 14:53:14 | gibi | cdent: at least I made TODO on my desk about it but I'm not promising anything :) | |
| 14:54:11 | dansmith | mriedem: okay | |
| 14:54:34 | mriedem | dansmith: cdent: question in https://review.openstack.org/#/c/502157/5//COMMIT_MSG but just to make sure i know what we're doing in the series | |
| 14:55:41 | dansmith | mriedem: I'm not sure I understand what you're asking | |
| 14:56:07 | dansmith | oh I see, | |
| 14:56:16 | dansmith | because there weren't any actual object references anywhere | |
| 14:56:21 | dansmith | in that patch specifically | |
| 14:57:10 | cdent | yeah, that’s a pasto | |
| 14:57:15 | mriedem | yeah, ok | |
| 14:57:17 | openstackgerrit | Eric Berglund proposed openstack/nova master: Add PowerVM hypervisor configuration doc https://review.openstack.org/505665 | |
| 14:57:19 | mriedem | alright moving on | |
| 14:58:05 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646 | |
| 14:58:09 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708 | |
| 14:58:51 | mriedem | cfriesen: on that migrate thread, i think in pike ed added the new stuff to pass the number of instances from the request spec (something like that) to the scheduler, so it could be why it's thinking it wants to migrate all 10 at once, from the persisted request spec | |
| 14:58:56 | mriedem | might be good for edleafe to look at that | |
| 14:59:40 | mriedem | https://review.openstack.org/#/c/465171/ | |
| 15:00:47 | mriedem | but this passes just the single instance being live migrated https://review.openstack.org/#/c/465171/11/nova/conductor/tasks/live_migrate.py | |
| 15:05:47 | sahid | melwitt: hum i do not understand your comment | |
| 15:05:49 | mriedem | cfriesen: this looks like a latent issue https://github.com/openstack/nova/blob/8a386b055c82df67092a1abc683e7225ef80671e/nova/scheduler/filter_scheduler.py#L81 | |
| 15:05:55 | sahid | https://review.openstack.org/#/c/334614/ | |
| 15:05:59 | mriedem | we're checking the number of instances from the request spec | |
| 15:06:05 | mriedem | which is persisted when we created the instance with multi-create | |
| 15:06:11 | mriedem | and using that during the live migration of the single instance | |
| 15:06:17 | cdent | bauzas: are you still invested in keeping your -1 on https://review.openstack.org/#/c/419502/ | |
| 15:06:20 | sahid | not sure if you want me to comment an other time or if we can discuss about it | |
| 15:06:36 | mriedem | cfriesen: "num_instances = spec_obj.num_instances" should probably be changed to "num_instances = len(instance_uuids)" now | |
| 15:07:06 | mriedem | cfriesen: i'm not subscribed to the general list but you might ask them to change that line and see if it works | |
| 15:07:32 | bauzas | cdent: I think my concern is still valid | |
| 15:07:42 | bauzas | cdent: renaming an AZ is having an user impact | |
| 15:08:18 | dansmith | bauzas: cdent dear god | |
| 15:08:46 | mriedem | you should just not be able to rename an az while there are instances in that az | |
| 15:08:47 | dansmith | that operation could be across cells, and across a loooot of instances. It could fail in the middle leaving things highly confusing | |
| 15:08:47 | cdent | bauzas: I was checking because as far as I could read the bug report, the “bad” solution was considered good enough | |
| 15:08:52 | dansmith | mriedem: ++ | |
| 15:09:08 | bauzas | mriedem: that was my point | |
| 15:09:10 | mriedem | if you want to rename the az, migrate the instances out of it | |
| 15:09:23 | mriedem | but i'm not sure if that's possible? | |
| 15:09:31 | mriedem | unless you use the force flag... | |
| 15:09:54 | mriedem | but if the operator forces a live migration of an instance from az1 to az2, do we even update the instance.availability_zone field? | |
| 15:10:03 | cdent | I think the implementation was following sean’s advice: https://bugs.launchpad.net/nova/+bug/1378904/comments/5 | |
| 15:10:04 | openstack | Launchpad bug 1378904 in OpenStack Compute (nova) "renaming availability zone doesn't modify host's availability zone" [Low,In progress] - Assigned to Radoslav Gerganov (rgerganov) | |
| 15:10:21 | cdent | but if there’s a bigger problem than that, would be great to see those problems on the review | |
| 15:10:43 | bauzas | mriedem: I think migrating a VM requires some kind of discussion between the operator and the end user | |
| 15:10:52 | bauzas | not something magical per so | |
| 15:10:53 | bauzas | se | |
| 15:11:03 | dansmith | migrating across AZs without the user's input is dangerous too | |
| 15:11:06 | mriedem | what if the host is going to fail? i guess live migrate to another host in the same az | |
| 15:11:12 | bauzas | and yeah, you can't live migrate from one AZ to the other | |
| 15:11:21 | bauzas | unless you force of course | |
| 15:11:22 | mriedem | bauzas: with the force flag you can :) | |
| 15:11:23 | mriedem | right? | |
| 15:11:25 | bauzas | yup | |
| 15:11:34 | bauzas | but that'd be a terrible experience again | |
| 15:11:50 | bauzas | because I'm not sure we update the AZ record honestly | |
| 15:11:57 | sdague | AZs really exist to bound failure domains so that you can HA across them correctly | |
| 15:12:11 | sdague | moving an instance across AZ boundaries completely ruins that strategy | |
| 15:12:24 | bauzas | so, the point is, if you made a typo, then you're screwed up if some users began to use your cloud, and you have to explain to them that you screwed up | |
| 15:12:48 | bauzas | but you shouldn't magically fix your issue | |
| 15:13:01 | bauzas | tl;dr: assume your mistakes | |
| 15:13:12 | dansmith | sdague: that was my point yeah | |
| 15:13:28 | cfriesen | mriedem: will pass on the suggestion | |
| 15:13:56 | sdague | dansmith: ++ | |
| 15:13:58 | mriedem | edleafe: think it's sorted out, but still looks like a latent bug | |
| 15:14:09 | edleafe | mriedem: Agree on the change for num_instances | |
| 15:14:24 | edleafe | I can do that quickly - maybe for backport? | |
| 15:14:33 | mriedem | don't use multi-create, and if you do, don't migrate any of those instances if len(hosts) < len(instances) | |
| 15:14:42 | mriedem | edleafe: i think it would be good to have a functional test for thisfirst | |
| 15:15:17 | mriedem | e.g. 2 computes, 2 instances created in a single boot request to compute 1, disable compute 1 and live migrate the isntances to compute 2 | |
| 15:15:30 | mriedem | it should fail on the first live migration attempt since you're trying to move 2 instances and we have 1 host | |
| 15:15:39 | sdague | dansmith: I'm +1 on your block of az renames if there are instances in them | |
| 15:15:50 | sdague | honestly, these things should probably be idempotent like flavors | |