Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-20
14:54:34 mriedem dansmith: cdent: question in https://review.openstack.org/#/c/502157/5//COMMIT_MSG but just to make sure i know what we're doing in the series
14:55:41 dansmith mriedem: I'm not sure I understand what you're asking
14:56:07 dansmith oh I see,
14:56:16 dansmith because there weren't any actual object references anywhere
14:56:21 dansmith in that patch specifically
14:57:10 cdent yeah, that’s a pasto
14:57:15 mriedem yeah, ok
14:57:17 openstackgerrit Eric Berglund proposed openstack/nova master: Add PowerVM hypervisor configuration doc https://review.openstack.org/505665
14:57:19 mriedem alright moving on
14:58:05 openstackgerrit OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646
14:58:09 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708
14:58:51 mriedem cfriesen: on that migrate thread, i think in pike ed added the new stuff to pass the number of instances from the request spec (something like that) to the scheduler, so it could be why it's thinking it wants to migrate all 10 at once, from the persisted request spec
14:58:56 mriedem might be good for edleafe to look at that
14:59:40 mriedem https://review.openstack.org/#/c/465171/
15:00:47 mriedem but this passes just the single instance being live migrated https://review.openstack.org/#/c/465171/11/nova/conductor/tasks/live_migrate.py
15:05:47 sahid melwitt: hum i do not understand your comment
15:05:49 mriedem cfriesen: this looks like a latent issue https://github.com/openstack/nova/blob/8a386b055c82df67092a1abc683e7225ef80671e/nova/scheduler/filter_scheduler.py#L81
15:05:55 sahid https://review.openstack.org/#/c/334614/
15:05:59 mriedem we're checking the number of instances from the request spec
15:06:05 mriedem which is persisted when we created the instance with multi-create
15:06:11 mriedem and using that during the live migration of the single instance
15:06:17 cdent bauzas: are you still invested in keeping your -1 on https://review.openstack.org/#/c/419502/
15:06:20 sahid not sure if you want me to comment an other time or if we can discuss about it
15:06:36 mriedem cfriesen: "num_instances = spec_obj.num_instances" should probably be changed to "num_instances = len(instance_uuids)" now
15:07:06 mriedem cfriesen: i'm not subscribed to the general list but you might ask them to change that line and see if it works
15:07:32 bauzas cdent: I think my concern is still valid
15:07:42 bauzas cdent: renaming an AZ is having an user impact
15:08:18 dansmith bauzas: cdent dear god
15:08:46 mriedem you should just not be able to rename an az while there are instances in that az
15:08:47 dansmith that operation could be across cells, and across a loooot of instances. It could fail in the middle leaving things highly confusing
15:08:47 cdent bauzas: I was checking because as far as I could read the bug report, the “bad” solution was considered good enough
15:08:52 dansmith mriedem: ++
15:09:08 bauzas mriedem: that was my point
15:09:10 mriedem if you want to rename the az, migrate the instances out of it
15:09:23 mriedem but i'm not sure if that's possible?
15:09:31 mriedem unless you use the force flag...
15:09:54 mriedem but if the operator forces a live migration of an instance from az1 to az2, do we even update the instance.availability_zone field?
15:10:03 cdent I think the implementation was following sean’s advice: https://bugs.launchpad.net/nova/+bug/1378904/comments/5
15:10:04 openstack Launchpad bug 1378904 in OpenStack Compute (nova) "renaming availability zone doesn't modify host's availability zone" [Low,In progress] - Assigned to Radoslav Gerganov (rgerganov)
15:10:21 cdent but if there’s a bigger problem than that, would be great to see those problems on the review
15:10:43 bauzas mriedem: I think migrating a VM requires some kind of discussion between the operator and the end user
15:10:52 bauzas not something magical per so
15:10:53 bauzas se
15:11:03 dansmith migrating across AZs without the user's input is dangerous too
15:11:06 mriedem what if the host is going to fail? i guess live migrate to another host in the same az
15:11:12 bauzas and yeah, you can't live migrate from one AZ to the other
15:11:21 bauzas unless you force of course
15:11:22 mriedem bauzas: with the force flag you can :)
15:11:23 mriedem right?
15:11:25 bauzas yup
15:11:34 bauzas but that'd be a terrible experience again
15:11:50 bauzas because I'm not sure we update the AZ record honestly
15:11:57 sdague AZs really exist to bound failure domains so that you can HA across them correctly
15:12:11 sdague moving an instance across AZ boundaries completely ruins that strategy
15:12:24 bauzas so, the point is, if you made a typo, then you're screwed up if some users began to use your cloud, and you have to explain to them that you screwed up
15:12:48 bauzas but you shouldn't magically fix your issue
15:13:01 bauzas tl;dr: assume your mistakes
15:13:12 dansmith sdague: that was my point yeah
15:13:28 cfriesen mriedem: will pass on the suggestion
15:13:56 sdague dansmith: ++
15:13:58 mriedem edleafe: think it's sorted out, but still looks like a latent bug
15:14:09 edleafe mriedem: Agree on the change for num_instances
15:14:24 edleafe I can do that quickly - maybe for backport?
15:14:33 mriedem don't use multi-create, and if you do, don't migrate any of those instances if len(hosts) < len(instances)
15:14:42 mriedem edleafe: i think it would be good to have a functional test for thisfirst
15:15:17 mriedem e.g. 2 computes, 2 instances created in a single boot request to compute 1, disable compute 1 and live migrate the isntances to compute 2
15:15:30 mriedem it should fail on the first live migration attempt since you're trying to move 2 instances and we have 1 host
15:15:39 sdague dansmith: I'm +1 on your block of az renames if there are instances in them
15:15:50 sdague honestly, these things should probably be idempotent like flavors
15:15:51 dansmith sdague: cool
15:16:23 dansmith yeah, you'll still have to count instances across cells in the az in question to know whether or not to block it
15:16:31 dansmith just not letting that happen at all is easier still
15:17:14 mriedem edleafe: it's actually a weird check in the filter scheduler driver, i don't really understand why we compare the number of instances to the number of hosts, surely we can create more than one instance per host
15:17:21 openstackgerrit Merged openstack/nova master: doc: Split flavors docs into admin and user guides https://review.openstack.org/501342
15:18:01 edleafe mriedem: are you refrerring to https://github.com/openstack/nova/blob/8a386b055c82df67092a1abc683e7225ef80671e/nova/scheduler/filter_scheduler.py#L86 ?
15:18:12 openstackgerrit Merged openstack/nova master: doc: Add documentation for emulator_thread_policy https://review.openstack.org/501721
15:18:24 mriedem edleafe: yes
15:18:25 edleafe mriedem: if so, that's the number of *selected* hosts, not the total number of hosts
15:18:28 mriedem cfriesen: edleafe: in fact https://review.openstack.org/#/c/491439/
15:18:36 edleafe IOW, we couldn't find hosts for all the instances
15:18:44 esberglu sdague: Can you restore this for us? https://review.openstack.org/#/c/422512/
15:18:52 esberglu If I were to submit a new patch would it restore the changeset?
15:19:13 sdague esberglu: restored
15:19:18 esberglu sdague: tx
15:19:37 sdague esberglu: the patch owner in gerrit, or a core can do restores
15:19:46 sdague but you have to restore before pushing an updated patch
15:20:02 esberglu sdague: Good to know thanks
15:21:43 mriedem claudiub|2: the master branch change for https://review.openstack.org/#/c/505285/ is merged now
15:24:17 edleafe mriedem: I'm confused. If https://review.openstack.org/#/c/491439/ merged, why is the old code still in master?
15:25:23 tasker having trouble live-migrating my last instance out of compute-1. after a suggestion from melwitt, I looked into the scheduler logs and I see that it scheduling the instance, it states that it's looking at the target host, notes that the target host fails and is removed from contention, but doesn't state why it failed -- even with debug logging on.
15:25:34 tasker any thoughts? or is this infrastructure problems?
15:25:56 tasker it's obvious that nova is doing its job, it's just not being verbose enough.
15:26:42 mriedem edleafe: different place
15:27:23 edleafe mriedem: ah
15:27:24 mriedem tasker: depends on if it's the pre-live migration check on the target host, those logs would either be in nova-conductor or nova-compute for the target host
15:27:33 mriedem edleafe: this is why we probably need a functional test for this scenario
15:31:31 cfriesen mriedem: edleave: did bauzas' patch cause the problem? (because the mailing list thread didn't see the IndexError)
15:32:17 openstackgerrit OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646
15:32:20 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708
15:32:26 mriedem cfriesen: not sure

Earlier   Later