Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-09-20
15:08:47 dansmith that operation could be across cells, and across a loooot of instances. It could fail in the middle leaving things highly confusing
15:08:47 cdent bauzas: I was checking because as far as I could read the bug report, the “bad” solution was considered good enough
15:08:52 dansmith mriedem: ++
15:09:08 bauzas mriedem: that was my point
15:09:10 mriedem if you want to rename the az, migrate the instances out of it
15:09:23 mriedem but i'm not sure if that's possible?
15:09:31 mriedem unless you use the force flag...
15:09:54 mriedem but if the operator forces a live migration of an instance from az1 to az2, do we even update the instance.availability_zone field?
15:10:03 cdent I think the implementation was following sean’s advice: https://bugs.launchpad.net/nova/+bug/1378904/comments/5
15:10:04 openstack Launchpad bug 1378904 in OpenStack Compute (nova) "renaming availability zone doesn't modify host's availability zone" [Low,In progress] - Assigned to Radoslav Gerganov (rgerganov)
15:10:21 cdent but if there’s a bigger problem than that, would be great to see those problems on the review
15:10:43 bauzas mriedem: I think migrating a VM requires some kind of discussion between the operator and the end user
15:10:52 bauzas not something magical per so
15:10:53 bauzas se
15:11:03 dansmith migrating across AZs without the user's input is dangerous too
15:11:06 mriedem what if the host is going to fail? i guess live migrate to another host in the same az
15:11:12 bauzas and yeah, you can't live migrate from one AZ to the other
15:11:21 bauzas unless you force of course
15:11:22 mriedem bauzas: with the force flag you can :)
15:11:23 mriedem right?
15:11:25 bauzas yup
15:11:34 bauzas but that'd be a terrible experience again
15:11:50 bauzas because I'm not sure we update the AZ record honestly
15:11:57 sdague AZs really exist to bound failure domains so that you can HA across them correctly
15:12:11 sdague moving an instance across AZ boundaries completely ruins that strategy
15:12:24 bauzas so, the point is, if you made a typo, then you're screwed up if some users began to use your cloud, and you have to explain to them that you screwed up
15:12:48 bauzas but you shouldn't magically fix your issue
15:13:01 bauzas tl;dr: assume your mistakes
15:13:12 dansmith sdague: that was my point yeah
15:13:28 cfriesen mriedem: will pass on the suggestion
15:13:56 sdague dansmith: ++
15:13:58 mriedem edleafe: think it's sorted out, but still looks like a latent bug
15:14:09 edleafe mriedem: Agree on the change for num_instances
15:14:24 edleafe I can do that quickly - maybe for backport?
15:14:33 mriedem don't use multi-create, and if you do, don't migrate any of those instances if len(hosts) < len(instances)
15:14:42 mriedem edleafe: i think it would be good to have a functional test for thisfirst
15:15:17 mriedem e.g. 2 computes, 2 instances created in a single boot request to compute 1, disable compute 1 and live migrate the isntances to compute 2
15:15:30 mriedem it should fail on the first live migration attempt since you're trying to move 2 instances and we have 1 host
15:15:39 sdague dansmith: I'm +1 on your block of az renames if there are instances in them
15:15:50 sdague honestly, these things should probably be idempotent like flavors
15:15:51 dansmith sdague: cool
15:16:23 dansmith yeah, you'll still have to count instances across cells in the az in question to know whether or not to block it
15:16:31 dansmith just not letting that happen at all is easier still
15:17:14 mriedem edleafe: it's actually a weird check in the filter scheduler driver, i don't really understand why we compare the number of instances to the number of hosts, surely we can create more than one instance per host
15:17:21 openstackgerrit Merged openstack/nova master: doc: Split flavors docs into admin and user guides https://review.openstack.org/501342
15:18:01 edleafe mriedem: are you refrerring to https://github.com/openstack/nova/blob/8a386b055c82df67092a1abc683e7225ef80671e/nova/scheduler/filter_scheduler.py#L86 ?
15:18:12 openstackgerrit Merged openstack/nova master: doc: Add documentation for emulator_thread_policy https://review.openstack.org/501721
15:18:24 mriedem edleafe: yes
15:18:25 edleafe mriedem: if so, that's the number of *selected* hosts, not the total number of hosts
15:18:28 mriedem cfriesen: edleafe: in fact https://review.openstack.org/#/c/491439/
15:18:36 edleafe IOW, we couldn't find hosts for all the instances
15:18:44 esberglu sdague: Can you restore this for us? https://review.openstack.org/#/c/422512/
15:18:52 esberglu If I were to submit a new patch would it restore the changeset?
15:19:13 sdague esberglu: restored
15:19:18 esberglu sdague: tx
15:19:37 sdague esberglu: the patch owner in gerrit, or a core can do restores
15:19:46 sdague but you have to restore before pushing an updated patch
15:20:02 esberglu sdague: Good to know thanks
15:21:43 mriedem claudiub|2: the master branch change for https://review.openstack.org/#/c/505285/ is merged now
15:24:17 edleafe mriedem: I'm confused. If https://review.openstack.org/#/c/491439/ merged, why is the old code still in master?
15:25:23 tasker having trouble live-migrating my last instance out of compute-1. after a suggestion from melwitt, I looked into the scheduler logs and I see that it scheduling the instance, it states that it's looking at the target host, notes that the target host fails and is removed from contention, but doesn't state why it failed -- even with debug logging on.
15:25:34 tasker any thoughts? or is this infrastructure problems?
15:25:56 tasker it's obvious that nova is doing its job, it's just not being verbose enough.
15:26:42 mriedem edleafe: different place
15:27:23 edleafe mriedem: ah
15:27:24 mriedem tasker: depends on if it's the pre-live migration check on the target host, those logs would either be in nova-conductor or nova-compute for the target host
15:27:33 mriedem edleafe: this is why we probably need a functional test for this scenario
15:31:31 cfriesen mriedem: edleave: did bauzas' patch cause the problem? (because the mailing list thread didn't see the IndexError)
15:32:17 openstackgerrit OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646
15:32:20 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708
15:32:26 mriedem cfriesen: not sure
15:32:42 mriedem it's clear that we need to be consistent in both places though
15:32:50 cfriesen mriedem: for sure
15:38:50 mriedem dansmith: ok that series is all +W
15:48:54 openstackgerrit OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646
15:48:57 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708
15:52:02 openstackgerrit Chris Dent proposed openstack/nova master: WIP: [placement] manage cache headers https://review.openstack.org/495380
15:53:58 cdent Is now a good time to get this merged, since we are at the beginning of the cycle: https://review.openstack.org/#/c/486829/
15:54:11 gibi could one of the cores look at test only patch? https://review.openstack.org/#/c/496202/ only needs a second +2
15:54:17 cdent fix on MiniDNS
15:55:58 openstackgerrit Evgeny Antyshev proposed openstack/nova master: Vzstorage: synchronize volume connect/disconnect https://review.openstack.org/505708
15:57:34 stephenfin gibi: Looking
15:59:10 gibi stephenfin: thanks
16:05:50 openstackgerrit OpenStack Proposal Bot proposed openstack/os-traits master: Updated from global requirements https://review.openstack.org/503646
16:05:54 openstackgerrit OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/502708
16:08:21 mriedem gibi: stephenfin: commented on https://review.openstack.org/#/c/496202/
16:08:29 mriedem that seems to be munging together a few different things
16:08:37 openstackgerrit Stephen Finucane proposed openstack/nova master: Add functional migrate force_complete test https://review.openstack.org/496202
16:09:26 stephenfin mriedem: Aye, but they all seemed valid
16:09:41 stephenfin There could be merit in splitting them out though, so I've rebased to take it out of the gate queue
16:09:56 mriedem if we're going to whitelist legacy notifications because they were missed, the change that adds them in should have a test to make sure they are sent
16:10:16 mriedem which i guess is the referenced patch...but those are linked together in any way
16:10:28 mriedem here is an example https://review.openstack.org/#/c/504978/
16:13:05 dansmith mriedem: right on right on right on
16:13:36 gibi mriedem: OK, let's make a separate bug for the missing force_complete test coverage and move the whitelisting into that
16:15:50 mriedem sounds good to me
16:16:01 mriedem dansmith: you're getting older and the patches are staying the same age?
16:16:02 gibi sorry for the mess
16:16:10 dansmith mriedem: lol
16:19:11 bauzas dansmith: sdague: sorry, was running a meeting, but like I said in my comment in https://review.openstack.org/#/c/419502/2 "Modifying an AZ name should not be possible if instances are still in the AZ"

Earlier   Later