Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-01-10
14:40:45 dansmith since I have to leave in just a few minutes
14:40:59 eglynn ... the pre-upgrade step *of running placement API ...
14:41:00 dansmith but newton/ocata was the optional/required seam,
14:41:22 dansmith and while it could have been messaged better, I think we'll have the same set of surprised people if we make the seam ocata/pike
14:41:40 dansmith doing it in a point release is not reasonable (at all) IMHO, so we just kick this can down the road one more if we do that
14:41:58 edleafe dansmith: +1
14:42:11 dansmith if that's what we have to do, then it's what we have to do, but I think it just delays realizing scheduler gains and I think we'll have the same argument in +6mo
14:42:15 dansmith that's MHO
14:43:16 andreykurilin mriedem: did you resolve an issue with novaclient's gates? :)
14:44:06 cdent presumably one option, not entirely friendly but existing, is we can just put the seam in the upstream where it needs to be and the distros can cope in whatever fashion they want?
14:44:21 diana_clarke andreykurilin: The last comment from Kevin appears to solve the mystery: https://review.openstack.org/#/c/418190/
14:44:25 cdent I state this not to say I'm in favor of it or anything, but just to list it as one of the options.
14:45:39 dansmith cdent: it's a big seam... people will unpatch it if they have to of course
14:45:53 dansmith cdent: the other thing to remember, is that if you haven't setup placement before the upgrade,
14:46:06 eglynn dansmith: IMO the main problem here is that there's an emergent aspect of the cut-over (the requirement to deploy placement and restart computes in *advance* of doing the main upgrade)
14:46:08 dansmith then you really just have a few extra minutes of downtime
14:46:17 andreykurilin diana_clarke: lol. my patch to requirements fixed an issue (I tried to fix another one)
14:46:22 cdent dansmith: Yeah, just trying to enumerate the several facets of reality because without all the facets its hard to see.
14:46:26 dansmith while placement fills and if you just keep your maintenance window open to fill that, then it's not as big of a deployment issue
14:46:46 dansmith eglynn: that's only if you really need to minimize downtime
14:46:48 cdent dansmith: yeah, that's what confuses me, I'm still unsure why this is a big deal, since we're sort of self headling
14:47:17 dansmith eglynn: if you don't, and I assert anyone running, say, tripleo does not obsess over 120 seconds of downtime, then you don't need to pre-deploy it
14:47:29 eglynn the "sort of" in "sort of self healing" makes me very nervous :)
14:47:37 dansmith cdent: yeah
14:47:48 edleafe cdent: any single node is ~1 minute to self-heal; a huge deployment may take much longer
14:47:49 dansmith eglynn: we're exactly as self-healing with placement as we are with resource-tracker
14:47:51 dansmith more so actually
14:47:59 cdent eglynn: heh. I don't like the term self healing in any context, really. It's more: the info is updated every 60s
14:48:04 cdent which results in good info
14:48:10 cdent so it isn't that a failure was healed
14:48:37 andreykurilin mriedem: ok, it looks like I lied (unwittingly) you yesterday and I knew the root of novaclient's failures :))
14:49:10 eglynn dansmith: the 120s of downtime would just be the inability to schedule new instances while placement get warmed up with reported inventories & allocations?
14:49:42 dansmith eglynn: yep, and you could just keep your api down to prevent failed schedules if you want
14:49:43 eglynn (assuming all the compute were SIGHUP'd before the "real" rolling upgrade of computes kicks off)
14:49:45 cdent (when bauzas's pending code is done, we can test this)
14:50:35 openstackgerrit Gábor Antal proposed openstack/nova: Transform instance.volume_attach notification https://review.openstack.org/401992
14:50:48 dansmith eglynn: not sure what you mean about SIGHUP
14:51:44 dansmith you should be able to roll ocata services in any order and have them start reporting whenever the placement service becomes available,
14:51:48 dansmith no SIGHUP required, AFAIK
14:51:49 eglynn dansmith: bauzas reckoned the nova-compute services would have to restarted (even if still possibly based on Newton code) in order to re-trigger their reporting into the placement API
14:52:01 dansmith I think we dropped the try-once thing, but if not we could certainly tweak that
14:52:25 eglynn dansmith: a-ha, k that would simplify
14:52:36 dansmith eglynn: newton ones because they try not to obsess if the catalog was missing the placement endpoint I think
14:52:37 bauzas dansmith: the safe_connect decorator prevents us to retry
14:52:49 dansmith eglynn: but *that* is something we could tweak in a point release for newton,
14:52:56 dansmith and if it's not gone from ocata, it needs to be,
14:53:08 dansmith so when you rolled those computes you'd be good once they're upgraded, regardless of the ordering
14:53:16 bauzas I mean https://github.com/openstack/nova/blob/180e6340a595ec047c59365465f36fed7a669ec3/nova/scheduler/client/report.py#L40-L67
14:53:37 dansmith bauzas: yeah, we have to pull most of that out for ocata anyway
14:53:40 dansmith ocata nodes *should* keep obsessing
14:53:53 bauzas dansmith: but newton computes would still stop reporting ?
14:53:58 bauzas dansmith: I mean
14:54:11 bauzas dansmith: if we say we would run the placement service before upgrading
14:54:19 bauzas and then the computes would self-heal
14:54:34 eglynn dansmith: yeah, if we could get rid of the try-once on a point release for newton, and quantify the warm-up time / downtime to populate placement data, then we could have a path forward here
14:54:35 bauzas dansmith: that wouldn't work for newton computes because of that decorator, right?
14:55:07 openstackgerrit Gábor Antal proposed openstack/nova: Transform instance.volume_detach notification https://review.openstack.org/408676
14:55:08 eglynn bauzas: IIUC dansmith was suggesting that the try-once logic is removed in a point release of newton
14:55:10 bauzas oh, if we remove that in a point release, then there is a path indeed
14:55:14 eglynn yep
14:55:44 bauzas but that would mean you couldn't upgrade from 14.0.0 straight to 15.0.1
14:55:46 bauzas oops
14:55:49 dansmith I really have to go now
14:55:50 bauzas to 15.0.0 rather
14:55:57 cdent thanks dansmith
14:56:06 bauzas you'd have to apply the point release before upgrading to Ocata
14:56:20 bauzas that's a reasonable trade-off but that requires documenting
14:56:31 bauzas dansmith: go off, we can discuss that later
14:57:42 cdent bauzas: is that only need to apply the point release prior to upgrading if you were concerned about limiting any downtime for launching new instances as much as possible? Or always ned to apply the point release?
14:58:40 bauzas cdent: if you wanna do rolling upgrade of your computes, I guess the point release upgrade is a necessary prereq
14:59:54 bauzas cdent: because if you don't apply the point release, then once you upgrade your scheduler, it won't prevent you to pick any newton node
15:00:38 cdent "won't" or "will"?
15:00:49 bauzas oh oops will
15:01:05 cdent yeah, what I'm asking is: for some people, presumably that's not a problem
15:01:16 cdent new instances would only go on upgraded nodes
15:01:27 cdent existing instances would stay wherever they happened to be
15:01:41 bauzas not only new instances
15:01:42 cdent for a small number of seconds zero nodes would available
15:01:56 bauzas if you migrate, then good luck
15:02:18 cdent good luck why?
15:02:44 bauzas the number of seconds where you have zero nodes is the duration you take for upgrading your first compute to ocata
15:32:57 openstack Launchpad bug 1646127 in OpenStack-Gate "gate-grenade-dsvm-neutron-multinode-live-migration-nv failing on stable/newton due to devstack not supporting xenial during Mitaka" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood)
15:32:57 markus_z lyarwood_: That's probably closed, isn't it? https://bugs.launchpad.net/openstack-gate/+bug/1646127
15:33:39 lyarwood_ markus_z: hey, yeah that should be closed now
15:34:29 markus_z Cool. I simply stumbled over it while looking for another thing.
15:51:44 openstackgerrit Davanum Srinivas (dims) proposed openstack/nova: [WIP] Fix failures in gate-nova-tox-db-functional-py35 https://review.openstack.org/418383
15:53:00 openstackgerrit Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying with an empty tests-functional-py3.txt https://review.openstack.org/418453
16:02:57 openstack bug 1649586 in OpenStack Compute (nova) ""HTTP exception thrown: Cannot 'os-migrateLive' instance while it is in task_state migrating" in gate-grenade-dsvm-neutron-multinode-live-migration-nv" [Medium,In progress] https://launchpad.net/bugs/1649586 - Assigned to Pawel Koniszewski (pawel-koniszewski)
16:02:57 markus_z You worked on bug 1649586 and I was wondering what's necessary to fix that. I'm asking because https://review.openstack.org/#/c/347471/ would benefit from a vote by grenade.
16:03:04 markus_z pkoniszewski: ^
16:04:07 markus_z pkoniszewski: I debugged it for some hours but I'm still clueless
16:04:31 pkoniszewski markus_z: one change is merged already https://review.openstack.org/#/c/412922/
16:04:44 pkoniszewski markus_z: but this one is also required https://review.openstack.org/#/c/413595/
16:05:43 markus_z pkoniszewski: I see, that one isn't mentioned in the bug report.
16:05:44 pkoniszewski you can run your change on top of #413595
16:06:30 markus_z pkoniszewski: I'll check the review and ask around in #openstack-qa if we can merge that quickly.
16:06:40 pkoniszewski thanks a lot
16:06:50 markus_z nah, thanks to you, you solved it :)
16:13:10 openstackgerrit Alex Szarka proposed openstack/nova: Transform instance-exists notification https://review.openstack.org/403660
16:14:59 openstackgerrit Alex Szarka proposed openstack/nova: Transform instance-exists notification https://review.openstack.org/403660

Earlier   Later