| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-01-10 | |||
| 14:46:06 | eglynn | dansmith: IMO the main problem here is that there's an emergent aspect of the cut-over (the requirement to deploy placement and restart computes in *advance* of doing the main upgrade) | |
| 14:46:08 | dansmith | then you really just have a few extra minutes of downtime | |
| 14:46:17 | andreykurilin | diana_clarke: lol. my patch to requirements fixed an issue (I tried to fix another one) | |
| 14:46:22 | cdent | dansmith: Yeah, just trying to enumerate the several facets of reality because without all the facets its hard to see. | |
| 14:46:26 | dansmith | while placement fills and if you just keep your maintenance window open to fill that, then it's not as big of a deployment issue | |
| 14:46:46 | dansmith | eglynn: that's only if you really need to minimize downtime | |
| 14:46:48 | cdent | dansmith: yeah, that's what confuses me, I'm still unsure why this is a big deal, since we're sort of self headling | |
| 14:47:17 | dansmith | eglynn: if you don't, and I assert anyone running, say, tripleo does not obsess over 120 seconds of downtime, then you don't need to pre-deploy it | |
| 14:47:29 | eglynn | the "sort of" in "sort of self healing" makes me very nervous :) | |
| 14:47:37 | dansmith | cdent: yeah | |
| 14:47:48 | edleafe | cdent: any single node is ~1 minute to self-heal; a huge deployment may take much longer | |
| 14:47:49 | dansmith | eglynn: we're exactly as self-healing with placement as we are with resource-tracker | |
| 14:47:51 | dansmith | more so actually | |
| 14:47:59 | cdent | eglynn: heh. I don't like the term self healing in any context, really. It's more: the info is updated every 60s | |
| 14:48:04 | cdent | which results in good info | |
| 14:48:10 | cdent | so it isn't that a failure was healed | |
| 14:48:37 | andreykurilin | mriedem: ok, it looks like I lied (unwittingly) you yesterday and I knew the root of novaclient's failures :)) | |
| 14:49:10 | eglynn | dansmith: the 120s of downtime would just be the inability to schedule new instances while placement get warmed up with reported inventories & allocations? | |
| 14:49:42 | dansmith | eglynn: yep, and you could just keep your api down to prevent failed schedules if you want | |
| 14:49:43 | eglynn | (assuming all the compute were SIGHUP'd before the "real" rolling upgrade of computes kicks off) | |
| 14:49:45 | cdent | (when bauzas's pending code is done, we can test this) | |
| 14:50:35 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.volume_attach notification https://review.openstack.org/401992 | |
| 14:50:48 | dansmith | eglynn: not sure what you mean about SIGHUP | |
| 14:51:44 | dansmith | you should be able to roll ocata services in any order and have them start reporting whenever the placement service becomes available, | |
| 14:51:48 | dansmith | no SIGHUP required, AFAIK | |
| 14:51:49 | eglynn | dansmith: bauzas reckoned the nova-compute services would have to restarted (even if still possibly based on Newton code) in order to re-trigger their reporting into the placement API | |
| 14:52:01 | dansmith | I think we dropped the try-once thing, but if not we could certainly tweak that | |
| 14:52:25 | eglynn | dansmith: a-ha, k that would simplify | |
| 14:52:36 | dansmith | eglynn: newton ones because they try not to obsess if the catalog was missing the placement endpoint I think | |
| 14:52:37 | bauzas | dansmith: the safe_connect decorator prevents us to retry | |
| 14:52:49 | dansmith | eglynn: but *that* is something we could tweak in a point release for newton, | |
| 14:52:56 | dansmith | and if it's not gone from ocata, it needs to be, | |
| 14:53:08 | dansmith | so when you rolled those computes you'd be good once they're upgraded, regardless of the ordering | |
| 14:53:16 | bauzas | I mean https://github.com/openstack/nova/blob/180e6340a595ec047c59365465f36fed7a669ec3/nova/scheduler/client/report.py#L40-L67 | |
| 14:53:37 | dansmith | bauzas: yeah, we have to pull most of that out for ocata anyway | |
| 14:53:40 | dansmith | ocata nodes *should* keep obsessing | |
| 14:53:53 | bauzas | dansmith: but newton computes would still stop reporting ? | |
| 14:53:58 | bauzas | dansmith: I mean | |
| 14:54:11 | bauzas | dansmith: if we say we would run the placement service before upgrading | |
| 14:54:19 | bauzas | and then the computes would self-heal | |
| 14:54:34 | eglynn | dansmith: yeah, if we could get rid of the try-once on a point release for newton, and quantify the warm-up time / downtime to populate placement data, then we could have a path forward here | |
| 14:54:35 | bauzas | dansmith: that wouldn't work for newton computes because of that decorator, right? | |
| 14:55:07 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.volume_detach notification https://review.openstack.org/408676 | |
| 14:55:08 | eglynn | bauzas: IIUC dansmith was suggesting that the try-once logic is removed in a point release of newton | |
| 14:55:10 | bauzas | oh, if we remove that in a point release, then there is a path indeed | |
| 14:55:14 | eglynn | yep | |
| 14:55:44 | bauzas | but that would mean you couldn't upgrade from 14.0.0 straight to 15.0.1 | |
| 14:55:46 | bauzas | oops | |
| 14:55:49 | dansmith | I really have to go now | |
| 14:55:50 | bauzas | to 15.0.0 rather | |
| 14:55:57 | cdent | thanks dansmith | |
| 14:56:06 | bauzas | you'd have to apply the point release before upgrading to Ocata | |
| 14:56:20 | bauzas | that's a reasonable trade-off but that requires documenting | |
| 14:56:31 | bauzas | dansmith: go off, we can discuss that later | |
| 14:57:42 | cdent | bauzas: is that only need to apply the point release prior to upgrading if you were concerned about limiting any downtime for launching new instances as much as possible? Or always ned to apply the point release? | |
| 14:58:40 | bauzas | cdent: if you wanna do rolling upgrade of your computes, I guess the point release upgrade is a necessary prereq | |
| 14:59:54 | bauzas | cdent: because if you don't apply the point release, then once you upgrade your scheduler, it won't prevent you to pick any newton node | |
| 15:00:38 | cdent | "won't" or "will"? | |
| 15:00:49 | bauzas | oh oops will | |
| 15:01:05 | cdent | yeah, what I'm asking is: for some people, presumably that's not a problem | |
| 15:01:16 | cdent | new instances would only go on upgraded nodes | |
| 15:01:27 | cdent | existing instances would stay wherever they happened to be | |
| 15:01:41 | bauzas | not only new instances | |
| 15:01:42 | cdent | for a small number of seconds zero nodes would available | |
| 15:01:56 | bauzas | if you migrate, then good luck | |
| 15:02:18 | cdent | good luck why? | |
| 15:02:44 | bauzas | the number of seconds where you have zero nodes is the duration you take for upgrading your first compute to ocata | |
| 15:32:57 | openstack | Launchpad bug 1646127 in OpenStack-Gate "gate-grenade-dsvm-neutron-multinode-live-migration-nv failing on stable/newton due to devstack not supporting xenial during Mitaka" [Undecided,In progress] - Assigned to Lee Yarwood (lyarwood) | |
| 15:32:57 | markus_z | lyarwood_: That's probably closed, isn't it? https://bugs.launchpad.net/openstack-gate/+bug/1646127 | |
| 15:33:39 | lyarwood_ | markus_z: hey, yeah that should be closed now | |
| 15:34:29 | markus_z | Cool. I simply stumbled over it while looking for another thing. | |
| 15:51:44 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] Fix failures in gate-nova-tox-db-functional-py35 https://review.openstack.org/418383 | |
| 15:53:00 | openstackgerrit | Davanum Srinivas (dims) proposed openstack/nova: [WIP] ignore me - trying with an empty tests-functional-py3.txt https://review.openstack.org/418453 | |
| 16:02:57 | openstack | bug 1649586 in OpenStack Compute (nova) ""HTTP exception thrown: Cannot 'os-migrateLive' instance while it is in task_state migrating" in gate-grenade-dsvm-neutron-multinode-live-migration-nv" [Medium,In progress] https://launchpad.net/bugs/1649586 - Assigned to Pawel Koniszewski (pawel-koniszewski) | |
| 16:02:57 | markus_z | You worked on bug 1649586 and I was wondering what's necessary to fix that. I'm asking because https://review.openstack.org/#/c/347471/ would benefit from a vote by grenade. | |
| 16:03:04 | markus_z | pkoniszewski: ^ | |
| 16:04:07 | markus_z | pkoniszewski: I debugged it for some hours but I'm still clueless | |
| 16:04:31 | pkoniszewski | markus_z: one change is merged already https://review.openstack.org/#/c/412922/ | |
| 16:04:44 | pkoniszewski | markus_z: but this one is also required https://review.openstack.org/#/c/413595/ | |
| 16:05:43 | markus_z | pkoniszewski: I see, that one isn't mentioned in the bug report. | |
| 16:05:44 | pkoniszewski | you can run your change on top of #413595 | |
| 16:06:30 | markus_z | pkoniszewski: I'll check the review and ask around in #openstack-qa if we can merge that quickly. | |
| 16:06:40 | pkoniszewski | thanks a lot | |
| 16:06:50 | markus_z | nah, thanks to you, you solved it :) | |
| 16:13:10 | openstackgerrit | Alex Szarka proposed openstack/nova: Transform instance-exists notification https://review.openstack.org/403660 | |
| 16:14:59 | openstackgerrit | Alex Szarka proposed openstack/nova: Transform instance-exists notification https://review.openstack.org/403660 | |
| 16:36:41 | openstackgerrit | Julian proposed openstack/nova: Removed duplicated _UNUSED enum in InstancePowerState Validation of the JSON schema fails when the enum has a duplicate https://review.openstack.org/418479 | |
| 16:57:49 | openstackgerrit | Christopher Martin proposed openstack/nova: Stop using mox in test_compute_cells.py https://review.openstack.org/389750 | |
| 16:59:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova: Remove **kwargs passing in payload __init__ https://review.openstack.org/418489 | |
| 16:59:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova: Refactor InstancePayload creation https://review.openstack.org/418488 | |
| 17:31:36 | openstackgerrit | Rafael Folco proposed openstack/nova: DELETE all inventory for a resource provider https://review.openstack.org/416669 | |
| 17:39:16 | openstackgerrit | Chris Dent proposed openstack/nova: [WIP] Structure for simply managing placement-api-ref https://review.openstack.org/409340 | |
| 17:39:17 | openstackgerrit | Chris Dent proposed openstack/nova: [placement] add api-ref for GET /resource_providers https://review.openstack.org/418503 | |
| 18:24:57 | openstackgerrit | Ian Wells proposed openstack/nova-specs: Add the information Nova requires for a port to the port itself https://review.openstack.org/390513 | |
| 18:26:22 | openstackgerrit | Ian Wells proposed openstack/nova-specs: VIF plugging negotiation between Nova and Neutron https://review.openstack.org/390512 | |
| 18:29:37 | openstackgerrit | Gábor Antal proposed openstack/nova: Transform instance.volume_attach notification https://review.openstack.org/401992 | |
| 18:30:58 | openstackgerrit | John Garbutt proposed openstack/nova: Populate network data in _validate_requested_network_ids https://review.openstack.org/418511 | |
| 18:30:58 | openstackgerrit | John Garbutt proposed openstack/nova: Pass objects into _validate_requested_network_ids https://review.openstack.org/418027 | |
| 18:30:59 | openstackgerrit | John Garbutt proposed openstack/nova: Pass objects into _update_ports_for_instance https://review.openstack.org/418513 | |
| 18:30:59 | openstackgerrit | John Garbutt proposed openstack/nova: Pass objects into _create_ports_for_instance https://review.openstack.org/418512 | |