| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 19:15:18 | openstackgerrit | Sean Dague proposed openstack/nova master: see if we can use pg wheels with no bindep https://review.openstack.org/466081 | |
| 19:26:09 | mriedem | dansmith: did you figure out http://logs.openstack.org/37/458537/20/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/ab364da/logs/testr_results.html.gz ? | |
| 19:47:47 | openstackgerrit | Cedric Brandily proposed openstack/nova master: Correct _ensure_console_log_for_instance implementation https://review.openstack.org/466088 | |
| 19:48:56 | mriedem | this seems wrong http://logs.openstack.org/42/465042/6/check/gate-nova-tox-functional-py35-ubuntu-xenial/9829a45/console.html#_2017-05-18_18_43_03_020297 | |
| 19:49:00 | mriedem | b"KeyError: 'allocations'" | |
| 19:49:56 | dansmith | mriedem: I found the hook yeah, still working on a fix | |
| 19:50:07 | mriedem | ok, | |
| 19:50:21 | mriedem | i'm working on disabling the grenade live migration job from running on pike until we fix it, or queens | |
| 19:50:28 | mriedem | since right now it's just burning nodes | |
| 19:51:53 | mriedem | this is very weird - http://logs.openstack.org/42/465042/6/check/gate-nova-tox-functional-py35-ubuntu-xenial/9829a45/console.html#_2017-05-18_18_43_03_020297 it's like in the middle of a functional test, we got a different response from placement | |
| 19:52:07 | mriedem | like it's a GET / response | |
| 19:52:21 | mriedem | but was expecting GET /resource_providers/d41180aa-047e-4bc3-b0f3-11532683400d/allocations | |
| 19:54:44 | openstackgerrit | Cedric Brandily proposed openstack/nova master: Correct _ensure_console_log_for_instance implementation https://review.openstack.org/466088 | |
| 19:58:31 | melwitt | mriedem: yeah, I didn't know what to make of that | |
| 20:03:21 | mriedem | could have something to do with running those services under eventlet in our functional tests? | |
| 20:20:29 | efried | Where does the hashy thing at the end of a release note .yaml's basename come from? | |
| 20:21:36 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Canary test against multi-cell devstack https://review.openstack.org/458537 | |
| 20:21:54 | artom | efried, the 'reno' command does it | |
| 20:23:04 | efried | artom Is there a doc somewhere that describes the release note process? | |
| 20:23:12 | efried | (Cause your answer just raised bigger questions ;-) | |
| 20:23:35 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip ceph in grenade live migration job due to restart failures https://review.openstack.org/466097 | |
| 20:23:35 | artom | efried, not that I know of, but if you have a devstack env somewhere reno should be in there | |
| 20:23:36 | mriedem | sdague: i think i've found a poor man's compromise ^ | |
| 20:23:44 | artom | efried, in which case reno --help will help :) | |
| 20:24:01 | mriedem | efried: artom: https://docs.openstack.org/developer/nova/releasenotes.html | |
| 20:24:06 | efried | artom Yeah, found reno --help; it kinda assumes you know what youre doing already. | |
| 20:24:10 | efried | Okay, thank you artom | |
| 20:24:16 | mriedem | the hash is auto-generated | |
| 20:24:26 | mriedem | tox -e venv -- reno new <your reno name> | |
| 20:25:01 | mriedem | dhellmann said he's found that sometimes people copy renos and re-use the same hash which messes things up | |
| 20:25:26 | efried | I guessed that would be a no-no | |
| 20:25:33 | efried | Thanks mriedem & artom | |
| 20:25:42 | mriedem | yeah, plus reviewers wouldn't be able to eyeball that | |
| 20:25:58 | mriedem | seems the reno docs build itself should fail if that happens | |
| 21:25:13 | openstackgerrit | Alex Szarka proposed openstack/nova master: Add method for verify multiple notifications https://review.openstack.org/465526 | |
| 21:25:33 | openstackgerrit | Alex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification https://review.openstack.org/403660 | |
| 21:33:10 | openstackgerrit | Monty Taylor proposed openstack/nova master: WIP Add description of server addresses objects https://review.openstack.org/464331 | |
| 21:33:10 | openstackgerrit | Monty Taylor proposed openstack/nova master: WIP Show results of nested parameters in api-ref https://review.openstack.org/464277 | |
| 21:39:32 | cfriesen | mriedem: you around? | |
| 21:40:03 | mriedem | sort of | |
| 21:40:50 | cfriesen | mriedem: got a weird scenario...upgrades are in progress, one compute node hasn't been upgraded. It gets rebooted without being upgraded and hits exception.ServiceTooOld | |
| 21:41:16 | cfriesen | mriedem: are we just expecting people to upgrade in that case? | |
| 21:41:35 | openstackgerrit | Alex Szarka proposed openstack/nova master: Add method for verify multiple notifications https://review.openstack.org/465526 | |
| 21:42:34 | cfriesen | mriedem: actually, thinking about it a bit I'm not clear why the version of that compute node isn't being seen in the DB | |
| 21:42:44 | openstackgerrit | Alex Szarka proposed openstack/nova master: [WIP] Transform instance.exists notification https://review.openstack.org/403660 | |
| 21:44:01 | cfriesen | mriedem: maybe the service is still being forced down by our management software, causing its version to be ignored by service_get_minimum_version() | |
| 21:45:53 | mriedem | that's a dansmith question | |
| 21:46:50 | mriedem | cfriesen: it's blowing up here right? https://github.com/openstack/nova/blob/master/nova/objects/service.py#L307 | |
| 21:47:14 | cfriesen | mriedem: yep. | |
| 21:47:39 | cfriesen | down in the sql code we ignore the version of "forced-down" services, so maybe that's it | |
| 21:49:12 | mriedem | right this query https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L490 | |
| 21:50:46 | mriedem | so all of your other computes are newton, and you've got a mitaka forced-down compute, and you're restarting it and it's blowing up because that mitaka compute is older than the minimum for all of the other computes, which is newton | |
| 21:51:38 | mriedem | so upgrade that compute to newton | |
| 21:51:57 | cfriesen | mriedem: yeah, I'm not certain it's forced down but it's the only thing that makes sense | |
| 21:52:06 | mriedem | do you have the service record from the db? | |
| 21:53:06 | cfriesen | mriedem: could probably track it down. would indeed be able to tell from that. I'm just reviewing a proposed change, wasn't actually my issue. one of the guys wants to bypass the check and I'm not a big fan. | |
| 21:53:24 | mriedem | i mean you could PUT /os-services/force-down and mark that service as forced_down=False, which would change the min service version check to say mitaka is the oldest, but that might give you other issues if other things are running thinking that newton is the minimum | |
| 21:54:34 | cfriesen | we're not using auto-rpc versioning (yet, at least) so during an uprade I think it'd be okay as long as we haven't set the rpc version to newton yet. will have to go get more info I think. | |
| 21:56:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip ceph in grenade live migration job due to restart failures https://review.openstack.org/466097 | |
| 22:00:08 | mriedem | cfriesen: did you happen to try this out in your live migration recreate with the 16 ports and network-changed event issues? https://review.openstack.org/#/c/465787/ | |
| 22:00:33 | mriedem | before i go too far down that rabbit hole (probably next week at this rate) it'd be good to know if it helps at all | |
| 22:08:00 | cfriesen | mriedem: related to the other issue, it turns out that if you force the "old" nova-compute down, then try to remove the "force-down" it will fail because Service.save() checks the minimum version | |
| 22:08:28 | cfriesen | mriedem: haven't tried that patch yet, but I plan to. | |
| 22:08:29 | mriedem | oh that's a fun chicken and egg | |
| 22:08:45 | cfriesen | isn't it just. :) | |
| 22:14:54 | mriedem | well, seems like a bug to report. we could bypass the service min version check on save() i'd think, but i'm not sure what kind of weird situations that puts us in | |
| 22:15:35 | mriedem | again, like if you've already restarted control services that are caching what they think is the min service version (newton) and all of a sudden you have a running mitaka compute service | |
| 22:15:36 | cfriesen | mriedem: just for fun, the original scenario was that we were trying to take down the compute node to upgrade it, so we marked nova-compute as forced-down, then we tried to down the neutron agent and hit an exception, so we tried to roll back and enable nova-compute and hit exception.ServiceTooOld | |
| 22:16:11 | cfriesen | not sure there's a good solution here. | |
| 22:16:11 | mriedem | cfriesen: why not just disable the service? | |
| 22:16:33 | mriedem | PUT /os-services/disable-log-reason | |
| 22:16:41 | mriedem | that should take that nova-compute out of scheduling | |
| 22:17:36 | cfriesen | mriedem: that's a good point...usually we only force down on a fault. will follow up. | |
| 22:21:18 | mriedem | honestly i'm not sure what the use case is for forced-down which is different from disabling the service | |
| 22:21:49 | mriedem | i think it has to do with external tooling forcing the service to be 'down' rather than waiting for the service group api check to do it | |
| 22:22:01 | mriedem | we could document that better in the api ref | |
| 22:22:21 | mriedem | like, what does doing this actually mean, what impacts / side effects does it have, etc | |
| 22:22:31 | mriedem | cfriesen: ^ probably another bug to open for lack of clear docs on these | |
| 22:22:47 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/mark-host-down.html was the spec | |
| 22:23:34 | mriedem | oh i see, the idea is external service finds out the host is about to die, forces it down and then evacuates the instances off that host | |
| 22:23:41 | mriedem | since evacuate checks this flag | |
| 22:24:11 | mriedem | we should state in the api-ref that forced-down is not something for routine upgrade/maintenance | |
| 22:24:44 | mriedem | as documented here https://docs.openstack.org/ops-guide/ops-maintenance-compute.html#planned-maintenance | |
| 22:30:02 | mriedem | cfriesen: https://bugs.launchpad.net/nova/+bug/1691871 | |
| 22:30:02 | openstack | Launchpad bug 1691871 in OpenStack Compute (nova) "forced-down vs service disable is not documented well in the compute API reference" [Medium,Confirmed] | |
| 22:37:30 | cfriesen | mriedem: it's actually kind of ironic (always feels funny using that word around openstack) since we were one of the groups pushing for the introduction of the "force-down" feature. now looks like someone's misusing it. | |
| 22:39:20 | cfriesen | mriedem: in our environment we have something else that monitors compute node health, so it'll detect the loss of a compute node way earlier than nova will, hence the proper use of force-down | |
| 22:40:49 | cfriesen | agree with your bug though | |
| 22:55:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip ceph in grenade live migration job due to restart failures https://review.openstack.org/466097 | |
| 22:55:30 | mriedem1 | melwitt: dansmith: ^ "fixes" the grenade live migration job | |
| 22:55:43 | mriedem1 | we lose ceph grenade + live migration until the bug is fixed or we get to queens, | |
| 22:55:53 | mriedem1 | but we at least get back testing for mixed-level compute upgrade testing | |
| 22:56:00 | mriedem1 | *live migration testing i mean | |
| 22:56:01 | mriedem1 | on mixed computes | |
| 22:56:12 | mriedem1 | regular multinode grenade doesn't test live migration | |
| 22:56:39 | mriedem1 | ps2 had a clean run, ps3 just removed a debug echo statement | |
| 23:24:01 | melwitt | mriedem: +2 | |
| 23:24:51 | melwitt | I'm interested in fixing the bug but I don't understand your comment about using common functions like in https://github.com/openstack-dev/devstack/commit/c006bbdeb26df2c60f43d222bdf918f9e24d551f | |
| 23:28:12 | mriedem | sdague said there were some common process start/stop/restart functions in devstack and we could overhaul these scripts to use those, | |
| 23:28:20 | mriedem | which would hide the screen vs systemd stuff i think | |