| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 22:00:08 | mriedem | cfriesen: did you happen to try this out in your live migration recreate with the 16 ports and network-changed event issues? https://review.openstack.org/#/c/465787/ | |
| 22:00:33 | mriedem | before i go too far down that rabbit hole (probably next week at this rate) it'd be good to know if it helps at all | |
| 22:08:00 | cfriesen | mriedem: related to the other issue, it turns out that if you force the "old" nova-compute down, then try to remove the "force-down" it will fail because Service.save() checks the minimum version | |
| 22:08:28 | cfriesen | mriedem: haven't tried that patch yet, but I plan to. | |
| 22:08:29 | mriedem | oh that's a fun chicken and egg | |
| 22:08:45 | cfriesen | isn't it just. :) | |
| 22:14:54 | mriedem | well, seems like a bug to report. we could bypass the service min version check on save() i'd think, but i'm not sure what kind of weird situations that puts us in | |
| 22:15:35 | mriedem | again, like if you've already restarted control services that are caching what they think is the min service version (newton) and all of a sudden you have a running mitaka compute service | |
| 22:15:36 | cfriesen | mriedem: just for fun, the original scenario was that we were trying to take down the compute node to upgrade it, so we marked nova-compute as forced-down, then we tried to down the neutron agent and hit an exception, so we tried to roll back and enable nova-compute and hit exception.ServiceTooOld | |
| 22:16:11 | cfriesen | not sure there's a good solution here. | |
| 22:16:11 | mriedem | cfriesen: why not just disable the service? | |
| 22:16:33 | mriedem | PUT /os-services/disable-log-reason | |
| 22:16:41 | mriedem | that should take that nova-compute out of scheduling | |
| 22:17:36 | cfriesen | mriedem: that's a good point...usually we only force down on a fault. will follow up. | |
| 22:21:18 | mriedem | honestly i'm not sure what the use case is for forced-down which is different from disabling the service | |
| 22:21:49 | mriedem | i think it has to do with external tooling forcing the service to be 'down' rather than waiting for the service group api check to do it | |
| 22:22:01 | mriedem | we could document that better in the api ref | |
| 22:22:21 | mriedem | like, what does doing this actually mean, what impacts / side effects does it have, etc | |
| 22:22:31 | mriedem | cfriesen: ^ probably another bug to open for lack of clear docs on these | |
| 22:22:47 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/mark-host-down.html was the spec | |
| 22:23:34 | mriedem | oh i see, the idea is external service finds out the host is about to die, forces it down and then evacuates the instances off that host | |
| 22:23:41 | mriedem | since evacuate checks this flag | |
| 22:24:11 | mriedem | we should state in the api-ref that forced-down is not something for routine upgrade/maintenance | |
| 22:24:44 | mriedem | as documented here https://docs.openstack.org/ops-guide/ops-maintenance-compute.html#planned-maintenance | |
| 22:30:02 | mriedem | cfriesen: https://bugs.launchpad.net/nova/+bug/1691871 | |
| 22:30:02 | openstack | Launchpad bug 1691871 in OpenStack Compute (nova) "forced-down vs service disable is not documented well in the compute API reference" [Medium,Confirmed] | |
| 22:37:30 | cfriesen | mriedem: it's actually kind of ironic (always feels funny using that word around openstack) since we were one of the groups pushing for the introduction of the "force-down" feature. now looks like someone's misusing it. | |
| 22:39:20 | cfriesen | mriedem: in our environment we have something else that monitors compute node health, so it'll detect the loss of a compute node way earlier than nova will, hence the proper use of force-down | |
| 22:40:49 | cfriesen | agree with your bug though | |
| 22:55:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip ceph in grenade live migration job due to restart failures https://review.openstack.org/466097 | |
| 22:55:30 | mriedem1 | melwitt: dansmith: ^ "fixes" the grenade live migration job | |
| 22:55:43 | mriedem1 | we lose ceph grenade + live migration until the bug is fixed or we get to queens, | |
| 22:55:53 | mriedem1 | but we at least get back testing for mixed-level compute upgrade testing | |
| 22:56:00 | mriedem1 | *live migration testing i mean | |
| 22:56:01 | mriedem1 | on mixed computes | |
| 22:56:12 | mriedem1 | regular multinode grenade doesn't test live migration | |
| 22:56:39 | mriedem1 | ps2 had a clean run, ps3 just removed a debug echo statement | |
| 23:24:01 | melwitt | mriedem: +2 | |
| 23:24:51 | melwitt | I'm interested in fixing the bug but I don't understand your comment about using common functions like in https://github.com/openstack-dev/devstack/commit/c006bbdeb26df2c60f43d222bdf918f9e24d551f | |
| 23:28:12 | mriedem | sdague said there were some common process start/stop/restart functions in devstack and we could overhaul these scripts to use those, | |
| 23:28:20 | mriedem | which would hide the screen vs systemd stuff i think | |
| 23:28:34 | melwitt | ah, okay | |
| 23:28:36 | mriedem | but if those are available, i'm not sure why these live migration scripts weren't using them in the first place | |
| 23:28:47 | mriedem | so i assume there was a good reason those functions from devstack wouldn't work in the first place | |
| 23:28:57 | mriedem | but without tdurakov anymore it's hard to say | |
| 23:29:38 | melwitt | yeah. okay, I shall endeavor to dig into that a bit. because on the surface, I was just thinking to do something dumb like "if ocata, then use old way" | |
| 23:29:51 | melwitt | if it's even possible to check "if ocata" in that script | |
| 23:35:25 | mriedem | the branch is available, the devstack-gate vars are in there | |
| 23:35:43 | mriedem | but yeah i thought similar, basically we'd need to revert a patch that removed the old functions it was using | |
| 23:35:50 | mriedem | and then put in branching logic for doing old vs new things | |
| 23:36:02 | mriedem | did you get the quotas changes done that you were talking about? | |
| 23:36:08 | mriedem | for dict something or other | |
| 23:51:50 | melwitt | mriedem: still working through it now. ran into some snags (of course) on the bottom patch. now it should be mechanical changes going through the stack. so I hope to push it up in an hour or so | |
| 23:52:36 | melwitt | this series has been such a pain | |
| 23:54:44 | mriedem | ok. i decided that between two evils on my todo list, doing forum session recaps was easier for now. :) | |
| 23:54:53 | mriedem | and i can continue to say i'm not reviewing your series becaues i'm waiting for you to update it | |
| 23:55:19 | melwitt | yup, that you can. | |
| #openstack-nova - 2017-05-19 | |||
| 00:15:16 | melwitt | dansmith: is it ever possible to change the return type of a remotable method? like is there a way for new code to know old code is calling so the return type can be formatted to what the old code expects? | |
| 00:15:53 | dansmith | melwitt: nope | |
| 00:16:10 | dansmith | pass a flag or create a new method | |
| 00:16:14 | dansmith | ideally the latter | |
| 00:16:28 | melwitt | okay. so to change all count() to return a dict instead of an int, I have to do something like that | |
| 00:16:44 | dansmith | yeah | |
| 00:17:28 | melwitt | I'm trying to think if it's even possible for old code to call new code count(). it's only called in nova-api | |
| 00:17:42 | melwitt | Quotas.count() | |
| 00:17:44 | dansmith | if you change it you should make a new method | |
| 00:17:48 | dansmith | don't cheat :) | |
| 00:18:01 | melwitt | :( | |
| 00:18:19 | melwitt | the hard part is thinking of a name that isn't dumb | |
| 00:18:39 | dansmith | count_less_sucky_than_the_first_attempt() | |
| 00:18:44 | melwitt | lol, I wish | |
| 00:19:36 | melwitt | maybe count_as_dict since probably peeps will expect an integer | |
| 00:19:56 | dansmith | oh please please call it count_dicts() | |
| 00:21:35 | melwitt | I don't get it | |
| 00:24:17 | cfriesen | I think that's cause you're too mature | |
| 00:24:43 | melwitt | I shall use that excuse :) | |
| 00:26:26 | dansmith | I shall remain immature and unashamed | |
| 00:33:10 | fishbone_ | Hello all, I am having a serious problem. I am performing an upgrade from Newton to Ocata and every time I run the upgrade for nova-compute I receive "ERROR: could not access cell mapping database - has api db been created?" | |
| 00:35:06 | cfriesen | fishbone: have you read the nova upgrade notes in the release notes? | |
| 00:35:50 | fishbone_ | cfriesen: yes I believe so, it only seemed to mention designate changes | |
| 00:37:37 | fishbone_ | I see all of my nova services except for the compute services are up | |
| 00:37:58 | fishbone_ | I have tried creating the cells database manually and also by using the simple_cells command | |
| 00:38:07 | cfriesen | fishbone: I think the section you want begins with "A new nova-manage command has been added which will upgrade a deployment to cells v2..." at https://docs.openstack.org/releasenotes/nova/ocata.html | |
| 00:39:03 | cfriesen | dansmith: is this correct? ^ | |
| 00:39:11 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 00:42:23 | fishbone_ | cfriesen: Thanks for the link, although I tried that method as well. I have my system backed up and was able to retry using that command from scratch. Also the issue only seems to lie with the compute hosts, my controller node can access the api database fine | |
| 00:42:40 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 00:53:38 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 01:20:23 | mriedem | fishbone_: "could not access cell mapping database" has nothing to do with the computes | |
| 01:20:31 | mriedem | you're getting that from running nova-manage db sync right? | |
| 01:20:53 | mriedem | nova.conf on the compute node doesn't need access to the API database (or any database for that matter) | |
| 01:22:00 | fishbone_ | mriedem: I didn't think so either, however; I don't get the error when syncing the database, that seems to work with no errors. I get it when upgrading nova-compute on the compute nodes | |
| 01:22:34 | mriedem | why are you running nova-manage db sync on the compute nodes? | |
| 01:23:33 | mriedem | also fyi https://docs.openstack.org/developer/nova/cells.html#upgrade-minimal | |
| 01:24:02 | mriedem | what you're hitting is this https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L649 | |
| 01:24:11 | mriedem | when it's trying to automatically sync the nova_cell0 database | |
| 01:25:16 | mriedem | anyway, you shouldn't need to sync the db from the compute nodes | |
| 01:28:51 | fishbone_ | mriedem, are you saying the error can be ignored, I don't think I am able to compete the nova-compute installation because of this error | |
| 01:29:16 | fishbone_ | mriedem: Is there a way to suppress that sync process? | |
| 01:29:40 | mriedem | i'm saying you don't need to sync the database from a compute node | |