| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-05-18 | |||
| 22:22:21 | mriedem | like, what does doing this actually mean, what impacts / side effects does it have, etc | |
| 22:22:31 | mriedem | cfriesen: ^ probably another bug to open for lack of clear docs on these | |
| 22:22:47 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/mark-host-down.html was the spec | |
| 22:23:34 | mriedem | oh i see, the idea is external service finds out the host is about to die, forces it down and then evacuates the instances off that host | |
| 22:23:41 | mriedem | since evacuate checks this flag | |
| 22:24:11 | mriedem | we should state in the api-ref that forced-down is not something for routine upgrade/maintenance | |
| 22:24:44 | mriedem | as documented here https://docs.openstack.org/ops-guide/ops-maintenance-compute.html#planned-maintenance | |
| 22:30:02 | mriedem | cfriesen: https://bugs.launchpad.net/nova/+bug/1691871 | |
| 22:30:02 | openstack | Launchpad bug 1691871 in OpenStack Compute (nova) "forced-down vs service disable is not documented well in the compute API reference" [Medium,Confirmed] | |
| 22:37:30 | cfriesen | mriedem: it's actually kind of ironic (always feels funny using that word around openstack) since we were one of the groups pushing for the introduction of the "force-down" feature. now looks like someone's misusing it. | |
| 22:39:20 | cfriesen | mriedem: in our environment we have something else that monitors compute node health, so it'll detect the loss of a compute node way earlier than nova will, hence the proper use of force-down | |
| 22:40:49 | cfriesen | agree with your bug though | |
| 22:55:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip ceph in grenade live migration job due to restart failures https://review.openstack.org/466097 | |
| 22:55:30 | mriedem1 | melwitt: dansmith: ^ "fixes" the grenade live migration job | |
| 22:55:43 | mriedem1 | we lose ceph grenade + live migration until the bug is fixed or we get to queens, | |
| 22:55:53 | mriedem1 | but we at least get back testing for mixed-level compute upgrade testing | |
| 22:56:00 | mriedem1 | *live migration testing i mean | |
| 22:56:01 | mriedem1 | on mixed computes | |
| 22:56:12 | mriedem1 | regular multinode grenade doesn't test live migration | |
| 22:56:39 | mriedem1 | ps2 had a clean run, ps3 just removed a debug echo statement | |
| 23:24:01 | melwitt | mriedem: +2 | |
| 23:24:51 | melwitt | I'm interested in fixing the bug but I don't understand your comment about using common functions like in https://github.com/openstack-dev/devstack/commit/c006bbdeb26df2c60f43d222bdf918f9e24d551f | |
| 23:28:12 | mriedem | sdague said there were some common process start/stop/restart functions in devstack and we could overhaul these scripts to use those, | |
| 23:28:20 | mriedem | which would hide the screen vs systemd stuff i think | |
| 23:28:34 | melwitt | ah, okay | |
| 23:28:36 | mriedem | but if those are available, i'm not sure why these live migration scripts weren't using them in the first place | |
| 23:28:47 | mriedem | so i assume there was a good reason those functions from devstack wouldn't work in the first place | |
| 23:28:57 | mriedem | but without tdurakov anymore it's hard to say | |
| 23:29:38 | melwitt | yeah. okay, I shall endeavor to dig into that a bit. because on the surface, I was just thinking to do something dumb like "if ocata, then use old way" | |
| 23:29:51 | melwitt | if it's even possible to check "if ocata" in that script | |
| 23:35:25 | mriedem | the branch is available, the devstack-gate vars are in there | |
| 23:35:43 | mriedem | but yeah i thought similar, basically we'd need to revert a patch that removed the old functions it was using | |
| 23:35:50 | mriedem | and then put in branching logic for doing old vs new things | |
| 23:36:02 | mriedem | did you get the quotas changes done that you were talking about? | |
| 23:36:08 | mriedem | for dict something or other | |
| 23:51:50 | melwitt | mriedem: still working through it now. ran into some snags (of course) on the bottom patch. now it should be mechanical changes going through the stack. so I hope to push it up in an hour or so | |
| 23:52:36 | melwitt | this series has been such a pain | |
| 23:54:44 | mriedem | ok. i decided that between two evils on my todo list, doing forum session recaps was easier for now. :) | |
| 23:54:53 | mriedem | and i can continue to say i'm not reviewing your series becaues i'm waiting for you to update it | |
| 23:55:19 | melwitt | yup, that you can. | |
| #openstack-nova - 2017-05-19 | |||
| 00:15:16 | melwitt | dansmith: is it ever possible to change the return type of a remotable method? like is there a way for new code to know old code is calling so the return type can be formatted to what the old code expects? | |
| 00:15:53 | dansmith | melwitt: nope | |
| 00:16:10 | dansmith | pass a flag or create a new method | |
| 00:16:14 | dansmith | ideally the latter | |
| 00:16:28 | melwitt | okay. so to change all count() to return a dict instead of an int, I have to do something like that | |
| 00:16:44 | dansmith | yeah | |
| 00:17:28 | melwitt | I'm trying to think if it's even possible for old code to call new code count(). it's only called in nova-api | |
| 00:17:42 | melwitt | Quotas.count() | |
| 00:17:44 | dansmith | if you change it you should make a new method | |
| 00:17:48 | dansmith | don't cheat :) | |
| 00:18:01 | melwitt | :( | |
| 00:18:19 | melwitt | the hard part is thinking of a name that isn't dumb | |
| 00:18:39 | dansmith | count_less_sucky_than_the_first_attempt() | |
| 00:18:44 | melwitt | lol, I wish | |
| 00:19:36 | melwitt | maybe count_as_dict since probably peeps will expect an integer | |
| 00:19:56 | dansmith | oh please please call it count_dicts() | |
| 00:21:35 | melwitt | I don't get it | |
| 00:24:17 | cfriesen | I think that's cause you're too mature | |
| 00:24:43 | melwitt | I shall use that excuse :) | |
| 00:26:26 | dansmith | I shall remain immature and unashamed | |
| 00:33:10 | fishbone_ | Hello all, I am having a serious problem. I am performing an upgrade from Newton to Ocata and every time I run the upgrade for nova-compute I receive "ERROR: could not access cell mapping database - has api db been created?" | |
| 00:35:06 | cfriesen | fishbone: have you read the nova upgrade notes in the release notes? | |
| 00:35:50 | fishbone_ | cfriesen: yes I believe so, it only seemed to mention designate changes | |
| 00:37:37 | fishbone_ | I see all of my nova services except for the compute services are up | |
| 00:37:58 | fishbone_ | I have tried creating the cells database manually and also by using the simple_cells command | |
| 00:38:07 | cfriesen | fishbone: I think the section you want begins with "A new nova-manage command has been added which will upgrade a deployment to cells v2..." at https://docs.openstack.org/releasenotes/nova/ocata.html | |
| 00:39:03 | cfriesen | dansmith: is this correct? ^ | |
| 00:39:11 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 00:42:23 | fishbone_ | cfriesen: Thanks for the link, although I tried that method as well. I have my system backed up and was able to retry using that command from scratch. Also the issue only seems to lie with the compute hosts, my controller node can access the api database fine | |
| 00:42:40 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 00:53:38 | openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954 | |
| 01:20:23 | mriedem | fishbone_: "could not access cell mapping database" has nothing to do with the computes | |
| 01:20:31 | mriedem | you're getting that from running nova-manage db sync right? | |
| 01:20:53 | mriedem | nova.conf on the compute node doesn't need access to the API database (or any database for that matter) | |
| 01:22:00 | fishbone_ | mriedem: I didn't think so either, however; I don't get the error when syncing the database, that seems to work with no errors. I get it when upgrading nova-compute on the compute nodes | |
| 01:22:34 | mriedem | why are you running nova-manage db sync on the compute nodes? | |
| 01:23:33 | mriedem | also fyi https://docs.openstack.org/developer/nova/cells.html#upgrade-minimal | |
| 01:24:02 | mriedem | what you're hitting is this https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L649 | |
| 01:24:11 | mriedem | when it's trying to automatically sync the nova_cell0 database | |
| 01:25:16 | mriedem | anyway, you shouldn't need to sync the db from the compute nodes | |
| 01:28:51 | fishbone_ | mriedem, are you saying the error can be ignored, I don't think I am able to compete the nova-compute installation because of this error | |
| 01:29:16 | fishbone_ | mriedem: Is there a way to suppress that sync process? | |
| 01:29:40 | mriedem | i'm saying you don't need to sync the database from a compute node | |
| 01:29:49 | mriedem | the compute node doesn't have direct access to the database | |
| 01:29:52 | mriedem | or shouldn't | |
| 01:33:30 | fishbone_ | mriedem: Do you know why I may be getting that error on a compute host then? | |
| 01:34:04 | fishbone_ | mriedem: I get the error even if I do a fresh install of nova-compute after removing old cofig files | |
| 01:34:10 | fishbone_ | config* | |
| 01:34:52 | mriedem | i assume because the compute host doesn't have direct access to the cell0 database | |
| 01:35:06 | mriedem | but it's weird that the sync on the nova (cell) database would not fail in that case | |
| 01:35:45 | mriedem | https://docs.openstack.org/ocata/install-guide-ubuntu/nova-compute-install.html doesn't say anything about running db sync from the compute host though | |
| 01:35:53 | mriedem | that's all done on the controller https://docs.openstack.org/ocata/install-guide-ubuntu/nova-controller-install.html | |
| 01:37:22 | fishbone_ | mriedem: Yes it is strange, I get the error at the apt install nova-compute" step | |
| 01:37:43 | mriedem | hmm, so that's something baked into the package then? | |
| 01:37:53 | mriedem | like a post-install script as part of the package? | |
| 01:38:07 | mriedem | are you using ubuntu packages? | |
| 01:38:22 | fishbone_ | mriedem: yes, I am using the ubuntu repositories | |
| 01:39:14 | openstackgerrit | Jianghua Wang proposed openstack/nova master: Skip policy rules on attach_network for none network allocation https://review.openstack.org/465903 | |
| 01:40:39 | mriedem | fishbone_: ok well those seem wrong, i'd probably report a bug against the ubuntu nova package LP site | |
| 01:41:01 | mriedem | fishbone_: on your control node, if you run "nova-manage cell_v2 list_cells" do you see 2 listed? | |