Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-05-18
21:51:57 cfriesen mriedem: yeah, I'm not certain it's forced down but it's the only thing that makes sense
21:52:06 mriedem do you have the service record from the db?
21:53:06 cfriesen mriedem: could probably track it down. would indeed be able to tell from that. I'm just reviewing a proposed change, wasn't actually my issue. one of the guys wants to bypass the check and I'm not a big fan.
21:53:24 mriedem i mean you could PUT /os-services/force-down and mark that service as forced_down=False, which would change the min service version check to say mitaka is the oldest, but that might give you other issues if other things are running thinking that newton is the minimum
21:54:34 cfriesen we're not using auto-rpc versioning (yet, at least) so during an uprade I think it'd be okay as long as we haven't set the rpc version to newton yet. will have to go get more info I think.
21:56:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip ceph in grenade live migration job due to restart failures https://review.openstack.org/466097
22:00:08 mriedem cfriesen: did you happen to try this out in your live migration recreate with the 16 ports and network-changed event issues? https://review.openstack.org/#/c/465787/
22:00:33 mriedem before i go too far down that rabbit hole (probably next week at this rate) it'd be good to know if it helps at all
22:08:00 cfriesen mriedem: related to the other issue, it turns out that if you force the "old" nova-compute down, then try to remove the "force-down" it will fail because Service.save() checks the minimum version
22:08:28 cfriesen mriedem: haven't tried that patch yet, but I plan to.
22:08:29 mriedem oh that's a fun chicken and egg
22:08:45 cfriesen isn't it just. :)
22:14:54 mriedem well, seems like a bug to report. we could bypass the service min version check on save() i'd think, but i'm not sure what kind of weird situations that puts us in
22:15:35 mriedem again, like if you've already restarted control services that are caching what they think is the min service version (newton) and all of a sudden you have a running mitaka compute service
22:15:36 cfriesen mriedem: just for fun, the original scenario was that we were trying to take down the compute node to upgrade it, so we marked nova-compute as forced-down, then we tried to down the neutron agent and hit an exception, so we tried to roll back and enable nova-compute and hit exception.ServiceTooOld
22:16:11 cfriesen not sure there's a good solution here.
22:16:11 mriedem cfriesen: why not just disable the service?
22:16:33 mriedem PUT /os-services/disable-log-reason
22:16:41 mriedem that should take that nova-compute out of scheduling
22:17:36 cfriesen mriedem: that's a good point...usually we only force down on a fault. will follow up.
22:21:18 mriedem honestly i'm not sure what the use case is for forced-down which is different from disabling the service
22:21:49 mriedem i think it has to do with external tooling forcing the service to be 'down' rather than waiting for the service group api check to do it
22:22:01 mriedem we could document that better in the api ref
22:22:21 mriedem like, what does doing this actually mean, what impacts / side effects does it have, etc
22:22:31 mriedem cfriesen: ^ probably another bug to open for lack of clear docs on these
22:22:47 mriedem https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/mark-host-down.html was the spec
22:23:34 mriedem oh i see, the idea is external service finds out the host is about to die, forces it down and then evacuates the instances off that host
22:23:41 mriedem since evacuate checks this flag
22:24:11 mriedem we should state in the api-ref that forced-down is not something for routine upgrade/maintenance
22:24:44 mriedem as documented here https://docs.openstack.org/ops-guide/ops-maintenance-compute.html#planned-maintenance
22:30:02 mriedem cfriesen: https://bugs.launchpad.net/nova/+bug/1691871
22:30:02 openstack Launchpad bug 1691871 in OpenStack Compute (nova) "forced-down vs service disable is not documented well in the compute API reference" [Medium,Confirmed]
22:37:30 cfriesen mriedem: it's actually kind of ironic (always feels funny using that word around openstack) since we were one of the groups pushing for the introduction of the "force-down" feature. now looks like someone's misusing it.
22:39:20 cfriesen mriedem: in our environment we have something else that monitors compute node health, so it'll detect the loss of a compute node way earlier than nova will, hence the proper use of force-down
22:40:49 cfriesen agree with your bug though
22:55:26 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip ceph in grenade live migration job due to restart failures https://review.openstack.org/466097
22:55:30 mriedem1 melwitt: dansmith: ^ "fixes" the grenade live migration job
22:55:43 mriedem1 we lose ceph grenade + live migration until the bug is fixed or we get to queens,
22:55:53 mriedem1 but we at least get back testing for mixed-level compute upgrade testing
22:56:00 mriedem1 *live migration testing i mean
22:56:01 mriedem1 on mixed computes
22:56:12 mriedem1 regular multinode grenade doesn't test live migration
22:56:39 mriedem1 ps2 had a clean run, ps3 just removed a debug echo statement
23:24:01 melwitt mriedem: +2
23:24:51 melwitt I'm interested in fixing the bug but I don't understand your comment about using common functions like in https://github.com/openstack-dev/devstack/commit/c006bbdeb26df2c60f43d222bdf918f9e24d551f
23:28:12 mriedem sdague said there were some common process start/stop/restart functions in devstack and we could overhaul these scripts to use those,
23:28:20 mriedem which would hide the screen vs systemd stuff i think
23:28:34 melwitt ah, okay
23:28:36 mriedem but if those are available, i'm not sure why these live migration scripts weren't using them in the first place
23:28:47 mriedem so i assume there was a good reason those functions from devstack wouldn't work in the first place
23:28:57 mriedem but without tdurakov anymore it's hard to say
23:29:38 melwitt yeah. okay, I shall endeavor to dig into that a bit. because on the surface, I was just thinking to do something dumb like "if ocata, then use old way"
23:29:51 melwitt if it's even possible to check "if ocata" in that script
23:35:25 mriedem the branch is available, the devstack-gate vars are in there
23:35:43 mriedem but yeah i thought similar, basically we'd need to revert a patch that removed the old functions it was using
23:35:50 mriedem and then put in branching logic for doing old vs new things
23:36:02 mriedem did you get the quotas changes done that you were talking about?
23:36:08 mriedem for dict something or other
23:51:50 melwitt mriedem: still working through it now. ran into some snags (of course) on the bottom patch. now it should be mechanical changes going through the stack. so I hope to push it up in an hour or so
23:52:36 melwitt this series has been such a pain
23:54:44 mriedem ok. i decided that between two evils on my todo list, doing forum session recaps was easier for now. :)
23:54:53 mriedem and i can continue to say i'm not reviewing your series becaues i'm waiting for you to update it
23:55:19 melwitt yup, that you can.
#openstack-nova - 2017-05-19
00:15:16 melwitt dansmith: is it ever possible to change the return type of a remotable method? like is there a way for new code to know old code is calling so the return type can be formatted to what the old code expects?
00:15:53 dansmith melwitt: nope
00:16:10 dansmith pass a flag or create a new method
00:16:14 dansmith ideally the latter
00:16:28 melwitt okay. so to change all count() to return a dict instead of an int, I have to do something like that
00:16:44 dansmith yeah
00:17:28 melwitt I'm trying to think if it's even possible for old code to call new code count(). it's only called in nova-api
00:17:42 melwitt Quotas.count()
00:17:44 dansmith if you change it you should make a new method
00:17:48 dansmith don't cheat :)
00:18:01 melwitt :(
00:18:19 melwitt the hard part is thinking of a name that isn't dumb
00:18:39 dansmith count_less_sucky_than_the_first_attempt()
00:18:44 melwitt lol, I wish
00:19:36 melwitt maybe count_as_dict since probably peeps will expect an integer
00:19:56 dansmith oh please please call it count_dicts()
00:21:35 melwitt I don't get it
00:24:17 cfriesen I think that's cause you're too mature
00:24:43 melwitt I shall use that excuse :)
00:26:26 dansmith I shall remain immature and unashamed
00:33:10 fishbone_ Hello all, I am having a serious problem. I am performing an upgrade from Newton to Ocata and every time I run the upgrade for nova-compute I receive "ERROR: could not access cell mapping database - has api db been created?"
00:35:06 cfriesen fishbone: have you read the nova upgrade notes in the release notes?
00:35:50 fishbone_ cfriesen: yes I believe so, it only seemed to mention designate changes
00:37:37 fishbone_ I see all of my nova services except for the compute services are up
00:37:58 fishbone_ I have tried creating the cells database manually and also by using the simple_cells command
00:38:07 cfriesen fishbone: I think the section you want begins with "A new nova-manage command has been added which will upgrade a deployment to cells v2..." at https://docs.openstack.org/releasenotes/nova/ocata.html
00:39:03 cfriesen dansmith: is this correct? ^
00:39:11 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954
00:42:23 fishbone_ cfriesen: Thanks for the link, although I tried that method as well. I have my system backed up and was able to retry using that command from scratch. Also the issue only seems to lie with the compute hosts, my controller node can access the api database fine
00:42:40 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954
00:53:38 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi: cached images should be cleaned up by time https://review.openstack.org/465954
01:20:23 mriedem fishbone_: "could not access cell mapping database" has nothing to do with the computes
01:20:31 mriedem you're getting that from running nova-manage db sync right?
01:20:53 mriedem nova.conf on the compute node doesn't need access to the API database (or any database for that matter)
01:22:00 fishbone_ mriedem: I didn't think so either, however; I don't get the error when syncing the database, that seems to work with no errors. I get it when upgrading nova-compute on the compute nodes
01:22:34 mriedem why are you running nova-manage db sync on the compute nodes?
01:23:33 mriedem also fyi https://docs.openstack.org/developer/nova/cells.html#upgrade-minimal

Earlier   Later