Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-25
19:38:07 mriedem ok i've got something hacked up for now
19:38:28 mriedem http://paste.openstack.org/show/730743/
19:39:41 sean-k-mooney mriedem: without messing with internal that looks resonably clean
19:40:17 sean-k-mooney mriedem: it would be nice to a "mark_dirty" function instead of setattr(new_obj, field, getattr(obj, field)) but that works
19:40:53 mriedem i'll post something to ovo that dan can look at while he's pooing in a cave
19:50:18 openstackgerrit Matt Riedemann proposed openstack/nova master: Option "scheduler_default_filters" is deprecated. https://review.openstack.org/604148
19:51:35 imacdonn mriedem: is there an easy way to see what 'nova-manage db online_data_migrations' would do, without actually doing it ?
19:51:37 sean-k-mooney melwitt: im going to call it a night an grab a beer. am im going to start hacking on that exection wrapper class and ill push something up in an hour or so while i wait for dinner. ill add you as a reviewer when i submit the patch.
19:52:46 mriedem imacdonn: as in a dry run option?
19:52:49 mriedem no there isn't
19:53:02 mriedem jaypipes: sean-k-mooney: like this https://review.openstack.org/605199
19:53:19 melwitt sean-k-mooney: cool, thanks
19:53:32 imacdonn mriedem: yeah. ok. I think that would be useful. I tried hacking out the check for max-count being greater than zero, but it's not that simple, apparently
19:54:27 mriedem imacdonn: well, you'd have to plumb that down through all of the online data migration routines
19:54:31 mriedem to not actually commit any changes
19:54:34 mriedem so it's not trivial
19:54:49 imacdonn mriedem: also, this doesn't seem right. The counts in the table should be non-zero? Caveat: it's Queens, so if anything's been fixed since then .... http://paste.openstack.org/show/730819/
19:55:30 sean-k-mooney mriedem: is the condition on line 401 reversed? e.g. should it be if leave_dirty?
19:57:34 sean-k-mooney mriedem: i would have expected the changed filed to be the dirty fields personally but i have not really looked at how this works closely
19:58:15 mriedem no
19:58:46 mriedem if not False (True): set the same changed fields on the copy as were on the primitive
19:58:53 mriedem which defaults to no changed fields
19:59:02 openstackgerrit Jonte Watford proposed openstack/nova master: Add __str__ method to objects NUMACell instances https://review.openstack.org/600269
20:01:07 sean-k-mooney mriedem: i have added myself as a review and ill step through it properlly tomorow.
20:01:07 openstackgerrit Merged openstack/nova stable/rocky: Follow devstack-plugin-ceph job rename https://review.openstack.org/602018
20:01:27 mriedem imacdonn: well that table definitely looks wrong...
20:01:46 imacdonn mriedem: yeah. Trying to figure out how that could happen
20:02:05 mriedem my guess is you're dumping the last result set,
20:02:11 mriedem which is when there were no more things to migrate
20:02:18 mriedem so you get 0s at the end
20:02:48 imacdonn I'm just running the command :)
20:02:54 imacdonn but yeah
20:02:59 mriedem you said you hacked something
20:03:19 imacdonn Oh, I only hacked out the check that max-count has to be >0 .. but I'm not applying that here
20:03:22 mriedem however,
20:03:23 mriedem http://logs.openstack.org/87/604687/4/check/neutron-grenade/6c10d43/logs/grenade.sh.txt.gz
20:03:30 mriedem shows that the table output is also busted in our grenade run
20:03:41 mriedem http://logs.openstack.org/87/604687/4/check/neutron-grenade/6c10d43/logs/grenade.sh.txt.gz#_2018-09-25_09_44_39_645
20:03:56 mriedem well...
20:04:02 imacdonn I think you're right .. I think that each call to _run_migration clobbers the previous result, instead of adding to it
20:04:02 mriedem maybe not, if we just don't have anything to migrate
20:05:39 mriedem that populate_uuids migration was added in queens, and looking at a grenade run from queens, we should see bdms getting migrated to have a uuid, and i'm not seeing any results in the table for a grenade run on queens
20:05:40 mriedem http://logs.openstack.org/48/604448/1/check/neutron-grenade/19d8166/logs/grenade.sh.txt.gz
20:05:44 mriedem so yeah, likely busted
20:05:47 mriedem imacdonn: please report a bug
20:05:59 imacdonn I think the clobberation is happening at https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L720
20:06:00 imacdonn will bug
20:06:32 mriedem yeah because of https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L689
20:06:36 mriedem resets everything to 0
20:07:03 imacdonn it makes a new "migrations" dict, with all of the members, for each pass
20:18:33 imacdonn https://bugs.launchpad.net/nova/+bug/1794364
20:18:33 openstack Launchpad bug 1794364 in OpenStack Compute (nova) "'nova-manage db online_data_migrations' count fail" [Undecided,New]
20:44:06 mriedem melwitt: can you this this queens change? https://review.openstack.org/#/c/600101/
20:44:15 mriedem *hit
20:44:25 melwitt sure
20:58:27 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930
21:07:41 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: add 'migrations' param to GET /os-migrations https://review.openstack.org/605232
21:18:47 mriedem https://www.openstack.org/summit/berlin-2018/vote-for-speakers#/22793
21:18:58 mriedem dansmith: ^ lemme know if you want to moderate that instead of me
21:19:02 mriedem i'm pretty sure you do
21:20:19 melwitt when is dan coming back?
21:23:54 mriedem next week
21:40:43 imacdonn Error attempting to run <function create_incomplete_consumers at 0x7f60e7498668>
21:40:57 imacdonn but it didn't log anything :/ This migrations stuff is finicky ..... ?
21:45:54 openstack Launchpad bug 1790701 in OpenStack Compute (nova) rocky "online_data_migrations fail in rocky+" [Critical,Fix committed] - Assigned to Matt Riedemann (mriedem)
21:47:44 openstackgerrit Jonte Watford proposed openstack/nova master: Add __str__ method to objects NUMACell objects https://review.openstack.org/600269
21:48:35 mriedem imacdonn: https://review.openstack.org/#/c/600085/
21:49:14 imacdonn mriedem: ack
21:49:32 mriedem just haven't updated that yet
22:00:54 melwitt mriedem: I made the mistake of thinking nova-consoleauth wasn't required anymore because the proxy code defaults to only using the db backend to verify a console. but the nova-compute code is still storing the token auth in both nova-consoleauth and the database backend, for reasons I don't know, when I took over the patch series
22:01:39 melwitt this cycle, I can remove the nova-compute code that stores the token auth in nova-consoleauth, at which point nova-consoleauth would not be needed on a fresh install
22:03:24 melwitt as for the deprecation, we could un-deprecate it
22:03:28 mriedem well,
22:03:34 mriedem if nova-compute in stein doesn't need it,
22:03:38 mriedem that would be the time to deprecate it,
22:03:45 mriedem but rocky wasn't really the right time to deprecate it sounds like
22:03:50 mriedem since something was still using it
22:04:04 mriedem i do'nt think i recall a retroactive undeprecate on stable
22:05:35 melwitt yeah, I think that's accurate, that rocky was not the right time. the only benefit to it that I can think of is, a heads up the cycle before it's no longer needed. but that's a stretch, I guess
22:08:01 mriedem i don't think there would be anything wrong with undeprecating it on stable/rocky,
22:08:12 mriedem if it's wrong, we shouldn't report it as deprecated b/c people will be getting warnings
22:08:42 mriedem we have done something like this in the past when we removed the original [placement_database] options b/c they weren't used
22:08:52 mriedem and them being in config made people think they could use them
22:08:59 imacdonn I discovered this issue because I was trying to be a good boy, and not to deprecated things, when looking at rocky upgrade, so I turned off the service, and things broke
22:09:15 imacdonn not do* deprecated things
22:09:49 melwitt yeah. I have caused a fair bit of operator pain with this mistake. sorry for that
22:09:59 mriedem yeah if there is no way to *not* use it in rocky, then saying it's deprecated (but still required) is weird
22:10:09 imacdonn not complaining .. just sharing my experience ;)
22:10:14 mriedem if ^ is true, then we likely want to undeprecate in stable/rocky
22:10:25 melwitt I'm sure it was not pleasant :P
22:10:35 mriedem gotta run
22:10:57 melwitt yeah, there's no way to not use it in rocky. I'll propose some reverts, because we need to undo the deprecation
22:16:48 melwitt s/nova-compute/nova-api/ earlier
22:20:35 openstackgerrit melanie witt proposed openstack/nova master: Revert "Deprecate the nova-consoleauth service" https://review.openstack.org/605250
22:21:30 openstackgerrit sean mooney proposed openstack/nova master: Add Result wrapper class to nova common https://review.openstack.org/605251
22:21:57 sean-k-mooney melwitt: ^ i think this is what we want for the cell gather thing
22:22:35 melwitt thanks
22:25:07 openstackgerrit melanie witt proposed openstack/nova master: Revert "Deprecate the nova-consoleauth service" https://review.openstack.org/605250
22:26:45 sean-k-mooney melwitt: no worries it was fun. i like creating special pourpose data structure like this. i rarely have a need to do it however :)
22:28:10 melwitt :)

Earlier   Later