| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-04 | |||
| 18:34:22 | cdent | I might be reading this wrong: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L677-L682 | |
| 18:36:03 | prometheanfire | cdent: http://paste.openstack.org/show/729439/ | |
| 18:37:42 | cdent | thanks | |
| 18:38:00 | mriedem | well if it failed like it is in prometheanfire's paste, we'd notice | |
| 18:38:25 | mriedem | but lemme check the logs | |
| 18:38:29 | mriedem | maybe that Exception is masking it | |
| 18:38:38 | prometheanfire | mriedem: one reason why I think it's an artifact of the old install | |
| 18:39:15 | mriedem | F ME http://logs.openstack.org/08/599208/2/check/tempest-full/608d60a/controller/logs/devstacklog.txt.gz#_2018-09-02_15_04_31_949 | |
| 18:39:21 | mriedem | yup, broken since rocky | |
| 18:39:28 | mriedem | melwitt: time for RC10 | |
| 18:39:32 | cdent | ugh | |
| 18:39:36 | mriedem | well, added to and broken in rocky | |
| 18:39:59 | cdent | didn't papa python always teach us never to catch exception :( | |
| 18:40:31 | mriedem | prometheanfire: you want to create the bug this time? | |
| 18:40:38 | mriedem | it's nearly 2pm and i haven't had lunh yet | |
| 18:40:40 | mriedem | *lunch | |
| 18:40:58 | prometheanfire | mriedem: ya, I'm about to go get lunch I think | |
| 18:41:10 | prometheanfire | mriedem: just the output of my migration in the bug? | |
| 18:41:32 | cdent | if nobody else is aching to fix this tonight, I can do it tomorrow morning | |
| 18:43:19 | openstack | Launchpad bug 1790701 in OpenStack Compute (nova) "online_data_migrations fail in rocky+" [Undecided,New] | |
| 18:43:19 | prometheanfire | https://bugs.launchpad.net/nova/+bug/1790701 | |
| 18:43:20 | prometheanfire | for now | |
| 18:44:40 | prometheanfire | 0 | |
| 18:44:40 | prometheanfire | echo $? | |
| 18:44:44 | prometheanfire | LOL, exits 0 | |
| 18:44:48 | prometheanfire | I'll add that to the bug | |
| 18:45:17 | cdent | should we: a) make a devstack bug to make it be more unhappy when doing the migrations, or b) simply fix the exist code on the script? | |
| 18:46:13 | prometheanfire | imo the exit code being fixed would should make devstack unhappy when it should be unhappy | |
| 18:46:40 | cdent | yes | |
| 18:47:06 | cdent | Until you mentioned the exit code my b was going to be something else that how now passed out of my mind... | |
| 18:47:31 | prometheanfire | :D | |
| 18:47:39 | prometheanfire | fixing the error is still nice though | |
| 18:47:39 | cdent | so two fixes, in nova: exit code handling on that command, intialize the placement db properly | |
| 18:47:49 | cdent | that's backportable then | |
| 18:48:17 | prometheanfire | is that what's happening (api_db sync not init'ing the placement stuff)? | |
| 18:48:20 | mriedem | i'll hack some stuff up | |
| 18:48:28 | mriedem | this isn't api_db sync | |
| 18:48:48 | mriedem | but yes the online_data_migraitons command isn't configuring a global properly | |
| 18:48:48 | cdent | then on top of that we need address that an online placement db migration probably shouldn't be in nova's db migrations (even if placement wasn't being extracted) | |
| 18:49:06 | prometheanfire | ok | |
| 18:49:34 | prometheanfire | I'll be around if you want me to test, I should be able to take a snapshot of my master for rollback type testing too if you want | |
| 18:49:42 | prometheanfire | zfs <3 | |
| 18:49:52 | cdent | mriedem: please ping me if you can't get around to it, or if you do add me to the review please | |
| 18:50:24 | mriedem | sure | |
| 18:55:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Swallow fewer exceptions in _run_migration https://review.openstack.org/599744 | |
| 18:56:59 | mriedem | fwiw that blanket try/except has been around since the command was added https://review.openstack.org/#/c/278078/ | |
| 18:58:01 | prometheanfire | mriedem: want a paste with my error with that patch | |
| 18:58:04 | prometheanfire | it's longer | |
| 18:58:14 | mriedem | sure | |
| 18:58:43 | prometheanfire | mriedem: seems to fix the exit code problem too | |
| 18:58:56 | mriedem | yeah b/c you're not getting a nova exception | |
| 18:59:04 | mriedem | so it kills the command | |
| 18:59:07 | mriedem | i expect my patch to make devstack fail | |
| 18:59:16 | prometheanfire | I don't think it's that helpful (my paste) | |
| 18:59:17 | prometheanfire | http://paste.openstack.org/show/729440/ | |
| 18:59:27 | prometheanfire | but it's there anyway | |
| 18:59:42 | mriedem | yeah it's what i'd expect | |
| 18:59:52 | mriedem | i'll push the real fix in a follow up and then squash them | |
| 19:00:00 | mriedem | so we can see the failure in devstack and the fix in the follow up | |
| 19:01:10 | prometheanfire | wfm, going to lunch | |
| 19:01:12 | prometheanfire | you should too | |
| 19:04:29 | mriedem | eating while typing | |
| 19:04:32 | mriedem | the healthy way | |
| 19:06:53 | prometheanfire | :D | |
| 19:08:19 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 19:19:35 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: allow transferring ownership of instance https://review.openstack.org/599598 | |
| 19:33:51 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: allow transferring ownership of instance https://review.openstack.org/599598 | |
| 19:41:43 | openstackgerrit | melanie witt proposed openstack/nova stable/queens: Fix the request context in ServiceFixture https://review.openstack.org/599762 | |
| 19:41:44 | openstackgerrit | melanie witt proposed openstack/nova stable/queens: Honor availability_zone hint via placement https://review.openstack.org/599763 | |
| 19:41:45 | openstackgerrit | melanie witt proposed openstack/nova stable/queens: Improve NeutronFixture and remove unncessary stubbing https://review.openstack.org/599764 | |
| 19:41:46 | openstackgerrit | melanie witt proposed openstack/nova stable/queens: Add functional test for affinity with multiple cells https://review.openstack.org/599765 | |
| 19:41:47 | openstackgerrit | melanie witt proposed openstack/nova stable/queens: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599766 | |
| 19:56:02 | mriedem | melwitt: rather than https://review.openstack.org/#/c/599763/ i'd probably just add that small 5 LOC to whatever you need that uses it and mention it in the commit message | |
| 19:57:01 | melwitt | mriedem: ok, can do | |
| 19:57:45 | melwitt | mriedem: it's a similar deal with the NeutronFixture changes. I could alternatively add a stub_network_* method call to my test. I didn't need it on master because of the NeutronFixture changes | |
| 19:59:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Configure placement DB context manager for online_data_migrations https://review.openstack.org/599822 | |
| 20:00:50 | mriedem | how big would the stub be on the patch in queens? | |
| 20:01:01 | mriedem | but https://review.openstack.org/#/c/599764/ is a bit gross yeah | |
| 20:02:17 | mriedem | prometheanfire: https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/44/599744/1/check/tempest-dsvm-full-xenial/9691036/devstacklog.txt.gz shows that failure as expected | |
| 20:02:21 | mriedem | 2018-09-04 19:18:56.723 | ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova_cell0.projects' doesn't exist") [SQL: u'SELECT projects.id \nFROM projects \nWHERE projects.external_id = %(external_id_1)s'] [parameters: {u'external_id_1': '00000000-0000-0000-0000-000000000000'}] (Background on this error at: http://sqlalche.me/e/f405) | |
| 20:03:10 | prometheanfire | :D | |
| 20:05:19 | cdent | mriedem: is it normal for there to be NovaExceptions that we would want to only print? | |
| 20:05:27 | mriedem | i'd say no | |
| 20:05:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Configure placement DB context manager for online_data_migrations https://review.openstack.org/599744 | |
| 20:05:31 | mriedem | i've removed it | |
| 20:05:36 | cdent | huzza | |
| 20:05:37 | mriedem | ^ is the old squasharoo | |
| 20:05:37 | cdent | h | |
| 20:06:52 | mriedem | oops guess i missed that in the squash | |
| 20:08:55 | melwitt | mriedem: I mean I could add a fake_network.set_stub_network_methods(self) all to my test instead of backporting the changes that removed all of those calls | |
| 20:08:59 | melwitt | *call | |
| 20:12:44 | mriedem | melwitt: you mean like *add* to your backport test what was removed from here right? https://review.openstack.org/#/c/599764/1/nova/tests/functional/db/test_archive.py | |
| 20:12:48 | mriedem | melwitt: if so, then yes just do that | |
| 20:12:56 | mriedem | the neutron fixture backport is very weird otherwise | |
| 20:13:25 | melwitt | mriedem: yes, add fake_network.set_stub_network_methods(self) to my backport test | |
| 20:13:29 | melwitt | k | |
| 20:13:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Configure placement DB context manager for online_data_migrations https://review.openstack.org/599744 | |
| 20:14:18 | mriedem | prometheanfire: ^ should make your dreams finally come true | |
| 20:15:01 | prometheanfire | mriedem: mostly, 'nova-status upgrade check' still shows a warning | |
| 20:15:11 | mriedem | paste me | |