Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-05
17:12:24 mriedem testing locally
17:13:02 mriedem not sure why it passed before though
17:13:54 openstack Launchpad bug 1695939 in python-novaclient "jenkins failing on older releasenote file" [Undecided,New]
17:13:54 cfriesen_ mriedem: https://bugs.launchpad.net/python-novaclient/+bug/1695939
17:14:46 cfriesen_ mriedem: I tried locally modifying test-requirements.txt but it still installed sphinx 1.6.2. odd
17:14:57 mriedem the release note itself is bad
17:15:14 cfriesen_ mriedem: yeah, I tried fixing it but couldn't figure out how to get it to pass
17:15:15 mriedem 1. there is no 'update' section
17:15:21 mriedem 2. it's not formatted for rst
17:15:59 sdague dansmith / melwitt: can either / both of you look at this devstack fix for cells that was proposed - https://review.openstack.org/#/c/470598
17:16:07 jaypipes mriedem: dansmith and I are on a hangout talking about something related... we're paying attention. give us a bit :)
17:16:09 cfriesen_ mriedem: for 2, do all the single dashes become asterisks?
17:16:19 mriedem cfriesen_: i'm fixing
17:25:50 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use util.extract_json in allocations handler https://review.openstack.org/471067
17:33:12 mriedem wtf i guess this is more complicated than i thought - i've made changes, committed them, rebuilt and it still picks up the old thing - probably because it's processing an old commit
17:33:18 mriedem and i can't re-write history
17:33:28 mriedem but reno should realize i'm changing something
17:39:39 cfriesen_ mriedem: bug in the CI?
17:40:17 mriedem might be a bug in reno, i'm talking to dhellmann about it
17:41:20 jaypipes OMG I want to friggin shoot myself in the head right now.
17:41:29 jaypipes I fucking hate SQL migrations.
17:41:42 dansmith sdague: done
17:41:42 jaypipes zzzeek: you around? could really use your help :(
17:43:58 zzzeek jaypipes: hey
17:44:27 jaypipes zzzeek: hi Mike. got a few minutes to help me with some very frustrating SQL migrations issues, pretty please?
17:44:31 zzzeek yes
17:44:45 sdague dansmith: thanks, it looked weird to me. Good to get the real expert view in there
17:44:52 jaypipes zzzeek: thanks :) it's about https://review.openstack.org/#/c/470645/
17:45:19 jaypipes zzzeek: I've tried a bunch of permutations (see revisions on the 044_placement_add_projects_users.py file
17:45:35 jaypipes zzzeek: and I'll fix PG and break SQLite/MySQL, and fix MySQL and fix PG.
17:45:43 jaypipes zzzeek: going round and round in circles :(
17:45:53 jaypipes zzzeek: http://logs.openstack.org/45/470645/4/check/gate-nova-tox-functional-ubuntu-xenial/0bf172a/testr_results.html.gz
17:46:15 zzzeek *obligatory grr sqlalchemy-migrate is ancient crap* irc message
17:46:23 zzzeek OK so onto the acvtual problem
17:46:29 cfriesen_ dansmith: If I'm reading this right, "nova-manage db online_data_migrations" returns 0 if we're successfully done all the migrations or if there were errors and we bailed out. This seems like a strange design.
17:46:29 jaypipes zzzeek: essentially, I first tried just doing an ALTER TABLE MODIFY COLUMN by changing the project_id and user_id fields in the consumers table from VARCHAR to INT.
17:46:43 jaypipes zzzeek: using a col.alter(...) command.
17:47:00 jaypipes zzzeek: PG hated that and requires me to drop and recreate the columns.
17:47:36 zzzeek jaypipes: it does? usually it just wants you to use a CAST so it can rewrite the data ...oh there'sa syntax for that hold on ... :)
17:47:48 jaypipes zzzeek: so I did that, then MySQL complains about indexes either existing or not existing depending on whether I drop the columns before or after dropping indexes. and when I drop indexes, PG complains there's no such indexes.
17:48:34 zzzeek jaypipes: what is the "PG hated that" error you are getting ?
17:48:42 jaypipes one sec, grabbing.
17:49:33 zzzeek jaypipes: also, your problem might be "migrate_engine.name == 'postgres'", sqlalchemy calls it "postgresql"
17:50:17 jaypipes zzzeek: no, that definitely runs only on postgresql AFAICT... it's the name of the db driver in sqlalchemy-migrate :(
17:50:29 zzzeek jaypipes: weird.
17:50:54 zzzeek jaypipes: so....why not put the whole drop/create thing inside of "if name == 'postgres'" and just use normal ALTER for the others ?
17:51:32 jaypipes zzzeek: I think I've tried that and it failed something about models being out of sync...
17:51:42 zzzeek keeping in mind that PG might just want you to say "ALTER COLUMN...USING", which is pg-specific anyway so you still need the conditional
17:51:54 jaypipes zzzeek: unfortunately, CI doesn't seem to want to allow me to see past test run failures :(
17:52:26 jaypipes zzzeek: how would I do the ALTER COLUMN ... USING thing if using sqlalchemy-migrate's col.alter() thing?
17:52:35 zzzeek jaypipes: that doesn't make any sense. program goes into the migration, and comes out, then it verifies things. if the migration works, then it worked, whether or not it did ALTER or drop/create
17:52:49 zzzeek jaypipes: you'd have to do migrate_engine.execute("ALTER TABLE...")
17:53:01 jaypipes k
17:53:12 jaypipes sdague: is there a way to view old CI test runs?
17:53:23 sdague jaypipes: how old?
17:53:25 zzzeek jaypipes: if that's the issue. not sure if PG has done things w/ the indexes that are more of a big deal to change
17:53:28 jaypipes sdague: a day
17:53:46 sdague yeh, unhide the CI and look through the jenkins comments
17:53:53 jaypipes sdague: oh, duh, thx
17:53:57 sdague the rollup of the last run is just a convenience function
17:54:36 jaypipes zzzeek: ok, here's the failure from the original alter: http://logs.openstack.org/45/470645/1/check/gate-nova-tox-functional-ubuntu-xenial/8180560/testr_results.html.gz
17:55:26 jaypipes zzzeek: that happened when I did the original here: https://review.openstack.org/#/c/470645/1/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/044_placement_add_projects_users.py
17:55:58 zzzeek jaypipes: ha! HINT: You might need to specify "USING project_id::integer". me wants another darth vader usb key
17:56:29 jaypipes zzzeek: yeah... so instead of that, I did this: https://review.openstack.org/#/c/470645/2/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/044_placement_add_projects_users.py
17:56:42 dansmith mriedem: so this passed the live migration job. Do you still want me to change it or what? https://review.openstack.org/#/c/458537/31/nova/tests/live_migration/hooks/ceph.sh
17:56:52 jaypipes zzzeek: which failed with this: http://logs.openstack.org/45/470645/2/check/gate-nova-tox-functional-ubuntu-xenial/f94cf09/testr_results.html.gz
17:57:51 zzzeek jaypipes: OK, so theres the most ancient issue in SQLA to allow index.drop(checkfirst=True), however, I can get you through that anyway, if you want to go that route. but still, "if engine != 'postgres'", then drop index
17:58:14 zzzeek jaypipes: but still, "if engine == 'postgres': migrate_engine.execute("ALTER.... USING")"
17:58:31 jaypipes zzzeek: not sure, frankly, why https://review.openstack.org/#/c/470645/2/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/044_placement_add_projects_users.py didn't work. I don't know why sqlalchemy found indexes in the consumers table to drop but when issuing the SQL statement, PG says no such index exists. :(
17:58:54 zzzeek jaypipes: looks like it's because dropping the columns drops the index
17:59:04 zzzeek jaypipes: then you go to emit DROP INDEX and PG says index does not exist
17:59:18 jaypipes zzzeek: right, which I then tried to fix in the next revision: :)
17:59:36 jaypipes https://review.openstack.org/#/c/470645/3/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/044_placement_add_projects_users.py
17:59:51 zzzeek jaypipes: oh, wait, you are assuming consumers.indexes is correct. no. this is not how that works :)
18:00:00 zzzeek jaypipes: throw this into "sqlalchemy-migrate is misleading"
18:00:02 jaypipes zzzeek: which PG then barfed on
18:00:16 zzzeek jaypipes: yeah no, consumers.indexes is not changing itself when you drop those columns
18:00:19 mordred cdent, mriedem: the shade test patch is all green!
18:00:30 jaypipes zzzeek: hmm. ok.
18:00:42 zzzeek jaypipes: so, if you want *that*, do this: "from sqlalchemy import inspect; insp = inspect(migrate_engine); indexes = insp.get_indexes("consumers")"
18:00:44 cdent yayness. I think sdague said he's fast approve the devstack change
18:00:50 mordred although linuxbridge had a sad
18:01:27 jaypipes zzzeek: I think the simplest solution is to go back to revision 1 and do the execute(ALTER TABLE ... USING) thing for Postgresql?
18:01:38 zzzeek jaypipes: that will just give you names of indexes. if it were me I'd then just say "engine.execute("DROP INDEX <name>")", howver you can also say Index("myname").drop()
18:02:03 zzzeek jaypipes: I think it is yes. but all of these different scenarios can work, you're just having very bad luck stepping on every rake in the yard :)
18:02:23 jaypipes indeed :)
18:02:47 zzzeek jaypipes: try the USING with the conditional, i can make that or the other version work for you afte that
18:02:56 zzzeek jaypipes: you don't run mysql/pg functional tests locally ?
18:03:20 sdague that linuxbridge sad was a fail in test_device_tagging
18:03:34 jaypipes zzzeek: only mysql and sqlite, not postgreSQL, no...
18:03:34 sdague mriedem: that one being tracked yet, the error looks like it's going to be a race
18:03:43 jaypipes zzzeek: suppose I should :)
18:03:58 zzzeek jaypipes: it's just a yum install away... :)
18:04:15 jaypipes yeah yeah :)
18:04:35 zzzeek jaypipes: anyway I obv. have everythign running here so i can run through it later if you still cant get it working
18:04:58 jaypipes zzzeek: I'll just install PG :)
18:05:04 zzzeek jaypipes: OK.
18:10:19 mriedem sdague: huh? link?
18:12:33 openstackgerrit Merged openstack/nova master: Be more tolerant of keystone catalog configuration https://review.openstack.org/438049
18:12:49 mriedem sdague: oh you mean this http://logs.openstack.org/42/471042/2/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/4cc66aa/console.html#_2017-06-05_17_23_22_278409

Earlier   Later