Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-05
16:29:41 cdent that the problem hasn't come up sooner is odd
16:29:49 cdent you must be doing something faster than other things are
16:30:05 mordred yah. it's still interesting why it only started showing up friday
16:30:29 cdent did reverting that change you reverted make any difference?
16:30:32 mriedem i don't see anything in devstack or nova around friday that would cause the start time issue
16:30:35 mordred cdent: we have a very low overhead to start running tests and our tests are all pretty quick - so I think we're a really good race from end-of-devstack to "and now stuff is running"
16:31:32 mordred mriedem: onlything I can think of is that something about the fix for the stacktrace maybe also caused something to not issue an extra registration sync
16:31:57 mriedem my patch stopped doing the pointless update_instance_info rpc cast from n-cpu to n-sch
16:31:59 mordred so maybe there was an accidental retry happening that the stacktrace fix removed?
16:31:59 cdent periodic job timing?
16:32:00 mriedem sending an empty list
16:32:38 mriedem the periodic kicks in ~1 minute after init_host on the compute
16:33:11 mriedem if update_instance_info were an rpc call that would make sense why things were slower during n-cpu startup, but it's a cast
16:33:17 cdent If I remember right, init_host calls all the same stuff that the periodic does/
16:33:33 jaypipes edleafe, dansmith, cdent, alex_xu: https://etherpad.openstack.org/p/placement-allocations-straw-man
16:33:57 mriedem cdent: the pre_start_hook calls update_available_resource
16:34:06 mriedem which calls the RT to create the ComputeNode record and do some things yeah
16:34:19 mriedem that's different from the update_instance_info stuff cast to the scheduler
16:34:24 cdent thanks jaypipes, I'm going home, will look from there, where the lack of people ought to allow proper attention
16:34:29 mriedem ^ is all about server group affinity/anti-affinity filters
16:34:30 jaypipes kk
16:35:13 dansmith jaypipes: can I reset colors on that so we can see only deltas from this?
16:35:21 cdent dansmith++
16:35:22 jaypipes dansmith: go for it.
16:35:25 cfriesen_ there are so many races around server group affinity/antiaffinity it's not even funny
16:37:36 cfriesen_ jaypipes: is it possible to have floating point values for resources?
16:37:45 cfriesen_ or is it integer only?
16:37:59 jaypipes cfriesen_: integer only.
16:39:03 cfriesen_ jaypipes: we currently allow a mix of instances with shared/dedicated cpus on the same compute node (with cpusets to keep the affinity straight) and so report usage as floating point. I guess we'll move to fixed point with placement.
16:43:35 edleafe jaypipes: running out for a bit, so I'll go more in depth when I get back.
16:44:27 sdague mriedem: https://review.openstack.org/#/c/467242/ is the glance request_id change ready to go in
16:46:31 jaypipes cfriesen_: what do you mean "report usage as floating point"?
16:51:51 jaypipes dansmith, mriedem: the three of us are having a very Reggae-colored conversation on this etherpad.
16:52:35 mriedem wish i had some ganja
16:53:39 jaypipes mriedem: if you lived in Sarasota...
16:56:12 cfriesen_ jaypipes: we report the CPU usage of a compute node as sum(dedicated vcpus) + sum(shared vcpus)/cpu_allocation_ratio. The total cannot be more than the available CPUs on the compute node.
16:57:27 cfriesen_ jaypipes: with integer usage we'll just have to multiply everything by cpu_allocation_ratio
16:57:36 jaypipes cfriesen_: yup.
17:04:04 cfriesen_ who's responsible for releasenotes issues? My commit https://review.openstack.org/#/c/435141 keeps failing jenkins because it chokes on an older releasenotes file. Something seems to have gotten more strict recently.
17:04:15 mriedem sdague: +2
17:05:22 mriedem cfriesen_: open a novaclient bug
17:06:56 mriedem my guess would be something with sphinx 1.6.1 or 1.6.2
17:07:01 mriedem 1.6.1 is blacklisted but 1.6.2 isn't
17:07:08 cfriesen_ mriedem: okay
17:08:22 mriedem jaypipes: we should have a separate bp and spec for this new placement api thing, the claims in the scheduler bp will depend on it
17:08:42 mriedem but i assume you want to communicate it to the ML so it sinks in a bit first
17:12:13 sdague cfriesen_: it might be worth checking with dhellman or the like
17:12:18 mriedem i know the fix
17:12:20 mriedem just need the bug
17:12:24 mriedem testing locally
17:13:02 mriedem not sure why it passed before though
17:13:54 cfriesen_ mriedem: https://bugs.launchpad.net/python-novaclient/+bug/1695939
17:13:54 openstack Launchpad bug 1695939 in python-novaclient "jenkins failing on older releasenote file" [Undecided,New]
17:14:46 cfriesen_ mriedem: I tried locally modifying test-requirements.txt but it still installed sphinx 1.6.2. odd
17:14:57 mriedem the release note itself is bad
17:15:14 cfriesen_ mriedem: yeah, I tried fixing it but couldn't figure out how to get it to pass
17:15:15 mriedem 1. there is no 'update' section
17:15:21 mriedem 2. it's not formatted for rst
17:15:59 sdague dansmith / melwitt: can either / both of you look at this devstack fix for cells that was proposed - https://review.openstack.org/#/c/470598
17:16:07 jaypipes mriedem: dansmith and I are on a hangout talking about something related... we're paying attention. give us a bit :)
17:16:09 cfriesen_ mriedem: for 2, do all the single dashes become asterisks?
17:16:19 mriedem cfriesen_: i'm fixing
17:25:50 openstackgerrit Chris Dent proposed openstack/nova master: [placement] Use util.extract_json in allocations handler https://review.openstack.org/471067
17:33:12 mriedem wtf i guess this is more complicated than i thought - i've made changes, committed them, rebuilt and it still picks up the old thing - probably because it's processing an old commit
17:33:18 mriedem and i can't re-write history
17:33:28 mriedem but reno should realize i'm changing something
17:39:39 cfriesen_ mriedem: bug in the CI?
17:40:17 mriedem might be a bug in reno, i'm talking to dhellmann about it
17:41:20 jaypipes OMG I want to friggin shoot myself in the head right now.
17:41:29 jaypipes I fucking hate SQL migrations.
17:41:42 jaypipes zzzeek: you around? could really use your help :(
17:41:42 dansmith sdague: done
17:43:58 zzzeek jaypipes: hey
17:44:27 jaypipes zzzeek: hi Mike. got a few minutes to help me with some very frustrating SQL migrations issues, pretty please?
17:44:31 zzzeek yes
17:44:45 sdague dansmith: thanks, it looked weird to me. Good to get the real expert view in there
17:44:52 jaypipes zzzeek: thanks :) it's about https://review.openstack.org/#/c/470645/
17:45:19 jaypipes zzzeek: I've tried a bunch of permutations (see revisions on the 044_placement_add_projects_users.py file
17:45:35 jaypipes zzzeek: and I'll fix PG and break SQLite/MySQL, and fix MySQL and fix PG.
17:45:43 jaypipes zzzeek: going round and round in circles :(
17:45:53 jaypipes zzzeek: http://logs.openstack.org/45/470645/4/check/gate-nova-tox-functional-ubuntu-xenial/0bf172a/testr_results.html.gz
17:46:15 zzzeek *obligatory grr sqlalchemy-migrate is ancient crap* irc message
17:46:23 zzzeek OK so onto the acvtual problem
17:46:29 jaypipes zzzeek: essentially, I first tried just doing an ALTER TABLE MODIFY COLUMN by changing the project_id and user_id fields in the consumers table from VARCHAR to INT.
17:46:29 cfriesen_ dansmith: If I'm reading this right, "nova-manage db online_data_migrations" returns 0 if we're successfully done all the migrations or if there were errors and we bailed out. This seems like a strange design.
17:46:43 jaypipes zzzeek: using a col.alter(...) command.
17:47:00 jaypipes zzzeek: PG hated that and requires me to drop and recreate the columns.
17:47:36 zzzeek jaypipes: it does? usually it just wants you to use a CAST so it can rewrite the data ...oh there'sa syntax for that hold on ... :)
17:47:48 jaypipes zzzeek: so I did that, then MySQL complains about indexes either existing or not existing depending on whether I drop the columns before or after dropping indexes. and when I drop indexes, PG complains there's no such indexes.
17:48:34 zzzeek jaypipes: what is the "PG hated that" error you are getting ?
17:48:42 jaypipes one sec, grabbing.
17:49:33 zzzeek jaypipes: also, your problem might be "migrate_engine.name == 'postgres'", sqlalchemy calls it "postgresql"
17:50:17 jaypipes zzzeek: no, that definitely runs only on postgresql AFAICT... it's the name of the db driver in sqlalchemy-migrate :(
17:50:29 zzzeek jaypipes: weird.
17:50:54 zzzeek jaypipes: so....why not put the whole drop/create thing inside of "if name == 'postgres'" and just use normal ALTER for the others ?
17:51:32 jaypipes zzzeek: I think I've tried that and it failed something about models being out of sync...
17:51:42 zzzeek keeping in mind that PG might just want you to say "ALTER COLUMN...USING", which is pg-specific anyway so you still need the conditional
17:51:54 jaypipes zzzeek: unfortunately, CI doesn't seem to want to allow me to see past test run failures :(
17:52:26 jaypipes zzzeek: how would I do the ALTER COLUMN ... USING thing if using sqlalchemy-migrate's col.alter() thing?
17:52:35 zzzeek jaypipes: that doesn't make any sense. program goes into the migration, and comes out, then it verifies things. if the migration works, then it worked, whether or not it did ALTER or drop/create

Earlier   Later