| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-05 | |||
| 16:18:27 | mriedem | the comment is fine, the note is confusing | |
| 16:18:32 | cdent | k | |
| 16:19:03 | cdent | done | |
| 16:19:20 | mriedem | thanks | |
| 16:28:21 | mordred | cdent, mriedem: woot! thank you! | |
| 16:29:13 | mordred | I'm glad this turned out to be potentially an actual issue and not just a random testing weird | |
| 16:29:21 | cdent | was a good spot | |
| 16:29:41 | cdent | that the problem hasn't come up sooner is odd | |
| 16:29:49 | cdent | you must be doing something faster than other things are | |
| 16:30:05 | mordred | yah. it's still interesting why it only started showing up friday | |
| 16:30:29 | cdent | did reverting that change you reverted make any difference? | |
| 16:30:32 | mriedem | i don't see anything in devstack or nova around friday that would cause the start time issue | |
| 16:30:35 | mordred | cdent: we have a very low overhead to start running tests and our tests are all pretty quick - so I think we're a really good race from end-of-devstack to "and now stuff is running" | |
| 16:31:32 | mordred | mriedem: onlything I can think of is that something about the fix for the stacktrace maybe also caused something to not issue an extra registration sync | |
| 16:31:57 | mriedem | my patch stopped doing the pointless update_instance_info rpc cast from n-cpu to n-sch | |
| 16:31:59 | mordred | so maybe there was an accidental retry happening that the stacktrace fix removed? | |
| 16:31:59 | cdent | periodic job timing? | |
| 16:32:00 | mriedem | sending an empty list | |
| 16:32:38 | mriedem | the periodic kicks in ~1 minute after init_host on the compute | |
| 16:33:11 | mriedem | if update_instance_info were an rpc call that would make sense why things were slower during n-cpu startup, but it's a cast | |
| 16:33:17 | cdent | If I remember right, init_host calls all the same stuff that the periodic does/ | |
| 16:33:33 | jaypipes | edleafe, dansmith, cdent, alex_xu: https://etherpad.openstack.org/p/placement-allocations-straw-man | |
| 16:33:57 | mriedem | cdent: the pre_start_hook calls update_available_resource | |
| 16:34:06 | mriedem | which calls the RT to create the ComputeNode record and do some things yeah | |
| 16:34:19 | mriedem | that's different from the update_instance_info stuff cast to the scheduler | |
| 16:34:24 | cdent | thanks jaypipes, I'm going home, will look from there, where the lack of people ought to allow proper attention | |
| 16:34:29 | mriedem | ^ is all about server group affinity/anti-affinity filters | |
| 16:34:30 | jaypipes | kk | |
| 16:35:13 | dansmith | jaypipes: can I reset colors on that so we can see only deltas from this? | |
| 16:35:21 | cdent | dansmith++ | |
| 16:35:22 | jaypipes | dansmith: go for it. | |
| 16:35:25 | cfriesen_ | there are so many races around server group affinity/antiaffinity it's not even funny | |
| 16:37:36 | cfriesen_ | jaypipes: is it possible to have floating point values for resources? | |
| 16:37:45 | cfriesen_ | or is it integer only? | |
| 16:37:59 | jaypipes | cfriesen_: integer only. | |
| 16:39:03 | cfriesen_ | jaypipes: we currently allow a mix of instances with shared/dedicated cpus on the same compute node (with cpusets to keep the affinity straight) and so report usage as floating point. I guess we'll move to fixed point with placement. | |
| 16:43:35 | edleafe | jaypipes: running out for a bit, so I'll go more in depth when I get back. | |
| 16:44:27 | sdague | mriedem: https://review.openstack.org/#/c/467242/ is the glance request_id change ready to go in | |
| 16:46:31 | jaypipes | cfriesen_: what do you mean "report usage as floating point"? | |
| 16:51:51 | jaypipes | dansmith, mriedem: the three of us are having a very Reggae-colored conversation on this etherpad. | |
| 16:52:35 | mriedem | wish i had some ganja | |
| 16:53:39 | jaypipes | mriedem: if you lived in Sarasota... | |
| 16:56:12 | cfriesen_ | jaypipes: we report the CPU usage of a compute node as sum(dedicated vcpus) + sum(shared vcpus)/cpu_allocation_ratio. The total cannot be more than the available CPUs on the compute node. | |
| 16:57:27 | cfriesen_ | jaypipes: with integer usage we'll just have to multiply everything by cpu_allocation_ratio | |
| 16:57:36 | jaypipes | cfriesen_: yup. | |
| 17:04:04 | cfriesen_ | who's responsible for releasenotes issues? My commit https://review.openstack.org/#/c/435141 keeps failing jenkins because it chokes on an older releasenotes file. Something seems to have gotten more strict recently. | |
| 17:04:15 | mriedem | sdague: +2 | |
| 17:05:22 | mriedem | cfriesen_: open a novaclient bug | |
| 17:06:56 | mriedem | my guess would be something with sphinx 1.6.1 or 1.6.2 | |
| 17:07:01 | mriedem | 1.6.1 is blacklisted but 1.6.2 isn't | |
| 17:07:08 | cfriesen_ | mriedem: okay | |
| 17:08:22 | mriedem | jaypipes: we should have a separate bp and spec for this new placement api thing, the claims in the scheduler bp will depend on it | |
| 17:08:42 | mriedem | but i assume you want to communicate it to the ML so it sinks in a bit first | |
| 17:12:13 | sdague | cfriesen_: it might be worth checking with dhellman or the like | |
| 17:12:18 | mriedem | i know the fix | |
| 17:12:20 | mriedem | just need the bug | |
| 17:12:24 | mriedem | testing locally | |
| 17:13:02 | mriedem | not sure why it passed before though | |
| 17:13:54 | cfriesen_ | mriedem: https://bugs.launchpad.net/python-novaclient/+bug/1695939 | |
| 17:13:54 | openstack | Launchpad bug 1695939 in python-novaclient "jenkins failing on older releasenote file" [Undecided,New] | |
| 17:14:46 | cfriesen_ | mriedem: I tried locally modifying test-requirements.txt but it still installed sphinx 1.6.2. odd | |
| 17:14:57 | mriedem | the release note itself is bad | |
| 17:15:14 | cfriesen_ | mriedem: yeah, I tried fixing it but couldn't figure out how to get it to pass | |
| 17:15:15 | mriedem | 1. there is no 'update' section | |
| 17:15:21 | mriedem | 2. it's not formatted for rst | |
| 17:15:59 | sdague | dansmith / melwitt: can either / both of you look at this devstack fix for cells that was proposed - https://review.openstack.org/#/c/470598 | |
| 17:16:07 | jaypipes | mriedem: dansmith and I are on a hangout talking about something related... we're paying attention. give us a bit :) | |
| 17:16:09 | cfriesen_ | mriedem: for 2, do all the single dashes become asterisks? | |
| 17:16:19 | mriedem | cfriesen_: i'm fixing | |
| 17:25:50 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Use util.extract_json in allocations handler https://review.openstack.org/471067 | |
| 17:33:12 | mriedem | wtf i guess this is more complicated than i thought - i've made changes, committed them, rebuilt and it still picks up the old thing - probably because it's processing an old commit | |
| 17:33:18 | mriedem | and i can't re-write history | |
| 17:33:28 | mriedem | but reno should realize i'm changing something | |
| 17:39:39 | cfriesen_ | mriedem: bug in the CI? | |
| 17:40:17 | mriedem | might be a bug in reno, i'm talking to dhellmann about it | |
| 17:41:20 | jaypipes | OMG I want to friggin shoot myself in the head right now. | |
| 17:41:29 | jaypipes | I fucking hate SQL migrations. | |
| 17:41:42 | jaypipes | zzzeek: you around? could really use your help :( | |
| 17:41:42 | dansmith | sdague: done | |
| 17:43:58 | zzzeek | jaypipes: hey | |
| 17:44:27 | jaypipes | zzzeek: hi Mike. got a few minutes to help me with some very frustrating SQL migrations issues, pretty please? | |
| 17:44:31 | zzzeek | yes | |
| 17:44:45 | sdague | dansmith: thanks, it looked weird to me. Good to get the real expert view in there | |
| 17:44:52 | jaypipes | zzzeek: thanks :) it's about https://review.openstack.org/#/c/470645/ | |
| 17:45:19 | jaypipes | zzzeek: I've tried a bunch of permutations (see revisions on the 044_placement_add_projects_users.py file | |
| 17:45:35 | jaypipes | zzzeek: and I'll fix PG and break SQLite/MySQL, and fix MySQL and fix PG. | |
| 17:45:43 | jaypipes | zzzeek: going round and round in circles :( | |
| 17:45:53 | jaypipes | zzzeek: http://logs.openstack.org/45/470645/4/check/gate-nova-tox-functional-ubuntu-xenial/0bf172a/testr_results.html.gz | |
| 17:46:15 | zzzeek | *obligatory grr sqlalchemy-migrate is ancient crap* irc message | |
| 17:46:23 | zzzeek | OK so onto the acvtual problem | |
| 17:46:29 | jaypipes | zzzeek: essentially, I first tried just doing an ALTER TABLE MODIFY COLUMN by changing the project_id and user_id fields in the consumers table from VARCHAR to INT. | |
| 17:46:29 | cfriesen_ | dansmith: If I'm reading this right, "nova-manage db online_data_migrations" returns 0 if we're successfully done all the migrations or if there were errors and we bailed out. This seems like a strange design. | |
| 17:46:43 | jaypipes | zzzeek: using a col.alter(...) command. | |
| 17:47:00 | jaypipes | zzzeek: PG hated that and requires me to drop and recreate the columns. | |
| 17:47:36 | zzzeek | jaypipes: it does? usually it just wants you to use a CAST so it can rewrite the data ...oh there'sa syntax for that hold on ... :) | |
| 17:47:48 | jaypipes | zzzeek: so I did that, then MySQL complains about indexes either existing or not existing depending on whether I drop the columns before or after dropping indexes. and when I drop indexes, PG complains there's no such indexes. | |
| 17:48:34 | zzzeek | jaypipes: what is the "PG hated that" error you are getting ? | |
| 17:48:42 | jaypipes | one sec, grabbing. | |
| 17:49:33 | zzzeek | jaypipes: also, your problem might be "migrate_engine.name == 'postgres'", sqlalchemy calls it "postgresql" | |
| 17:50:17 | jaypipes | zzzeek: no, that definitely runs only on postgresql AFAICT... it's the name of the db driver in sqlalchemy-migrate :( | |