| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-03 | |||
| 17:48:33 | mriedem | that creates the event start on entry, and event finish on exit | |
| 17:48:55 | mriedem | the action record is created in the API before casting to compute where the real business happens, and that real bidness is recorded with events | |
| 17:48:57 | mriedem | on the action | |
| 17:50:06 | melwitt | bidness | |
| 17:51:04 | melwitt | okay, so I guess someone forgot to ever finish_time instance actions after creating the data model | |
| 17:51:28 | mriedem | jesus i thought this was merged in pike https://review.openstack.org/#/c/480746/ | |
| 17:51:44 | mriedem | so ^ gives an example of how a user can use these things for polling when an operation is complete | |
| 17:52:24 | melwitt | that's leet | |
| 17:53:48 | mriedem | leet? | |
| 17:55:06 | mriedem | also, note that if we ever rename methods in the compute manager which have the @wrap_instance_event decorator, we break that api | |
| 17:55:14 | dansmith | which we have, I think | |
| 17:55:17 | melwitt | like elite use of that API | |
| 17:55:19 | dansmith | hasn't gibi complained? | |
| 17:55:26 | mriedem | don't remember | |
| 17:55:41 | dansmith | I think when we moved things from compute to conductor there was a thing about that | |
| 17:56:03 | dansmith | I don't recall if there was an override to fix it or just "meh, this is an internal api" | |
| 17:56:05 | mriedem | for events or notifications? | |
| 17:56:22 | dansmith | oh yeah I guess I'm thinking of notifications | |
| 17:56:30 | dansmith | but similar deal | |
| 18:01:45 | mriedem | heh coincidentally https://bugs.launchpad.net/nova/+bug/1719561 | |
| 18:01:46 | openstack | Launchpad bug 1719561 in OpenStack Compute (nova) "Instance action's updated_at doesn't updated when action created or action event updated." [Undecided,In progress] - Assigned to Yikun Jiang (yikunkero) | |
| 18:01:56 | mriedem | i guess they found this out on their own... | |
| 18:06:00 | mriedem | huh we have no indexes on the events table | |
| 18:10:36 | mriedem | might be time to fire up the ol' create 1000 instances and list some things out of the api machine | |
| 18:11:32 | cdent | mriedem: one of the last times you did that there were 409s coming out of placement, did that ever get narrowed down, or did the rest of the world come rushing in and flush that for a while? | |
| 18:12:30 | mriedem | i've got a patch | |
| 18:12:41 | mriedem | but zuul has been zuuling it up | |
| 18:12:58 | mriedem | https://review.openstack.org/#/c/507918/ | |
| 18:13:54 | cdent | that’s the recreate patch, right, not the “here’s my theory” patch? | |
| 18:14:38 | mriedem | it's a recreate yeah | |
| 18:14:44 | mriedem | to see if we can find out from the logs where the 409 comes from | |
| 18:14:49 | mriedem | what triggers it i mean | |
| 18:15:05 | cdent | ✔ | |
| 18:15:19 | cdent | cool, I just wanted to be sure I hadn’t missed a revelation | |
| 18:18:53 | mriedem | cdent: if you scroll down to the bottom http://logs.openstack.org/18/507918/4/check/legacy-tempest-dsvm-neutron-full/8f969e2/logs/devstacklog.txt.gz you'll see what's going on | |
| 18:19:03 | mriedem | my check is failing, because i've apparently failed at bash | |
| 18:19:56 | mriedem | sdague: maybe you know how to bashtastify this ^ https://review.openstack.org/#/c/507918/5/stack.sh | |
| 18:20:09 | cdent | bash failing is an easy thing to do | |
| 18:21:56 | sdague | mriedem: I can look after, I have a call think shortly | |
| 18:22:19 | mriedem | it could just be we didn't actually fail, or i timed out too early | |
| 18:24:37 | mriedem | this is where it starts in the scheduler http://logs.openstack.org/18/507918/4/check/legacy-tempest-dsvm-neutron-full/8f969e2/logs/screen-n-sch.txt.gz#_Sep_28_20_12_08_486249 | |
| 18:24:51 | mriedem | look at all of those beautiful uuids | |
| 18:25:55 | mriedem | i seem to have also totally f'ed up the logging of the request id | |
| 18:26:06 | mriedem | because the req-47868c4f-61d3-4c32-9ece-94fb6ad35404 used for the instance create it also being logged for running periodic tasks... | |
| 18:27:30 | mriedem | cdent: heh, yeah, i think it actually scheduled the 500 instances | |
| 18:27:44 | cdent | blargh | |
| 18:28:05 | mriedem | yup | |
| 18:28:09 | cdent | you want it to break, no break. want it to work, break. | |
| 18:28:10 | mriedem | "Starting instance..." shows up 500 times in n-cpu | |
| 18:28:20 | mriedem | well, i could bump it to 1000 instances | |
| 18:28:41 | dansmith | lol | |
| 18:30:43 | mriedem | ok bombs away with 1000 | |
| 18:32:56 | mriedem | so there is no unique constraint on instance_actions.request_id, | |
| 18:33:00 | mriedem | but we key off it in the API https://developer.openstack.org/api-ref/compute/#show-server-action-details | |
| 18:33:11 | mriedem | so there should probably be a unique constraint on that column yes? | |
| 18:34:19 | mriedem | comment in the code even says, | |
| 18:34:20 | mriedem | "The intention is that there will only be one of these per user request. A | |
| 18:34:21 | mriedem | lookup by (instance_uuid, request_id) should always return a single result." | |
| 18:34:35 | mriedem | but don't bother enforcing that in the table schema... | |
| 18:35:02 | exarr | Anyone around I can ask about rabbit connection problems? :-( | |
| 18:35:06 | exarr | Invalid credentials it says. So I readd the user, change password, check the transtport_url details, all seems correct. | |
| 18:35:09 | exarr | rabbit logs say "AMQPLAIN login refused: user 'openstack' - invalid credentials" | |
| 18:35:14 | exarr | Seems clear cut, huh? | |
| 18:40:55 | mriedem | exarr: are you using ocata+? | |
| 18:41:21 | mriedem | if so, you have to make sure the transport_url in your cell mappings records match | |
| 18:45:26 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP(5): PowerVM driver: ovs vif https://review.openstack.org/422512 | |
| 18:56:28 | exarr | mriedem: Ocata, yes. | |
| 18:56:35 | exarr | Just updated to latest today | |
| 18:56:57 | exarr | transport_url in the cell mapping? oooh. Let me have a look, thanks :-) | |
| 18:59:10 | mriedem | nova-manage cell_v2 list_cells | |
| 18:59:49 | mriedem | exarr: if you had to change the transport_url in config, the cell mappings are probably stale | |
| 18:59:58 | mriedem | and that's what's being used when switching rpc context at runtime | |
| 19:00:23 | mriedem | you can use the nova-manage cell_v2 update_cell command to update the transport_url for a given cell if needed | |
| 19:00:59 | mriedem | esberglu: i've left some comments in https://review.openstack.org/#/c/503061/ but you can address those in a follow up if you want, or don't, whatever | |
| 19:05:05 | openstackgerrit | Merged openstack/nova-specs master: PowerVM Driver Integration (Queens) https://review.openstack.org/503061 | |
| 19:05:17 | exarr | mriedem: hmmm. I think I have updated as you suggested. However I am still encountering the same problem. | |
| 19:07:09 | exarr | It's very much as though the credentials are simply incorrect, or rabbit is not allowing the authentication ptotocol. | |
| 19:07:15 | exarr | *protocol | |
| 19:07:37 | exarr | Can I check the cell mapping details somehow? :-/ | |
| 19:08:02 | mriedem | nova-manage cell_v2 list_cells --verbose | |
| 19:08:19 | mriedem | will dump the cells and their transport_url as they are stored in the db | |
| 19:08:36 | mriedem | keep in mind that if you update the cell's transport_url, you'll have to restart some services because those values are cached in memory | |
| 19:08:55 | exarr | mriedem: ahhh. I'll restart the machine, that should help. | |
| 19:09:01 | exarr | thanks for helping a noob. | |
| 19:10:09 | esberglu | mriedem: I can fix it quick in that review. Unless you guys don't want to have to re-review, you seem busy atm | |
| 19:10:24 | esberglu | In that case I'll take care of it later | |
| 19:10:28 | mriedem | exarr: also fyi https://docs.openstack.org/nova/pike/user/cells.html#faqs | |
| 19:10:47 | mriedem | see #2 | |
| 19:11:21 | exarr | mriedem: I could bloody kiss you. | |
| 19:11:29 | dansmith | ew, a bloody kiss? | |
| 19:11:31 | dansmith | sounds terrible. | |
| 19:11:40 | mriedem | only on goth thursdays please | |
| 19:11:47 | exarr | :-) | |
| 19:23:27 | openstackgerrit | Chris Dent proposed openstack/nova-specs master: Add spec for symmetric GET and PUT of allocations https://review.openstack.org/508164 | |
| 19:28:19 | penick | And now I have an image of mriedem dressed up as The Crow. | |
| 19:29:02 | cdent | penick: my eyes! | |
| 19:29:09 | penick | You're welcome. | |
| 19:32:27 | openstackgerrit | Chris Dent proposed openstack/nova-specs master: Add spec for symmetric GET and PUT of allocations https://review.openstack.org/508164 | |
| 19:33:30 | melwitt | I wish I knew why we didn't already have disk quota in nova | |
| 19:34:14 | penick | I thought it used to be there, but left when cinder split out | |