| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-03 | |||
| 18:12:41 | mriedem | but zuul has been zuuling it up | |
| 18:12:58 | mriedem | https://review.openstack.org/#/c/507918/ | |
| 18:13:54 | cdent | that’s the recreate patch, right, not the “here’s my theory” patch? | |
| 18:14:38 | mriedem | it's a recreate yeah | |
| 18:14:44 | mriedem | to see if we can find out from the logs where the 409 comes from | |
| 18:14:49 | mriedem | what triggers it i mean | |
| 18:15:05 | cdent | ✔ | |
| 18:15:19 | cdent | cool, I just wanted to be sure I hadn’t missed a revelation | |
| 18:18:53 | mriedem | cdent: if you scroll down to the bottom http://logs.openstack.org/18/507918/4/check/legacy-tempest-dsvm-neutron-full/8f969e2/logs/devstacklog.txt.gz you'll see what's going on | |
| 18:19:03 | mriedem | my check is failing, because i've apparently failed at bash | |
| 18:19:56 | mriedem | sdague: maybe you know how to bashtastify this ^ https://review.openstack.org/#/c/507918/5/stack.sh | |
| 18:20:09 | cdent | bash failing is an easy thing to do | |
| 18:21:56 | sdague | mriedem: I can look after, I have a call think shortly | |
| 18:22:19 | mriedem | it could just be we didn't actually fail, or i timed out too early | |
| 18:24:37 | mriedem | this is where it starts in the scheduler http://logs.openstack.org/18/507918/4/check/legacy-tempest-dsvm-neutron-full/8f969e2/logs/screen-n-sch.txt.gz#_Sep_28_20_12_08_486249 | |
| 18:24:51 | mriedem | look at all of those beautiful uuids | |
| 18:25:55 | mriedem | i seem to have also totally f'ed up the logging of the request id | |
| 18:26:06 | mriedem | because the req-47868c4f-61d3-4c32-9ece-94fb6ad35404 used for the instance create it also being logged for running periodic tasks... | |
| 18:27:30 | mriedem | cdent: heh, yeah, i think it actually scheduled the 500 instances | |
| 18:27:44 | cdent | blargh | |
| 18:28:05 | mriedem | yup | |
| 18:28:09 | cdent | you want it to break, no break. want it to work, break. | |
| 18:28:10 | mriedem | "Starting instance..." shows up 500 times in n-cpu | |
| 18:28:20 | mriedem | well, i could bump it to 1000 instances | |
| 18:28:41 | dansmith | lol | |
| 18:30:43 | mriedem | ok bombs away with 1000 | |
| 18:32:56 | mriedem | so there is no unique constraint on instance_actions.request_id, | |
| 18:33:00 | mriedem | but we key off it in the API https://developer.openstack.org/api-ref/compute/#show-server-action-details | |
| 18:33:11 | mriedem | so there should probably be a unique constraint on that column yes? | |
| 18:34:19 | mriedem | comment in the code even says, | |
| 18:34:20 | mriedem | "The intention is that there will only be one of these per user request. A | |
| 18:34:21 | mriedem | lookup by (instance_uuid, request_id) should always return a single result." | |
| 18:34:35 | mriedem | but don't bother enforcing that in the table schema... | |
| 18:35:02 | exarr | Anyone around I can ask about rabbit connection problems? :-( | |
| 18:35:06 | exarr | Invalid credentials it says. So I readd the user, change password, check the transtport_url details, all seems correct. | |
| 18:35:09 | exarr | rabbit logs say "AMQPLAIN login refused: user 'openstack' - invalid credentials" | |
| 18:35:14 | exarr | Seems clear cut, huh? | |
| 18:40:55 | mriedem | exarr: are you using ocata+? | |
| 18:41:21 | mriedem | if so, you have to make sure the transport_url in your cell mappings records match | |
| 18:45:26 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP(5): PowerVM driver: ovs vif https://review.openstack.org/422512 | |
| 18:56:28 | exarr | mriedem: Ocata, yes. | |
| 18:56:35 | exarr | Just updated to latest today | |
| 18:56:57 | exarr | transport_url in the cell mapping? oooh. Let me have a look, thanks :-) | |
| 18:59:10 | mriedem | nova-manage cell_v2 list_cells | |
| 18:59:49 | mriedem | exarr: if you had to change the transport_url in config, the cell mappings are probably stale | |
| 18:59:58 | mriedem | and that's what's being used when switching rpc context at runtime | |
| 19:00:23 | mriedem | you can use the nova-manage cell_v2 update_cell command to update the transport_url for a given cell if needed | |
| 19:00:59 | mriedem | esberglu: i've left some comments in https://review.openstack.org/#/c/503061/ but you can address those in a follow up if you want, or don't, whatever | |
| 19:05:05 | openstackgerrit | Merged openstack/nova-specs master: PowerVM Driver Integration (Queens) https://review.openstack.org/503061 | |
| 19:05:17 | exarr | mriedem: hmmm. I think I have updated as you suggested. However I am still encountering the same problem. | |
| 19:07:09 | exarr | It's very much as though the credentials are simply incorrect, or rabbit is not allowing the authentication ptotocol. | |
| 19:07:15 | exarr | *protocol | |
| 19:07:37 | exarr | Can I check the cell mapping details somehow? :-/ | |
| 19:08:02 | mriedem | nova-manage cell_v2 list_cells --verbose | |
| 19:08:19 | mriedem | will dump the cells and their transport_url as they are stored in the db | |
| 19:08:36 | mriedem | keep in mind that if you update the cell's transport_url, you'll have to restart some services because those values are cached in memory | |
| 19:08:55 | exarr | mriedem: ahhh. I'll restart the machine, that should help. | |
| 19:09:01 | exarr | thanks for helping a noob. | |
| 19:10:09 | esberglu | mriedem: I can fix it quick in that review. Unless you guys don't want to have to re-review, you seem busy atm | |
| 19:10:24 | esberglu | In that case I'll take care of it later | |
| 19:10:28 | mriedem | exarr: also fyi https://docs.openstack.org/nova/pike/user/cells.html#faqs | |
| 19:10:47 | mriedem | see #2 | |
| 19:11:21 | exarr | mriedem: I could bloody kiss you. | |
| 19:11:29 | dansmith | ew, a bloody kiss? | |
| 19:11:31 | dansmith | sounds terrible. | |
| 19:11:40 | mriedem | only on goth thursdays please | |
| 19:11:47 | exarr | :-) | |
| 19:23:27 | openstackgerrit | Chris Dent proposed openstack/nova-specs master: Add spec for symmetric GET and PUT of allocations https://review.openstack.org/508164 | |
| 19:28:19 | penick | And now I have an image of mriedem dressed up as The Crow. | |
| 19:29:02 | cdent | penick: my eyes! | |
| 19:29:09 | penick | You're welcome. | |
| 19:32:27 | openstackgerrit | Chris Dent proposed openstack/nova-specs master: Add spec for symmetric GET and PUT of allocations https://review.openstack.org/508164 | |
| 19:33:30 | melwitt | I wish I knew why we didn't already have disk quota in nova | |
| 19:34:14 | penick | I thought it used to be there, but left when cinder split out | |
| 19:34:27 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] gabbi tests for shared custom resource class https://review.openstack.org/485209 | |
| 19:35:00 | melwitt | let me check the archived scrolls | |
| 19:35:34 | penick | I think it was in folsom, but got 'et by the Grizzly | |
| 19:35:34 | cdent | I’m now picturing melwitt (in the boots) sitting in dark loft with mriedem as crow “I wish I knew why…” | |
| 19:36:13 | cdent | mriedem gets up slowly from his crushed velvet chair | |
| 19:36:30 | cdent | fist clenched he walks to the broken window | |
| 19:36:34 | cdent | and shouts into the city | |
| 19:36:37 | melwitt | lol | |
| 19:37:06 | penick | I just assumed that was how his home office was configured | |
| 19:37:09 | cdent | WE STIILL DON’T KNOW | |
| 19:37:24 | sdague | I really don't think it was ever implemented | |
| 19:37:25 | penick | mriedem is an enigma wrapped in a tortilla | |
| 19:37:34 | sdague | I thought I ran this code stream a couple months ago | |
| 19:38:19 | sdague | because quota only matters for the things you have the least of | |
| 19:38:52 | sdague | using local disk that's just there doing nothing on computers was never the limiting factor for rax | |
| 19:38:52 | melwitt | yeah, so far not finding anything in the folsom-eol tag | |
| 19:38:54 | mriedem | disk quota also gets goofed up when you're talking about boot from volume | |
| 19:39:14 | sdague | mriedem: right, there is quota on volumes, because that's off the expensive storage units | |
| 19:39:25 | mriedem | is it that time of the week to link in that change again? | |
| 19:39:39 | cdent | the words dripped from mriedem’s black lips | |
| 19:39:40 | cdent | boot | |
| 19:39:41 | cdent | from | |
| 19:39:44 | cdent | volume | |
| 19:39:49 | mriedem | https://review.openstack.org/#/c/428481/ | |
| 19:40:02 | melwitt | so not only will ppl be complaining about consuming > 0 disk with BFV they'll also complain about consuming > 0 quota with BFV | |
| 19:40:28 | sdague | heh | |