| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 20:46:46 | melwitt | mriedem, tssurya: I told dansmith we're skipping the meeting based on the earlier convo | |
| 20:47:07 | tssurya | melwitt: ack, thanks for the info :) | |
| 20:47:09 | melwitt | he's not going to be back in time anyway | |
| 20:47:42 | mriedem | he said he would be back in time | |
| 20:47:45 | tssurya | wfm, Its too late here anyways, you guys have a good day | |
| 20:47:59 | mriedem | <3 broken | |
| 20:48:06 | melwitt | heh | |
| 20:48:19 | tssurya | :) | |
| 20:49:38 | openstack | Launchpad bug 1795966 in OpenStack Compute (nova) "<class 'oslo_db.exception.DBNonExistentTable'> (HTTP 500)" [Undecided,Invalid] | |
| 20:49:38 | mriedem | https://bugs.launchpad.net/nova/+bug/1795966 | |
| 20:50:15 | melwitt | meanwhile, gd consoleauth. we have an API where you can 'show' your console auth token. and that is making the deprecation nightmare worse. have to figure out if/how to adjust this for the database backend | |
| 20:50:51 | melwitt | mriedem: that must be the shortest bug report ever | |
| 20:50:56 | mriedem | https://developer.openstack.org/api-ref/compute/#create-remote-console ? | |
| 20:51:11 | melwitt | https://developer.openstack.org/api-ref/compute/#show-console-connection-information | |
| 20:51:31 | mriedem | ah heh | |
| 20:51:36 | melwitt | FML | |
| 20:51:56 | mriedem | well, | |
| 20:52:18 | mriedem | oh heh you can't know which cell to route it to right | |
| 20:52:23 | mriedem | b/c the token isn't mapped in the api | |
| 20:52:36 | mriedem | you'll have to iterate the cell dbs looking for that token id | |
| 20:53:05 | melwitt | no... which I'm trying to remember, what did I find last time I looked at this. arrrrgghh | |
| 20:55:03 | mriedem | hmm, we only store the hashed token in the db right | |
| 20:55:14 | mriedem | and that's not what the API would have in it? | |
| 20:55:47 | melwitt | yeah only the hashed token. and I think the API takes the unhashed token from the user | |
| 20:56:22 | mriedem | ha, cool | |
| 20:56:58 | mriedem | for cell in all_cells(): for console_auth_token in all_console_auth_tokens_in_this_cell(): if console_auth_token == req_id: do that thing() | |
| 20:58:27 | mriedem | i bet we log that unhashed token in the nova-api logs too... | |
| 20:58:31 | mriedem | since it's on the path | |
| 20:59:09 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/requestlog.py#L41 | |
| 20:59:30 | melwitt | indeed, I can see it in the func test output | |
| 20:59:43 | mriedem | ha, cool | |
| 20:59:52 | melwitt | 2018-10-03 20:37:40,870 INFO [nova.api.openstack.requestlog] 127.0.0.1 "GET /v2.1/os-console-auth-tokens/714a26ff-d7e6-4698-bc30-9934ebf38807" | |
| 21:00:17 | mriedem | well luckily logging credentials isn't a CVE | |
| 21:01:44 | melwitt | ... | |
| 21:02:27 | melwitt | I guess people aren't paying too much attention to this API, myself included | |
| 21:02:34 | mriedem | it's admin-only by default | |
| 21:02:49 | melwitt | I see, ok | |
| 21:03:11 | melwitt | Note "This is only used in Xenserver VNC Proxy." | |
| 21:03:21 | melwitt | really? I wonder how | |
| 21:03:35 | mriedem | that's for the other 4 | |
| 21:03:37 | mriedem | i saw that as well | |
| 21:03:52 | mriedem | os-console-auth-tokens was added specifically for rdp consoles for hyperv | |
| 21:04:42 | melwitt | O.o | |
| 21:09:15 | melwitt | yeah, so we could scatter-gather a ConsoleAuthToken.validate(context, token) call and only one will return token object, the others will raise exceptions. that method takes an unhashed token and will hash it before looking for it in the db | |
| 21:10:53 | mriedem | sure | |
| 21:11:05 | mriedem | shitty performance but whatareyougonnado | |
| 21:11:13 | melwitt | yeah, exactly | |
| 21:11:13 | mriedem | plus it's admin-only and no one knew it existed | |
| 21:11:21 | melwitt | haha, right. we have that going for us | |
| 21:11:45 | mriedem | is there a bug for this? | |
| 21:11:46 | melwitt | I'll add it to the pile o poopatches | |
| 21:11:51 | melwitt | no | |
| 21:11:57 | melwitt | I'll open one | |
| 21:12:22 | mriedem | cool. not sure if we should report the token logging thing or just pretend i never said it. | |
| 21:12:28 | melwitt | I was just making the changes to only access consoleauth if [workarounds] and ran into this in the func tests | |
| 21:12:46 | melwitt | so I'm doing really good here | |
| 21:14:09 | melwitt | yeah, I'm not sure either. I expect it wouldn't cause a CVE because it's been this way for years | |
| 21:19:41 | mriedem | heh, well, we've had things "this way for years" that are CVEs, | |
| 21:19:52 | mriedem | but logging credentials and tokens and such isn't considered one of them | |
| 21:20:13 | mriedem | it's a "hardening opportunity" | |
| 21:20:18 | melwitt | haha, ok | |
| 21:20:22 | openstackgerrit | Jay Pipes proposed openstack/nova stable/ocata: Re-use existing ComputeNode on ironic rebalance https://review.openstack.org/607626 | |
| 21:21:12 | openstack | Launchpad bug 1795982 in OpenStack Compute (nova) "/os-console-auth-tokens/{console_token} API doesn't handle the database backend" [High,Triaged] - Assigned to melanie witt (melwitt) | |
| 21:21:12 | melwitt | https://bugs.launchpad.net/nova/+bug/1795982 | |
| 21:30:17 | melwitt | so, these other console create/delete/get APIs are connected to cell database models, with nothing at the API level to target cells for the consoles | |
| 21:31:38 | melwitt | "nova-console, which is a XenAPI-specific service that most recent VNC proxy architectures do not use." | |
| 21:32:34 | melwitt | it sounds like that should be deprecated. we didn't do anything to handle it in a cells v2 world | |
| 21:35:17 | melwitt | maybe I should send something to the ML to ask about it | |
| 21:45:37 | mriedem | i thought the xvp stuff was xen-only | |
| 21:46:21 | melwitt | yeah, the nova-console service is xen-only. but if someone ran multi-cell with xen, the nova-console part wouldn't work right | |
| 21:46:25 | mriedem | but yeah this is clearly busted in a cells v2 world | |
| 21:47:17 | melwitt | so the question will be, do we cells-v2-ify it or do we deprecate it. tbc, this is for the other APIs, not the consoleauth one I'm fixing | |
| 21:47:35 | melwitt | *the other 4 | |
| 21:48:50 | mriedem | yeah i know | |
| 21:50:10 | mriedem | idk, i've asked about killing xvp in the past | |
| 21:50:13 | mriedem | no one seems to know | |
| 21:50:21 | melwitt | ah, ok | |
| 21:51:03 | mriedem | i'd say if there are alternatives available for xenapi users, then we should deprecate it | |
| 21:51:21 | mriedem | so probably a question for naichuans and BobBall | |
| 21:51:52 | melwitt | yeah, that's what I wasn't sure about, because IIUC, xenapi has to use some ancient version of stuff, so they might actually need it because they can't use newer VNC | |
| 21:51:56 | mriedem | and yeah send something to the dev and ops MLs | |
| 21:52:06 | mriedem | oh b/c of python 2.4? | |
| 21:52:20 | mriedem | i might be thinking of something else | |
| 21:52:39 | mriedem | i guess start with the ML | |
| 21:52:41 | melwitt | maybe. when stephenfin worked on the encrypted console stuff, he had to exclude xenapi from the version requirement IIRC | |
| 21:52:45 | mriedem | b/c twould be nice to drop all this crap | |
| 21:52:50 | melwitt | yeah | |
| 21:54:51 | melwitt | I was thinking of this https://github.com/openstack/nova/blob/master/nova/cmd/novncproxy.py#L40 | |
| 21:55:13 | melwitt | so maybe unrelated since that implies xenapi users can use the regular novnc proxy | |
| 22:01:26 | openstack | Launchpad bug 1492140 in OpenStack Compute (nova) "consoleauth token displayed in log file" [Low,In progress] - Assigned to Tristan Cacqueray (tristan-cacqueray) | |
| 22:01:26 | melwitt | um, this is ancient. and not for exactly the same log you pointed out https://bugs.launchpad.net/nova/+bug/1492140 | |
| 22:05:52 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Explicitly fail if trying to attach SR-IOV port https://review.openstack.org/607729 | |
| 22:05:56 | mriedem | yeah that's for consoleauth | |
| 22:07:55 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Ignore VirtDriverNotReady in _sync_power_states periodic task https://review.openstack.org/607730 | |
| 22:16:43 | mriedem | 481 | |
| 22:16:43 | mriedem | gibi: efried: so i polished off this old dnm devstack test experiment patch to hammer the scheduler to create 1000 instances in a single request which used to give us a ConcurrentUpdate failure during scheduling and creating allocations, and now i can make it fail with consumer generation conflicts http://logs.openstack.org/18/507918/8/check/tempest-full/a9f3849/controller/logs/screen-n-sch.txt.gz?level=TRACE#_Oct_02_23_29_12 | |
| 22:16:55 | mriedem | .concurrent_update", "detail": "There was a conflict when trying to complete your request.\n\n consumer generation conflict - expected null but got 3 ", "title": "Conflict"}]} | |
| 22:16:55 | mriedem | Oct 02 23:29:12.475481 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: ERROR oslo_messaging.rpc.server [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Exception during message handling: AllocationDeleteFailed: Failed to delete allocations for consumer 6962f92b-7dca-4912-aeb2-dcae03c4b52e. Error: {"errors": [{"status": 409, "request_id": "req-13df41fe-cb55-49f1-a998-09b34e48f05b", "code": "place | |
| 22:17:21 | efried | mriedem: Are you using any in-flight patches under that, or just master? | |
| 22:17:24 | mriedem | i think that is newish right? | |