Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-03
21:03:52 mriedem os-console-auth-tokens was added specifically for rdp consoles for hyperv
21:04:42 melwitt O.o
21:09:15 melwitt yeah, so we could scatter-gather a ConsoleAuthToken.validate(context, token) call and only one will return token object, the others will raise exceptions. that method takes an unhashed token and will hash it before looking for it in the db
21:10:53 mriedem sure
21:11:05 mriedem shitty performance but whatareyougonnado
21:11:13 melwitt yeah, exactly
21:11:13 mriedem plus it's admin-only and no one knew it existed
21:11:21 melwitt haha, right. we have that going for us
21:11:45 mriedem is there a bug for this?
21:11:46 melwitt I'll add it to the pile o poopatches
21:11:51 melwitt no
21:11:57 melwitt I'll open one
21:12:22 mriedem cool. not sure if we should report the token logging thing or just pretend i never said it.
21:12:28 melwitt I was just making the changes to only access consoleauth if [workarounds] and ran into this in the func tests
21:12:46 melwitt so I'm doing really good here
21:14:09 melwitt yeah, I'm not sure either. I expect it wouldn't cause a CVE because it's been this way for years
21:19:41 mriedem heh, well, we've had things "this way for years" that are CVEs,
21:19:52 mriedem but logging credentials and tokens and such isn't considered one of them
21:20:13 mriedem it's a "hardening opportunity"
21:20:18 melwitt haha, ok
21:20:22 openstackgerrit Jay Pipes proposed openstack/nova stable/ocata: Re-use existing ComputeNode on ironic rebalance https://review.openstack.org/607626
21:21:12 openstack Launchpad bug 1795982 in OpenStack Compute (nova) "/os-console-auth-tokens/{console_token} API doesn't handle the database backend" [High,Triaged] - Assigned to melanie witt (melwitt)
21:21:12 melwitt https://bugs.launchpad.net/nova/+bug/1795982
21:30:17 melwitt so, these other console create/delete/get APIs are connected to cell database models, with nothing at the API level to target cells for the consoles
21:31:38 melwitt "nova-console, which is a XenAPI-specific service that most recent VNC proxy architectures do not use."
21:32:34 melwitt it sounds like that should be deprecated. we didn't do anything to handle it in a cells v2 world
21:35:17 melwitt maybe I should send something to the ML to ask about it
21:45:37 mriedem i thought the xvp stuff was xen-only
21:46:21 melwitt yeah, the nova-console service is xen-only. but if someone ran multi-cell with xen, the nova-console part wouldn't work right
21:46:25 mriedem but yeah this is clearly busted in a cells v2 world
21:47:17 melwitt so the question will be, do we cells-v2-ify it or do we deprecate it. tbc, this is for the other APIs, not the consoleauth one I'm fixing
21:47:35 melwitt *the other 4
21:48:50 mriedem yeah i know
21:50:10 mriedem idk, i've asked about killing xvp in the past
21:50:13 mriedem no one seems to know
21:50:21 melwitt ah, ok
21:51:03 mriedem i'd say if there are alternatives available for xenapi users, then we should deprecate it
21:51:21 mriedem so probably a question for naichuans and BobBall
21:51:52 melwitt yeah, that's what I wasn't sure about, because IIUC, xenapi has to use some ancient version of stuff, so they might actually need it because they can't use newer VNC
21:51:56 mriedem and yeah send something to the dev and ops MLs
21:52:06 mriedem oh b/c of python 2.4?
21:52:20 mriedem i might be thinking of something else
21:52:39 mriedem i guess start with the ML
21:52:41 melwitt maybe. when stephenfin worked on the encrypted console stuff, he had to exclude xenapi from the version requirement IIRC
21:52:45 mriedem b/c twould be nice to drop all this crap
21:52:50 melwitt yeah
21:54:51 melwitt I was thinking of this https://github.com/openstack/nova/blob/master/nova/cmd/novncproxy.py#L40
21:55:13 melwitt so maybe unrelated since that implies xenapi users can use the regular novnc proxy
22:01:26 openstack Launchpad bug 1492140 in OpenStack Compute (nova) "consoleauth token displayed in log file" [Low,In progress] - Assigned to Tristan Cacqueray (tristan-cacqueray)
22:01:26 melwitt um, this is ancient. and not for exactly the same log you pointed out https://bugs.launchpad.net/nova/+bug/1492140
22:05:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Explicitly fail if trying to attach SR-IOV port https://review.openstack.org/607729
22:05:56 mriedem yeah that's for consoleauth
22:07:55 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Ignore VirtDriverNotReady in _sync_power_states periodic task https://review.openstack.org/607730
22:16:43 mriedem 481
22:16:43 mriedem gibi: efried: so i polished off this old dnm devstack test experiment patch to hammer the scheduler to create 1000 instances in a single request which used to give us a ConcurrentUpdate failure during scheduling and creating allocations, and now i can make it fail with consumer generation conflicts http://logs.openstack.org/18/507918/8/check/tempest-full/a9f3849/controller/logs/screen-n-sch.txt.gz?level=TRACE#_Oct_02_23_29_12
22:16:55 mriedem .concurrent_update", "detail": "There was a conflict when trying to complete your request.\n\n consumer generation conflict - expected null but got 3 ", "title": "Conflict"}]}
22:16:55 mriedem Oct 02 23:29:12.475481 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: ERROR oslo_messaging.rpc.server [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Exception during message handling: AllocationDeleteFailed: Failed to delete allocations for consumer 6962f92b-7dca-4912-aeb2-dcae03c4b52e. Error: {"errors": [{"status": 409, "request_id": "req-13df41fe-cb55-49f1-a998-09b34e48f05b", "code": "place
22:17:21 efried mriedem: Are you using any in-flight patches under that, or just master?
22:17:24 mriedem i think that is newish right?
22:17:27 mriedem master
22:17:57 efried Yes, it's new, since the bottom few patches of gibi's consumer gen patches merged.
22:19:03 efried mriedem: You may want to try running it on top of https://review.openstack.org/#/c/583667/ and see if that fixes it.
22:20:12 efried mriedem: fyi, the ConcurrentUpdate and generation conflict are the same thing, we just switched the error code recently.
22:20:30 efried so it's not really "new", it's just wearing a different dress.
22:20:53 mriedem that change doesn't look like it would help here,
22:21:06 efried orly ynot?
22:21:21 mriedem it doesn't use the latest consumer generation when deleting allocations right?
22:21:34 mriedem we're failing to submit allocations because the host is full
22:21:35 mriedem ing to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider 'b7709a93-f14c-42ed-addf-9736fb721728'. The requested amount would exceed the capacity. ", "title": "Conflict"}]})
22:21:35 mriedem Oct 02 23:29:12.377430 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: WARNING nova.scheduler.client.report [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Unable to submit allocation for instance 63ae7544-7693-4749-886b-024dc93f09f9 (409 {"errors": [{"status": 409, "request_id": "req-0e85117c-871c-46c2-9e01-53c84e811b44", "code": "placement.undefined_code", "detail": "There was a conflict when
22:21:47 mriedem and then the scheduler is trying to cleanup allocations created for previously processed instances in the same request
22:22:03 mriedem and fails to do that cleanup b/c the consumer generation changed
22:22:26 efried hm, yeah, I didn't notice that the first one was a failure on deletion.
22:22:32 efried That window I bitched about us not really closing
22:22:40 efried apparently it's big enough for us to actually hit it.
22:23:06 mriedem note this is also an extreme case,
22:23:10 mriedem i'm creating 1000 instances in a single request
22:23:14 mriedem expecting to melt the scheduler
22:23:16 mriedem and i do
22:25:00 efried I'm trying to figure out where that message is really coming from. "expected null but got 3" <== does this mean we sent null or 3 to the API?
22:25:02 mriedem yeah so between the GET and PUT of the allocatoins, the consumer generation changed and we blew up
22:25:09 melwitt I wonder if a normaler number like 100 would do it? because I have heard of people doing that (oath)
22:25:31 efried right, I'm trying to figure out how that happens. What else is mucking with the allocations?
22:25:33 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L2083
22:25:50 mriedem "If between the GET and the PUT the consumer # generation changes then we raise AllocationDeleteFailed."
22:25:58 mriedem the consumer is just the project/user right?
22:26:07 melwitt (my comment was based on the use of the word "extreme")
22:26:16 mriedem i mean, i guess we have the instance_uuid for the consumer here.
22:26:22 efried That's the aforementioned window I bitched about, the result of which was the comment a couple of lines below that.
22:26:28 mriedem is the consumer in placement the unique constraint of the uuid/project_id/user_id?
22:26:29 efried There's a real consumer object now
22:26:41 mriedem so the other thing here,
22:27:01 efried the consumer object (a db table row) has a generation we have to update atomically or die.
22:27:04 sean-k-mooney melwitt: i have spawned 350 instance in one request before on newton and it worked fine
22:27:11 mriedem is that because we have a retry decorator on the select_destinations rpc call, if we get MessagingTimeout from the scheduler b/c it takes too long to schedule 1000 instances in a single request, it re-sends the request to the scheduler with the same list of instances
22:27:36 melwitt sean-k-mooney: ack, that's a data point
22:27:43 mriedem so at this point we would have 2 workers trying to create allocations for the same set of instances (consumers) against the same set of providers
22:27:49 mriedem which will stomp all over themselves
22:27:51 efried oh, okay. That'd do it.
22:28:08 mriedem what i'm wondering is if delete_allocation_for_instance should detect the consumer generation conflict and retry

Earlier   Later