Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-03
21:35:17 melwitt maybe I should send something to the ML to ask about it
21:45:37 mriedem i thought the xvp stuff was xen-only
21:46:21 melwitt yeah, the nova-console service is xen-only. but if someone ran multi-cell with xen, the nova-console part wouldn't work right
21:46:25 mriedem but yeah this is clearly busted in a cells v2 world
21:47:17 melwitt so the question will be, do we cells-v2-ify it or do we deprecate it. tbc, this is for the other APIs, not the consoleauth one I'm fixing
21:47:35 melwitt *the other 4
21:48:50 mriedem yeah i know
21:50:10 mriedem idk, i've asked about killing xvp in the past
21:50:13 mriedem no one seems to know
21:50:21 melwitt ah, ok
21:51:03 mriedem i'd say if there are alternatives available for xenapi users, then we should deprecate it
21:51:21 mriedem so probably a question for naichuans and BobBall
21:51:52 melwitt yeah, that's what I wasn't sure about, because IIUC, xenapi has to use some ancient version of stuff, so they might actually need it because they can't use newer VNC
21:51:56 mriedem and yeah send something to the dev and ops MLs
21:52:06 mriedem oh b/c of python 2.4?
21:52:20 mriedem i might be thinking of something else
21:52:39 mriedem i guess start with the ML
21:52:41 melwitt maybe. when stephenfin worked on the encrypted console stuff, he had to exclude xenapi from the version requirement IIRC
21:52:45 mriedem b/c twould be nice to drop all this crap
21:52:50 melwitt yeah
21:54:51 melwitt I was thinking of this https://github.com/openstack/nova/blob/master/nova/cmd/novncproxy.py#L40
21:55:13 melwitt so maybe unrelated since that implies xenapi users can use the regular novnc proxy
22:01:26 openstack Launchpad bug 1492140 in OpenStack Compute (nova) "consoleauth token displayed in log file" [Low,In progress] - Assigned to Tristan Cacqueray (tristan-cacqueray)
22:01:26 melwitt um, this is ancient. and not for exactly the same log you pointed out https://bugs.launchpad.net/nova/+bug/1492140
22:05:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Explicitly fail if trying to attach SR-IOV port https://review.openstack.org/607729
22:05:56 mriedem yeah that's for consoleauth
22:07:55 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Ignore VirtDriverNotReady in _sync_power_states periodic task https://review.openstack.org/607730
22:16:43 mriedem 481
22:16:43 mriedem gibi: efried: so i polished off this old dnm devstack test experiment patch to hammer the scheduler to create 1000 instances in a single request which used to give us a ConcurrentUpdate failure during scheduling and creating allocations, and now i can make it fail with consumer generation conflicts http://logs.openstack.org/18/507918/8/check/tempest-full/a9f3849/controller/logs/screen-n-sch.txt.gz?level=TRACE#_Oct_02_23_29_12
22:16:55 mriedem .concurrent_update", "detail": "There was a conflict when trying to complete your request.\n\n consumer generation conflict - expected null but got 3 ", "title": "Conflict"}]}
22:16:55 mriedem Oct 02 23:29:12.475481 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: ERROR oslo_messaging.rpc.server [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Exception during message handling: AllocationDeleteFailed: Failed to delete allocations for consumer 6962f92b-7dca-4912-aeb2-dcae03c4b52e. Error: {"errors": [{"status": 409, "request_id": "req-13df41fe-cb55-49f1-a998-09b34e48f05b", "code": "place
22:17:21 efried mriedem: Are you using any in-flight patches under that, or just master?
22:17:24 mriedem i think that is newish right?
22:17:27 mriedem master
22:17:57 efried Yes, it's new, since the bottom few patches of gibi's consumer gen patches merged.
22:19:03 efried mriedem: You may want to try running it on top of https://review.openstack.org/#/c/583667/ and see if that fixes it.
22:20:12 efried mriedem: fyi, the ConcurrentUpdate and generation conflict are the same thing, we just switched the error code recently.
22:20:30 efried so it's not really "new", it's just wearing a different dress.
22:20:53 mriedem that change doesn't look like it would help here,
22:21:06 efried orly ynot?
22:21:21 mriedem it doesn't use the latest consumer generation when deleting allocations right?
22:21:34 mriedem we're failing to submit allocations because the host is full
22:21:35 mriedem ing to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider 'b7709a93-f14c-42ed-addf-9736fb721728'. The requested amount would exceed the capacity. ", "title": "Conflict"}]})
22:21:35 mriedem Oct 02 23:29:12.377430 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: WARNING nova.scheduler.client.report [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Unable to submit allocation for instance 63ae7544-7693-4749-886b-024dc93f09f9 (409 {"errors": [{"status": 409, "request_id": "req-0e85117c-871c-46c2-9e01-53c84e811b44", "code": "placement.undefined_code", "detail": "There was a conflict when
22:21:47 mriedem and then the scheduler is trying to cleanup allocations created for previously processed instances in the same request
22:22:03 mriedem and fails to do that cleanup b/c the consumer generation changed
22:22:26 efried hm, yeah, I didn't notice that the first one was a failure on deletion.
22:22:32 efried That window I bitched about us not really closing
22:22:40 efried apparently it's big enough for us to actually hit it.
22:23:06 mriedem note this is also an extreme case,
22:23:10 mriedem i'm creating 1000 instances in a single request
22:23:14 mriedem expecting to melt the scheduler
22:23:16 mriedem and i do
22:25:00 efried I'm trying to figure out where that message is really coming from. "expected null but got 3" <== does this mean we sent null or 3 to the API?
22:25:02 mriedem yeah so between the GET and PUT of the allocatoins, the consumer generation changed and we blew up
22:25:09 melwitt I wonder if a normaler number like 100 would do it? because I have heard of people doing that (oath)
22:25:31 efried right, I'm trying to figure out how that happens. What else is mucking with the allocations?
22:25:33 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L2083
22:25:50 mriedem "If between the GET and the PUT the consumer # generation changes then we raise AllocationDeleteFailed."
22:25:58 mriedem the consumer is just the project/user right?
22:26:07 melwitt (my comment was based on the use of the word "extreme")
22:26:16 mriedem i mean, i guess we have the instance_uuid for the consumer here.
22:26:22 efried That's the aforementioned window I bitched about, the result of which was the comment a couple of lines below that.
22:26:28 mriedem is the consumer in placement the unique constraint of the uuid/project_id/user_id?
22:26:29 efried There's a real consumer object now
22:26:41 mriedem so the other thing here,
22:27:01 efried the consumer object (a db table row) has a generation we have to update atomically or die.
22:27:04 sean-k-mooney melwitt: i have spawned 350 instance in one request before on newton and it worked fine
22:27:11 mriedem is that because we have a retry decorator on the select_destinations rpc call, if we get MessagingTimeout from the scheduler b/c it takes too long to schedule 1000 instances in a single request, it re-sends the request to the scheduler with the same list of instances
22:27:36 melwitt sean-k-mooney: ack, that's a data point
22:27:43 mriedem so at this point we would have 2 workers trying to create allocations for the same set of instances (consumers) against the same set of providers
22:27:49 mriedem which will stomp all over themselves
22:27:51 efried oh, okay. That'd do it.
22:28:08 mriedem what i'm wondering is if delete_allocation_for_instance should detect the consumer generation conflict and retry
22:28:24 mriedem like we do on claim_resources https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1744
22:31:11 efried mriedem: We decided it should not
22:31:17 efried there was a whole long ML thread about it.
22:31:41 efried Because, we said, there really shouldn't be more than one thing acting on instance allocations at once, we said.
22:32:02 efried IMO the bug is that ^
22:33:26 mriedem yeah i'm writing up the bug now
22:33:40 efried mriedem: Do you need the ML thread?
22:33:47 mriedem no
22:33:52 mriedem https://bugs.launchpad.net/nova/+bug/1795992
22:33:53 openstack Launchpad bug 1795992 in OpenStack Compute (nova) "retry_select_destinations decorator can make a mess with allocations in placement in a large multi-create request" [Medium,Triaged]
22:34:24 efried Heh. "make a mess".
22:34:39 efried ...fond because they're *memories*.
22:34:40 mriedem total blowout
22:34:49 efried One time in IKEA
22:34:52 mriedem coincidentally, lbragstad is dealing with that right now
22:35:03 efried Oh, did he pop? Good deal.
22:35:11 mriedem black split pea soup coming out of everything
22:35:23 mriedem let me mind meld with him quick
22:35:40 melwitt congrats lbragstad
22:36:09 sean-k-mooney oh before i forget i popped back to say i just found out that kernel 4.16 added a new netdevsim driver that supports among other coolthings sriov. would people be ok with me creating an experimental gate job to test sriov using fedora28?
22:36:37 efried mriedem: Actually, that patch I mentioned before might possibly make the failure happen earlier in the sequence...
22:36:59 efried because surely the allocation is being overwritten
22:37:13 efried though it might be subject to the same window-teeninenss
23:07:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Use long_rpc_timeout in select_destinations RPC call https://review.openstack.org/607735
23:07:52 mriedem dansmith: ^
23:12:50 mriedem melwitt: ocata backport here should be ready to go https://review.openstack.org/#/c/605842/

Earlier   Later