Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-03
21:51:56 mriedem and yeah send something to the dev and ops MLs
21:52:06 mriedem oh b/c of python 2.4?
21:52:20 mriedem i might be thinking of something else
21:52:39 mriedem i guess start with the ML
21:52:41 melwitt maybe. when stephenfin worked on the encrypted console stuff, he had to exclude xenapi from the version requirement IIRC
21:52:45 mriedem b/c twould be nice to drop all this crap
21:52:50 melwitt yeah
21:54:51 melwitt I was thinking of this https://github.com/openstack/nova/blob/master/nova/cmd/novncproxy.py#L40
21:55:13 melwitt so maybe unrelated since that implies xenapi users can use the regular novnc proxy
22:01:26 openstack Launchpad bug 1492140 in OpenStack Compute (nova) "consoleauth token displayed in log file" [Low,In progress] - Assigned to Tristan Cacqueray (tristan-cacqueray)
22:01:26 melwitt um, this is ancient. and not for exactly the same log you pointed out https://bugs.launchpad.net/nova/+bug/1492140
22:05:52 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Explicitly fail if trying to attach SR-IOV port https://review.openstack.org/607729
22:05:56 mriedem yeah that's for consoleauth
22:07:55 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Ignore VirtDriverNotReady in _sync_power_states periodic task https://review.openstack.org/607730
22:16:43 mriedem 481
22:16:43 mriedem gibi: efried: so i polished off this old dnm devstack test experiment patch to hammer the scheduler to create 1000 instances in a single request which used to give us a ConcurrentUpdate failure during scheduling and creating allocations, and now i can make it fail with consumer generation conflicts http://logs.openstack.org/18/507918/8/check/tempest-full/a9f3849/controller/logs/screen-n-sch.txt.gz?level=TRACE#_Oct_02_23_29_12
22:16:55 mriedem .concurrent_update", "detail": "There was a conflict when trying to complete your request.\n\n consumer generation conflict - expected null but got 3 ", "title": "Conflict"}]}
22:16:55 mriedem Oct 02 23:29:12.475481 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: ERROR oslo_messaging.rpc.server [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Exception during message handling: AllocationDeleteFailed: Failed to delete allocations for consumer 6962f92b-7dca-4912-aeb2-dcae03c4b52e. Error: {"errors": [{"status": 409, "request_id": "req-13df41fe-cb55-49f1-a998-09b34e48f05b", "code": "place
22:17:21 efried mriedem: Are you using any in-flight patches under that, or just master?
22:17:24 mriedem i think that is newish right?
22:17:27 mriedem master
22:17:57 efried Yes, it's new, since the bottom few patches of gibi's consumer gen patches merged.
22:19:03 efried mriedem: You may want to try running it on top of https://review.openstack.org/#/c/583667/ and see if that fixes it.
22:20:12 efried mriedem: fyi, the ConcurrentUpdate and generation conflict are the same thing, we just switched the error code recently.
22:20:30 efried so it's not really "new", it's just wearing a different dress.
22:20:53 mriedem that change doesn't look like it would help here,
22:21:06 efried orly ynot?
22:21:21 mriedem it doesn't use the latest consumer generation when deleting allocations right?
22:21:34 mriedem we're failing to submit allocations because the host is full
22:21:35 mriedem ing to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider 'b7709a93-f14c-42ed-addf-9736fb721728'. The requested amount would exceed the capacity. ", "title": "Conflict"}]})
22:21:35 mriedem Oct 02 23:29:12.377430 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: WARNING nova.scheduler.client.report [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Unable to submit allocation for instance 63ae7544-7693-4749-886b-024dc93f09f9 (409 {"errors": [{"status": 409, "request_id": "req-0e85117c-871c-46c2-9e01-53c84e811b44", "code": "placement.undefined_code", "detail": "There was a conflict when
22:21:47 mriedem and then the scheduler is trying to cleanup allocations created for previously processed instances in the same request
22:22:03 mriedem and fails to do that cleanup b/c the consumer generation changed
22:22:26 efried hm, yeah, I didn't notice that the first one was a failure on deletion.
22:22:32 efried That window I bitched about us not really closing
22:22:40 efried apparently it's big enough for us to actually hit it.
22:23:06 mriedem note this is also an extreme case,
22:23:10 mriedem i'm creating 1000 instances in a single request
22:23:14 mriedem expecting to melt the scheduler
22:23:16 mriedem and i do
22:25:00 efried I'm trying to figure out where that message is really coming from. "expected null but got 3" <== does this mean we sent null or 3 to the API?
22:25:02 mriedem yeah so between the GET and PUT of the allocatoins, the consumer generation changed and we blew up
22:25:09 melwitt I wonder if a normaler number like 100 would do it? because I have heard of people doing that (oath)
22:25:31 efried right, I'm trying to figure out how that happens. What else is mucking with the allocations?
22:25:33 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L2083
22:25:50 mriedem "If between the GET and the PUT the consumer # generation changes then we raise AllocationDeleteFailed."
22:25:58 mriedem the consumer is just the project/user right?
22:26:07 melwitt (my comment was based on the use of the word "extreme")
22:26:16 mriedem i mean, i guess we have the instance_uuid for the consumer here.
22:26:22 efried That's the aforementioned window I bitched about, the result of which was the comment a couple of lines below that.
22:26:28 mriedem is the consumer in placement the unique constraint of the uuid/project_id/user_id?
22:26:29 efried There's a real consumer object now
22:26:41 mriedem so the other thing here,
22:27:01 efried the consumer object (a db table row) has a generation we have to update atomically or die.
22:27:04 sean-k-mooney melwitt: i have spawned 350 instance in one request before on newton and it worked fine
22:27:11 mriedem is that because we have a retry decorator on the select_destinations rpc call, if we get MessagingTimeout from the scheduler b/c it takes too long to schedule 1000 instances in a single request, it re-sends the request to the scheduler with the same list of instances
22:27:36 melwitt sean-k-mooney: ack, that's a data point
22:27:43 mriedem so at this point we would have 2 workers trying to create allocations for the same set of instances (consumers) against the same set of providers
22:27:49 mriedem which will stomp all over themselves
22:27:51 efried oh, okay. That'd do it.
22:28:08 mriedem what i'm wondering is if delete_allocation_for_instance should detect the consumer generation conflict and retry
22:28:24 mriedem like we do on claim_resources https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1744
22:31:11 efried mriedem: We decided it should not
22:31:17 efried there was a whole long ML thread about it.
22:31:41 efried Because, we said, there really shouldn't be more than one thing acting on instance allocations at once, we said.
22:32:02 efried IMO the bug is that ^
22:33:26 mriedem yeah i'm writing up the bug now
22:33:40 efried mriedem: Do you need the ML thread?
22:33:47 mriedem no
22:33:52 mriedem https://bugs.launchpad.net/nova/+bug/1795992
22:33:53 openstack Launchpad bug 1795992 in OpenStack Compute (nova) "retry_select_destinations decorator can make a mess with allocations in placement in a large multi-create request" [Medium,Triaged]
22:34:24 efried Heh. "make a mess".
22:34:39 efried ...fond because they're *memories*.
22:34:40 mriedem total blowout
22:34:49 efried One time in IKEA
22:34:52 mriedem coincidentally, lbragstad is dealing with that right now
22:35:03 efried Oh, did he pop? Good deal.
22:35:11 mriedem black split pea soup coming out of everything
22:35:23 mriedem let me mind meld with him quick
22:35:40 melwitt congrats lbragstad
22:36:09 sean-k-mooney oh before i forget i popped back to say i just found out that kernel 4.16 added a new netdevsim driver that supports among other coolthings sriov. would people be ok with me creating an experimental gate job to test sriov using fedora28?
22:36:37 efried mriedem: Actually, that patch I mentioned before might possibly make the failure happen earlier in the sequence...
22:36:59 efried because surely the allocation is being overwritten
22:37:13 efried though it might be subject to the same window-teeninenss
23:07:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Use long_rpc_timeout in select_destinations RPC call https://review.openstack.org/607735
23:07:52 mriedem dansmith: ^
23:12:50 mriedem melwitt: ocata backport here should be ready to go https://review.openstack.org/#/c/605842/
23:13:08 melwitt ok
23:24:05 mriedem lyarwood: if you want to get these live migration ipv6 changes into the final ocata release before we put it into EM mode you'll need to get the pike and ocata backports fixed up https://review.openstack.org/#/q/I1201db996ea6ceaebd49479b298d74585a78b006
23:38:24 melwitt TIL unified object string fields are six.text_type i.e. unicode
23:46:51 melwitt do we have any things where we compare strings agnostic to bytes vs unicode in unit tests?
23:47:21 melwitt this test is asserting the api response as a dict
23:47:47 melwitt and if consoleauth served the request, it's bytes strings and if the unified object served the request, it's unicode strings
#openstack-nova - 2018-10-04
00:42:48 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992
00:43:19 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993
00:45:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153
00:46:19 openstackgerrit Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955
00:51:04 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104
00:51:15 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106
00:51:28 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110

Earlier   Later