Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-03
22:16:43 mriedem 481
22:16:43 mriedem gibi: efried: so i polished off this old dnm devstack test experiment patch to hammer the scheduler to create 1000 instances in a single request which used to give us a ConcurrentUpdate failure during scheduling and creating allocations, and now i can make it fail with consumer generation conflicts http://logs.openstack.org/18/507918/8/check/tempest-full/a9f3849/controller/logs/screen-n-sch.txt.gz?level=TRACE#_Oct_02_23_29_12
22:16:55 mriedem .concurrent_update", "detail": "There was a conflict when trying to complete your request.\n\n consumer generation conflict - expected null but got 3 ", "title": "Conflict"}]}
22:16:55 mriedem Oct 02 23:29:12.475481 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: ERROR oslo_messaging.rpc.server [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Exception during message handling: AllocationDeleteFailed: Failed to delete allocations for consumer 6962f92b-7dca-4912-aeb2-dcae03c4b52e. Error: {"errors": [{"status": 409, "request_id": "req-13df41fe-cb55-49f1-a998-09b34e48f05b", "code": "place
22:17:21 efried mriedem: Are you using any in-flight patches under that, or just master?
22:17:24 mriedem i think that is newish right?
22:17:27 mriedem master
22:17:57 efried Yes, it's new, since the bottom few patches of gibi's consumer gen patches merged.
22:19:03 efried mriedem: You may want to try running it on top of https://review.openstack.org/#/c/583667/ and see if that fixes it.
22:20:12 efried mriedem: fyi, the ConcurrentUpdate and generation conflict are the same thing, we just switched the error code recently.
22:20:30 efried so it's not really "new", it's just wearing a different dress.
22:20:53 mriedem that change doesn't look like it would help here,
22:21:06 efried orly ynot?
22:21:21 mriedem it doesn't use the latest consumer generation when deleting allocations right?
22:21:34 mriedem we're failing to submit allocations because the host is full
22:21:35 mriedem ing to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider 'b7709a93-f14c-42ed-addf-9736fb721728'. The requested amount would exceed the capacity. ", "title": "Conflict"}]})
22:21:35 mriedem Oct 02 23:29:12.377430 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: WARNING nova.scheduler.client.report [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Unable to submit allocation for instance 63ae7544-7693-4749-886b-024dc93f09f9 (409 {"errors": [{"status": 409, "request_id": "req-0e85117c-871c-46c2-9e01-53c84e811b44", "code": "placement.undefined_code", "detail": "There was a conflict when
22:21:47 mriedem and then the scheduler is trying to cleanup allocations created for previously processed instances in the same request
22:22:03 mriedem and fails to do that cleanup b/c the consumer generation changed
22:22:26 efried hm, yeah, I didn't notice that the first one was a failure on deletion.
22:22:32 efried That window I bitched about us not really closing
22:22:40 efried apparently it's big enough for us to actually hit it.
22:23:06 mriedem note this is also an extreme case,
22:23:10 mriedem i'm creating 1000 instances in a single request
22:23:14 mriedem expecting to melt the scheduler
22:23:16 mriedem and i do
22:25:00 efried I'm trying to figure out where that message is really coming from. "expected null but got 3" <== does this mean we sent null or 3 to the API?
22:25:02 mriedem yeah so between the GET and PUT of the allocatoins, the consumer generation changed and we blew up
22:25:09 melwitt I wonder if a normaler number like 100 would do it? because I have heard of people doing that (oath)
22:25:31 efried right, I'm trying to figure out how that happens. What else is mucking with the allocations?
22:25:33 mriedem https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L2083
22:25:50 mriedem "If between the GET and the PUT the consumer # generation changes then we raise AllocationDeleteFailed."
22:25:58 mriedem the consumer is just the project/user right?
22:26:07 melwitt (my comment was based on the use of the word "extreme")
22:26:16 mriedem i mean, i guess we have the instance_uuid for the consumer here.
22:26:22 efried That's the aforementioned window I bitched about, the result of which was the comment a couple of lines below that.
22:26:28 mriedem is the consumer in placement the unique constraint of the uuid/project_id/user_id?
22:26:29 efried There's a real consumer object now
22:26:41 mriedem so the other thing here,
22:27:01 efried the consumer object (a db table row) has a generation we have to update atomically or die.
22:27:04 sean-k-mooney melwitt: i have spawned 350 instance in one request before on newton and it worked fine
22:27:11 mriedem is that because we have a retry decorator on the select_destinations rpc call, if we get MessagingTimeout from the scheduler b/c it takes too long to schedule 1000 instances in a single request, it re-sends the request to the scheduler with the same list of instances
22:27:36 melwitt sean-k-mooney: ack, that's a data point
22:27:43 mriedem so at this point we would have 2 workers trying to create allocations for the same set of instances (consumers) against the same set of providers
22:27:49 mriedem which will stomp all over themselves
22:27:51 efried oh, okay. That'd do it.
22:28:08 mriedem what i'm wondering is if delete_allocation_for_instance should detect the consumer generation conflict and retry
22:28:24 mriedem like we do on claim_resources https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1744
22:31:11 efried mriedem: We decided it should not
22:31:17 efried there was a whole long ML thread about it.
22:31:41 efried Because, we said, there really shouldn't be more than one thing acting on instance allocations at once, we said.
22:32:02 efried IMO the bug is that ^
22:33:26 mriedem yeah i'm writing up the bug now
22:33:40 efried mriedem: Do you need the ML thread?
22:33:47 mriedem no
22:33:52 mriedem https://bugs.launchpad.net/nova/+bug/1795992
22:33:53 openstack Launchpad bug 1795992 in OpenStack Compute (nova) "retry_select_destinations decorator can make a mess with allocations in placement in a large multi-create request" [Medium,Triaged]
22:34:24 efried Heh. "make a mess".
22:34:39 efried ...fond because they're *memories*.
22:34:40 mriedem total blowout
22:34:49 efried One time in IKEA
22:34:52 mriedem coincidentally, lbragstad is dealing with that right now
22:35:03 efried Oh, did he pop? Good deal.
22:35:11 mriedem black split pea soup coming out of everything
22:35:23 mriedem let me mind meld with him quick
22:35:40 melwitt congrats lbragstad
22:36:09 sean-k-mooney oh before i forget i popped back to say i just found out that kernel 4.16 added a new netdevsim driver that supports among other coolthings sriov. would people be ok with me creating an experimental gate job to test sriov using fedora28?
22:36:37 efried mriedem: Actually, that patch I mentioned before might possibly make the failure happen earlier in the sequence...
22:36:59 efried because surely the allocation is being overwritten
22:37:13 efried though it might be subject to the same window-teeninenss
23:07:46 openstackgerrit Matt Riedemann proposed openstack/nova master: Use long_rpc_timeout in select_destinations RPC call https://review.openstack.org/607735
23:07:52 mriedem dansmith: ^
23:12:50 mriedem melwitt: ocata backport here should be ready to go https://review.openstack.org/#/c/605842/
23:13:08 melwitt ok
23:24:05 mriedem lyarwood: if you want to get these live migration ipv6 changes into the final ocata release before we put it into EM mode you'll need to get the pike and ocata backports fixed up https://review.openstack.org/#/q/I1201db996ea6ceaebd49479b298d74585a78b006
23:38:24 melwitt TIL unified object string fields are six.text_type i.e. unicode
23:46:51 melwitt do we have any things where we compare strings agnostic to bytes vs unicode in unit tests?
23:47:21 melwitt this test is asserting the api response as a dict
23:47:47 melwitt and if consoleauth served the request, it's bytes strings and if the unified object served the request, it's unicode strings
#openstack-nova - 2018-10-04
00:42:48 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992
00:43:19 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993
00:45:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153
00:46:19 openstackgerrit Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955
00:51:04 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104
00:51:15 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106
00:51:28 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110
03:09:52 openstackgerrit Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955
03:11:21 openstackgerrit Merged openstack/nova stable/ocata: Fix the help for the disk_weight_multiplier option https://review.openstack.org/607537
03:11:27 openstackgerrit Merged openstack/nova stable/ocata: Fix host validity check for live-migration https://review.openstack.org/590611
03:11:33 openstackgerrit Merged openstack/nova stable/queens: stable-only: fix typo in IVS related privsep method https://review.openstack.org/604817
03:14:36 openstackgerrit Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting migrations https://review.openstack.org/607798
03:22:16 openstackgerrit Takashi NATSUME proposed openstack/python-novaclient master: Follow up "Fix up userdata argument to rebuild" https://review.openstack.org/607800
03:27:01 openstackgerrit Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting instance action records https://review.openstack.org/607801
03:30:44 openstackgerrit Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting migrations https://review.openstack.org/607798
04:36:34 pooja_jadhav Hi team, anyone knows about the api_sample_tests in detail.. I have some doubts in it.
05:16:46 gmann pooja_jadhav: hi, what is doubt
05:39:57 openstackgerrit Jake Yip proposed openstack/nova master: Add --before to nova-manage db archive_deleted_rows https://review.openstack.org/556751
05:56:01 openstackgerrit Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting migrations https://review.openstack.org/607798
06:12:03 openstackgerrit Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting instance action records https://review.openstack.org/607801
06:23:34 openstackgerrit OpenStack Proposal Bot proposed openstack/nova stable/rocky: Imported Translations from Zanata https://review.openstack.org/604260

Earlier   Later