| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 22:21:35 | mriedem | ing to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider 'b7709a93-f14c-42ed-addf-9736fb721728'. The requested amount would exceed the capacity. ", "title": "Conflict"}]}) | |
| 22:21:35 | mriedem | Oct 02 23:29:12.377430 ubuntu-xenial-limestone-regionone-0002536892 nova-scheduler[22653]: WARNING nova.scheduler.client.report [None req-f4fe43ea-d117-4b7d-a3a4-23dcb59f3058 admin admin] Unable to submit allocation for instance 63ae7544-7693-4749-886b-024dc93f09f9 (409 {"errors": [{"status": 409, "request_id": "req-0e85117c-871c-46c2-9e01-53c84e811b44", "code": "placement.undefined_code", "detail": "There was a conflict when | |
| 22:21:47 | mriedem | and then the scheduler is trying to cleanup allocations created for previously processed instances in the same request | |
| 22:22:03 | mriedem | and fails to do that cleanup b/c the consumer generation changed | |
| 22:22:26 | efried | hm, yeah, I didn't notice that the first one was a failure on deletion. | |
| 22:22:32 | efried | That window I bitched about us not really closing | |
| 22:22:40 | efried | apparently it's big enough for us to actually hit it. | |
| 22:23:06 | mriedem | note this is also an extreme case, | |
| 22:23:10 | mriedem | i'm creating 1000 instances in a single request | |
| 22:23:14 | mriedem | expecting to melt the scheduler | |
| 22:23:16 | mriedem | and i do | |
| 22:25:00 | efried | I'm trying to figure out where that message is really coming from. "expected null but got 3" <== does this mean we sent null or 3 to the API? | |
| 22:25:02 | mriedem | yeah so between the GET and PUT of the allocatoins, the consumer generation changed and we blew up | |
| 22:25:09 | melwitt | I wonder if a normaler number like 100 would do it? because I have heard of people doing that (oath) | |
| 22:25:31 | efried | right, I'm trying to figure out how that happens. What else is mucking with the allocations? | |
| 22:25:33 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L2083 | |
| 22:25:50 | mriedem | "If between the GET and the PUT the consumer # generation changes then we raise AllocationDeleteFailed." | |
| 22:25:58 | mriedem | the consumer is just the project/user right? | |
| 22:26:07 | melwitt | (my comment was based on the use of the word "extreme") | |
| 22:26:16 | mriedem | i mean, i guess we have the instance_uuid for the consumer here. | |
| 22:26:22 | efried | That's the aforementioned window I bitched about, the result of which was the comment a couple of lines below that. | |
| 22:26:28 | mriedem | is the consumer in placement the unique constraint of the uuid/project_id/user_id? | |
| 22:26:29 | efried | There's a real consumer object now | |
| 22:26:41 | mriedem | so the other thing here, | |
| 22:27:01 | efried | the consumer object (a db table row) has a generation we have to update atomically or die. | |
| 22:27:04 | sean-k-mooney | melwitt: i have spawned 350 instance in one request before on newton and it worked fine | |
| 22:27:11 | mriedem | is that because we have a retry decorator on the select_destinations rpc call, if we get MessagingTimeout from the scheduler b/c it takes too long to schedule 1000 instances in a single request, it re-sends the request to the scheduler with the same list of instances | |
| 22:27:36 | melwitt | sean-k-mooney: ack, that's a data point | |
| 22:27:43 | mriedem | so at this point we would have 2 workers trying to create allocations for the same set of instances (consumers) against the same set of providers | |
| 22:27:49 | mriedem | which will stomp all over themselves | |
| 22:27:51 | efried | oh, okay. That'd do it. | |
| 22:28:08 | mriedem | what i'm wondering is if delete_allocation_for_instance should detect the consumer generation conflict and retry | |
| 22:28:24 | mriedem | like we do on claim_resources https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1744 | |
| 22:31:11 | efried | mriedem: We decided it should not | |
| 22:31:17 | efried | there was a whole long ML thread about it. | |
| 22:31:41 | efried | Because, we said, there really shouldn't be more than one thing acting on instance allocations at once, we said. | |
| 22:32:02 | efried | IMO the bug is that ^ | |
| 22:33:26 | mriedem | yeah i'm writing up the bug now | |
| 22:33:40 | efried | mriedem: Do you need the ML thread? | |
| 22:33:47 | mriedem | no | |
| 22:33:52 | mriedem | https://bugs.launchpad.net/nova/+bug/1795992 | |
| 22:33:53 | openstack | Launchpad bug 1795992 in OpenStack Compute (nova) "retry_select_destinations decorator can make a mess with allocations in placement in a large multi-create request" [Medium,Triaged] | |
| 22:34:24 | efried | Heh. "make a mess". | |
| 22:34:39 | efried | ...fond because they're *memories*. | |
| 22:34:40 | mriedem | total blowout | |
| 22:34:49 | efried | One time in IKEA | |
| 22:34:52 | mriedem | coincidentally, lbragstad is dealing with that right now | |
| 22:35:03 | efried | Oh, did he pop? Good deal. | |
| 22:35:11 | mriedem | black split pea soup coming out of everything | |
| 22:35:23 | mriedem | let me mind meld with him quick | |
| 22:35:40 | melwitt | congrats lbragstad | |
| 22:36:09 | sean-k-mooney | oh before i forget i popped back to say i just found out that kernel 4.16 added a new netdevsim driver that supports among other coolthings sriov. would people be ok with me creating an experimental gate job to test sriov using fedora28? | |
| 22:36:37 | efried | mriedem: Actually, that patch I mentioned before might possibly make the failure happen earlier in the sequence... | |
| 22:36:59 | efried | because surely the allocation is being overwritten | |
| 22:37:13 | efried | though it might be subject to the same window-teeninenss | |
| 23:07:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use long_rpc_timeout in select_destinations RPC call https://review.openstack.org/607735 | |
| 23:07:52 | mriedem | dansmith: ^ | |
| 23:12:50 | mriedem | melwitt: ocata backport here should be ready to go https://review.openstack.org/#/c/605842/ | |
| 23:13:08 | melwitt | ok | |
| 23:24:05 | mriedem | lyarwood: if you want to get these live migration ipv6 changes into the final ocata release before we put it into EM mode you'll need to get the pike and ocata backports fixed up https://review.openstack.org/#/q/I1201db996ea6ceaebd49479b298d74585a78b006 | |
| 23:38:24 | melwitt | TIL unified object string fields are six.text_type i.e. unicode | |
| 23:46:51 | melwitt | do we have any things where we compare strings agnostic to bytes vs unicode in unit tests? | |
| 23:47:21 | melwitt | this test is asserting the api response as a dict | |
| 23:47:47 | melwitt | and if consoleauth served the request, it's bytes strings and if the unified object served the request, it's unicode strings | |
| #openstack-nova - 2018-10-04 | |||
| 00:42:48 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992 | |
| 00:43:19 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993 | |
| 00:45:10 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153 | |
| 00:46:19 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955 | |
| 00:51:04 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104 | |
| 00:51:15 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106 | |
| 00:51:28 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110 | |
| 03:09:52 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Nova Cyborg interaction specification. https://review.openstack.org/603955 | |
| 03:11:21 | openstackgerrit | Merged openstack/nova stable/ocata: Fix the help for the disk_weight_multiplier option https://review.openstack.org/607537 | |
| 03:11:27 | openstackgerrit | Merged openstack/nova stable/ocata: Fix host validity check for live-migration https://review.openstack.org/590611 | |
| 03:11:33 | openstackgerrit | Merged openstack/nova stable/queens: stable-only: fix typo in IVS related privsep method https://review.openstack.org/604817 | |
| 03:14:36 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting migrations https://review.openstack.org/607798 | |
| 03:22:16 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Follow up "Fix up userdata argument to rebuild" https://review.openstack.org/607800 | |
| 03:27:01 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting instance action records https://review.openstack.org/607801 | |
| 03:30:44 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting migrations https://review.openstack.org/607798 | |
| 04:36:34 | pooja_jadhav | Hi team, anyone knows about the api_sample_tests in detail.. I have some doubts in it. | |
| 05:16:46 | gmann | pooja_jadhav: hi, what is doubt | |
| 05:39:57 | openstackgerrit | Jake Yip proposed openstack/nova master: Add --before to nova-manage db archive_deleted_rows https://review.openstack.org/556751 | |
| 05:56:01 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting migrations https://review.openstack.org/607798 | |
| 06:12:03 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add restrictions on ``updated_at`` when getting instance action records https://review.openstack.org/607801 | |
| 06:23:34 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova stable/rocky: Imported Translations from Zanata https://review.openstack.org/604260 | |
| 06:24:54 | pooja_jadhav | gmann: hello, for the simple tenant usage api, if the instance is BFV then local_gb should not consider DISK_GB. that fix I have done. Now I am trying to add functional test for the same. and I am facing issue while creating BFV instance. | |
| 06:26:54 | pooja_jadhav | gmann: In request template i am passing block_device_mappingv2 object. and its trying to cinder from nova and getting Service catelog empty error. Also checked the existing code. they have used stub.out abd CinderFixture and Its not working in my case. | |
| 06:29:51 | pooja_jadhav | *trying to call cinder from nova | |
| 06:30:23 | brinzhang | When create network (https://developer.openstack.org/api-ref/compute/#create-network), What does mean of "injected" and "multi_host"? | |
| 06:31:35 | pooja_jadhav | it should not consider flavor.DISK_GB | |
| 06:38:36 | gmann | pooja_jadhav: api_sample_tests does not call to actual service, they are being mocked using fixture. | |
| 06:39:01 | gmann | pooja_jadhav: so you need to use cinder Fixture and change fixture if needed - https://github.com/openstack/nova/blob/85b36cd2f82ccd740057c1bee08fc722209604ab/nova/tests/fixtures.py#L1380 | |
| 06:42:06 | gmann | pooja_jadhav: same network test and other cinder test does. | |
| 06:43:23 | gmann | pooja_jadhav: you can write the tempest integration test where all actual services and their behaviour is being tested. but for sample tests you can simply stub the expected things from cinder and check nova expected response/behavior | |
| 06:50:13 | pooja_jadhav | gmann: yes, same point i am able to see that, I need to use Cinder Fixture for the same. Tried same thing which is done at [1]https://github.com/openstack/nova/blob/master/nova/tests/functional/api_sample_tests/test_block_device_mapping_boot.py#L23-L27 | |
| 06:51:31 | gmann | pooja_jadhav: yeah, any error you are getting? | |
| 06:51:35 | gmann | you have patch up | |
| 06:53:48 | pooja_jadhav | gmann: I have added same stubs in the https://github.com/openstack/nova/blob/master/nova/tests/functional/api_sample_tests/test_simple_tenant_usage.py and trying to add 4th as BFV instance. | |
| 06:55:34 | pooja_jadhav | gmann: here is the error log http://paste.openstack.org/show/731449/ | |
| 07:00:14 | gmann | pooja_jadhav: this is going on cinder call? did you use Cinder Fixture ? | |