| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-20 | |||
| 16:50:48 | mriedem | gibi_away: NOOOO | |
| 16:51:02 | mnaser | gibi_away: have fun :-) | |
| 16:51:07 | mriedem | gibi_away: enjoy, see you in a few weeks | |
| 16:51:07 | gibi_away | mriedem: I'm trying to look at gerrit time to time but no promises | |
| 16:51:12 | mriedem | don't | |
| 16:51:18 | mnaser | mriedem: so logs from august till now in scheduler ... grep 'Removing dead compute node' /var/log/nova/nova-scheduler.log | wc -l => 102490. | |
| 16:51:23 | mnaser | so um | |
| 16:51:27 | gibi_away | mriedem: see you | |
| 16:51:43 | mnaser | something is wrong there, i guess with all those services going dead, it was creating the race condition | |
| 16:51:47 | mnaser | now why they are dead thats an exercise for me | |
| 16:52:16 | openstackgerrit | rahul bardia proposed openstack/python-novaclient master: nova limits ERROR (Exception): Field names must be unique https://review.openstack.org/527922 | |
| 16:52:21 | mnaser | some being removed 4 minutes ago.. i think this is ntp related | |
| 16:52:52 | mnaser | the conductor is the one that updates the service record.. or is it the compute directly? | |
| 16:53:04 | mnaser | (as in, who decides what time goes in updated_at) | |
| 16:54:37 | mriedem | that's the servicegroup api i think | |
| 16:55:05 | mriedem | https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L56 | |
| 16:55:39 | mnaser | so one of the compute nodes that went down 4 minutes ago is 4 seconds ahead | |
| 16:55:57 | mriedem | are you seeing any of the logging messages in here? https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L86 | |
| 16:57:26 | mnaser | mriedem: no | |
| 16:57:39 | melwitt | mnaser: I agree that it sounds ntp related if you're seeing some computes ahead in time of others | |
| 16:58:09 | mnaser | melwitt: this one that went 'down' was actually 4 seconds ahead though so if anything it would have updated with a time that is more ahead | |
| 16:58:21 | mriedem | oopsy doodle http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-n-super-cond.txt.gz#_Dec_20_15_15_20_744636 | |
| 16:58:27 | mriedem | cdent: ^ | |
| 17:00:40 | mriedem | http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-placement-api.txt.gz#_Dec_20_15_15_20_666337 | |
| 17:01:17 | mriedem | File "/opt/stack/new/nova/nova/objects/resource_provider.py", line 2029, in _set_allocations | |
| 17:01:22 | mriedem | nova.exception.ObjectActionError: Object action create failed because: already created | |
| 17:03:26 | mriedem | https://bugs.launchpad.net/nova/+bug/1739453 | |
| 17:03:27 | openstack | Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed] | |
| 17:06:14 | mriedem | not sure if using POST /allocations would fix this or not | |
| 17:06:38 | melwitt | mnaser, mriedem: it's the Service object that joins the servicegroup, so I think updated_at has to be the time of the host the service is running on, in this case the compute host | |
| 17:07:34 | openstackgerrit | Merged openstack/nova master: [placement] Separate API schemas (allocation_candidate) https://review.openstack.org/528635 | |
| 17:10:42 | cdent | mriedem: that's rather novel | |
| 17:14:24 | mriedem | yeah 2 hits in 7 days | |
| 17:16:27 | cdent | the code is claiming that that error should only happen when the incoming allocation objects already have an 'id' field | |
| 17:18:37 | cdent | which will take some digging to figure out how is even possible | |
| 17:19:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move aggregates from report client to ProviderTree https://review.openstack.org/521685 | |
| 17:19:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Track provider traits in report client https://review.openstack.org/521686 | |
| 17:19:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Track associated sharing RPs in report client https://review.openstack.org/526539 | |
| 17:19:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Raise on API errors getting aggregates/traits https://review.openstack.org/526540 | |
| 17:19:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756 | |
| 17:19:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Track tree-associated providers in report client https://review.openstack.org/526541 | |
| 17:20:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add nested resources to server moving tests https://review.openstack.org/527728 | |
| 17:20:54 | stephenfin | jaypipes: Think there might be misunderstanding going on here https://review.openstack.org/#/c/527472/ | |
| 17:21:19 | stephenfin | Not saying I don't do overly clever things. Just that this isn't one of them ;) | |
| 17:21:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098 | |
| 17:21:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187 | |
| 17:22:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246 | |
| 17:26:46 | cdent | mriedem: I think I figured out what's up, I left a comment on the bug, assigned myself, and will try a fix | |
| 17:27:43 | mriedem | cdent: ah yeah didn't see the retry decorator | |
| 17:27:44 | mriedem | nice catch | |
| 17:28:10 | cdent | I can't decide what to shake it at, my my tiny fist is shaking | |
| 17:28:16 | cdent | s/my my/but my/ | |
| 17:28:56 | mriedem | yeah not sure how to fix that | |
| 17:29:30 | mdbooth | Sorry in advance, had a merge conflict half way up :/ | |
| 17:29:58 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add uuid column to BlockDeviceMapping https://review.openstack.org/242602 | |
| 17:29:58 | openstackgerrit | Matthew Booth proposed openstack/nova master: Make BlockDeviceMapping object support uuid https://review.openstack.org/242603 | |
| 17:29:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add an online migration for BDM.uuid https://review.openstack.org/525599 | |
| 17:29:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: DriverBlockDevice: make subclasses inherit _proxy_as_attr https://review.openstack.org/524167 | |
| 17:30:00 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose BDM uuid to drivers https://review.openstack.org/529037 | |
| 17:30:00 | openstackgerrit | Matthew Booth proposed openstack/nova master: Give volume DriverBlockDevice classes a common prefix https://review.openstack.org/526346 | |
| 17:30:01 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347 | |
| 17:30:01 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | |
| 17:30:02 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | |
| 17:30:02 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | |
| 17:30:03 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | |
| 17:30:03 | openstackgerrit | Matthew Booth proposed openstack/nova master: Use real block_device_info data in libvirt tests https://review.openstack.org/527916 | |
| 17:30:04 | openstackgerrit | Matthew Booth proposed openstack/nova master: Fix libvirt volume tests passing invalid disk_info https://review.openstack.org/529328 | |
| 17:30:04 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass disk_info dict to libvirt_info https://review.openstack.org/529329 | |
| 17:30:05 | openstackgerrit | Matthew Booth proposed openstack/nova master: Local disk serial numbers for the libvirt driver https://review.openstack.org/529380 | |
| 17:36:46 | cdent | jaypipes: can you recall why/if it is important that AllocationList.create_all() is side effecty on the objects that it passes in? https://github.com/openstack/nova/blob/master/nova/tests/unit/objects/test_resource_provider.py#L355-L357 and https://github.com/openstack/nova/blob/609ddc2244bf9ba84d66cced52e7d4e2987ec669/nova/objects/resource_provider.py#L2061-L2086 and https://bugs.launchpad.net/nova/+bug/1739453 | |
| 17:36:47 | openstack | Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed] - Assigned to Chris Dent (cdent) | |
| 17:41:15 | melwitt | mriedem: FYI the consoles series is ready to go, complete with devstack patch sandwich around it for the tempest vnc tests. stephenfin has gone through it already https://review.openstack.org/#/q/topic:bp/convert-consoles-to-objects+(status:open+OR+status:merged) | |
| 17:41:57 | mriedem | devstack sandwich worked? | |
| 17:42:06 | melwitt | yes. I owe you a cream soda | |
| 17:42:11 | mriedem | awesome | |
| 17:43:17 | edleafe | cdent: if the allocation had an id, then it already exists, no? | |
| 17:44:07 | cdent | no, it's being assigned the .id, but then db deadlock happens and the transaction is reverted and started again, but the list of allocs now contains that alloc with an id | |
| 17:44:25 | mdbooth | artom: Do I recall you being involved in device tagging? | |
| 17:44:56 | edleafe | cdent: ah, so it should change the object, but it's missing the object revert? | |
| 17:46:17 | cdent | edleafe: that's one way to look at it. The other way is that we should be side effecty on on method parameters that are list or dict when we have an option to re-enter. So I'm trying to figure out how to change that in the best way | |
| 17:46:29 | cdent | dammit, should _not_ be side effecty | |
| 17:47:15 | mriedem | cdent: right if you copy the allocs list, then retry, you'll likely then hit a duplicate entry error | |
| 17:47:23 | edleafe | ok, I see what you're saying. It's modifying a mutable param | |
| 17:48:32 | cdent | i need to poke at what the db retry handler is actually doing to be sure | |
| 17:50:32 | mriedem | maybe we need to move the insert portion to another method and that method has the retry on deadlock handler | |
| 17:50:47 | mriedem | apart from the _set_allocations code that's checking the object id | |
| 17:51:03 | mriedem | but, | |
| 17:51:19 | mriedem | we could still then hit duplicate entry errors i'd think | |
| 17:51:26 | mriedem | any time you're retrying to insert over a list... | |
| 17:52:18 | mriedem | so loop over the objects, then insert separately per object in another method with the deadlock retry on that method? would that work? | |
| 17:54:21 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Use instance.project_id when creating request specs for old instances https://review.openstack.org/529384 | |
| 17:54:22 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Workaround missing RequestSpec.project_id when moving an instance https://review.openstack.org/529385 | |
| 17:55:11 | cdent | mriedem: unclear. still trying to get all the pieces to line up. one thing that's breaking my brain is that the retry wrapper is not explicitly saying anything about the current transaction. will keep looking | |
| 17:56:58 | cdent | mriedem: i think you're right though, extracting a method for the individual insert seems right | |
| 17:57:49 | mriedem | seems easiest anyway | |
| 18:03:45 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Use instance.project_id when creating request specs for old instances https://review.openstack.org/529387 | |
| 18:08:21 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/newton: Use instance.project_id when creating request specs for old instances https://review.openstack.org/529389 | |
| 18:08:48 | cdent | mriedem: hmmm, jay's comment from when he added the retry is potentially derailing, he's got a diferent theory of what could be causing the issues: https://github.com/openstack/nova/commit/e001b0538a9d413f864 | |
| 18:10:06 | mriedem | that likely makes more sense, the inventory thing | |
| 18:10:43 | mriedem | but in that case, the deadlock isn't on the insert, it's on the generation update yeah? https://github.com/openstack/nova/commit/e001b0538a9d413f864#diff-a040eef16b56cc7862fb2a334c757deaR1855 | |