| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-20 | |||
| 16:46:56 | mriedem | i have a todo to update one of our CI jobs to do that (run with 2 schedulers) | |
| 16:47:03 | mriedem | mdbooth: jaypipes: ^ updated | |
| 16:47:07 | mnaser | mriedem: yeah before we have a little hacky trick of letting things reschedule more often | |
| 16:47:17 | mnaser | because in large volume schedulers all take the same decisions (before at least) | |
| 16:47:37 | mriedem | mnaser: yup, exactly why we're doing claims in the scheduler rather than rely on the compute to fail and reschedule | |
| 16:47:57 | mriedem | mnaser: so in pike, you should be good to remove the core/ram/disk filters, but you'll still need to rely on numa claims in the compute | |
| 16:48:52 | openstackgerrit | Ilya Shakhat proposed openstack/nova master: Initialize osprofiler in uWSGI application https://review.openstack.org/519664 | |
| 16:49:45 | mnaser | mriedem: ok i just found a really weird amount of compute nodes being marked as dead in logs | |
| 16:50:10 | mnaser | and i guess they're flapping for some reason and the state of self.host_state_map is always changing with the generator | |
| 16:50:26 | mriedem | mnaser: my guess would be the auto-disable thing if you don't have that disabled globally? | |
| 16:50:30 | mriedem | it's a per-compute config | |
| 16:50:32 | mnaser | or ntp | |
| 16:50:41 | gibi_away | I'm starting my vacation now. I will be back officially on 8th of Jan | |
| 16:50:43 | mriedem | yeah i guess service timeouts | |
| 16:50:48 | mriedem | gibi_away: NOOOO | |
| 16:51:02 | mnaser | gibi_away: have fun :-) | |
| 16:51:07 | gibi_away | mriedem: I'm trying to look at gerrit time to time but no promises | |
| 16:51:07 | mriedem | gibi_away: enjoy, see you in a few weeks | |
| 16:51:12 | mriedem | don't | |
| 16:51:18 | mnaser | mriedem: so logs from august till now in scheduler ... grep 'Removing dead compute node' /var/log/nova/nova-scheduler.log | wc -l => 102490. | |
| 16:51:23 | mnaser | so um | |
| 16:51:27 | gibi_away | mriedem: see you | |
| 16:51:43 | mnaser | something is wrong there, i guess with all those services going dead, it was creating the race condition | |
| 16:51:47 | mnaser | now why they are dead thats an exercise for me | |
| 16:52:16 | openstackgerrit | rahul bardia proposed openstack/python-novaclient master: nova limits ERROR (Exception): Field names must be unique https://review.openstack.org/527922 | |
| 16:52:21 | mnaser | some being removed 4 minutes ago.. i think this is ntp related | |
| 16:52:52 | mnaser | the conductor is the one that updates the service record.. or is it the compute directly? | |
| 16:53:04 | mnaser | (as in, who decides what time goes in updated_at) | |
| 16:54:37 | mriedem | that's the servicegroup api i think | |
| 16:55:05 | mriedem | https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L56 | |
| 16:55:39 | mnaser | so one of the compute nodes that went down 4 minutes ago is 4 seconds ahead | |
| 16:55:57 | mriedem | are you seeing any of the logging messages in here? https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L86 | |
| 16:57:26 | mnaser | mriedem: no | |
| 16:57:39 | melwitt | mnaser: I agree that it sounds ntp related if you're seeing some computes ahead in time of others | |
| 16:58:09 | mnaser | melwitt: this one that went 'down' was actually 4 seconds ahead though so if anything it would have updated with a time that is more ahead | |
| 16:58:21 | mriedem | oopsy doodle http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-n-super-cond.txt.gz#_Dec_20_15_15_20_744636 | |
| 16:58:27 | mriedem | cdent: ^ | |
| 17:00:40 | mriedem | http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-placement-api.txt.gz#_Dec_20_15_15_20_666337 | |
| 17:01:17 | mriedem | File "/opt/stack/new/nova/nova/objects/resource_provider.py", line 2029, in _set_allocations | |
| 17:01:22 | mriedem | nova.exception.ObjectActionError: Object action create failed because: already created | |
| 17:03:26 | mriedem | https://bugs.launchpad.net/nova/+bug/1739453 | |
| 17:03:27 | openstack | Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed] | |
| 17:06:14 | mriedem | not sure if using POST /allocations would fix this or not | |
| 17:06:38 | melwitt | mnaser, mriedem: it's the Service object that joins the servicegroup, so I think updated_at has to be the time of the host the service is running on, in this case the compute host | |
| 17:07:34 | openstackgerrit | Merged openstack/nova master: [placement] Separate API schemas (allocation_candidate) https://review.openstack.org/528635 | |
| 17:10:42 | cdent | mriedem: that's rather novel | |
| 17:14:24 | mriedem | yeah 2 hits in 7 days | |
| 17:16:27 | cdent | the code is claiming that that error should only happen when the incoming allocation objects already have an 'id' field | |
| 17:18:37 | cdent | which will take some digging to figure out how is even possible | |
| 17:19:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move aggregates from report client to ProviderTree https://review.openstack.org/521685 | |
| 17:19:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Raise on API errors getting aggregates/traits https://review.openstack.org/526540 | |
| 17:19:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Track associated sharing RPs in report client https://review.openstack.org/526539 | |
| 17:19:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Track provider traits in report client https://review.openstack.org/521686 | |
| 17:19:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Track tree-associated providers in report client https://review.openstack.org/526541 | |
| 17:19:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756 | |
| 17:20:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add nested resources to server moving tests https://review.openstack.org/527728 | |
| 17:20:54 | stephenfin | jaypipes: Think there might be misunderstanding going on here https://review.openstack.org/#/c/527472/ | |
| 17:21:19 | stephenfin | Not saying I don't do overly clever things. Just that this isn't one of them ;) | |
| 17:21:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098 | |
| 17:21:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187 | |
| 17:22:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246 | |
| 17:26:46 | cdent | mriedem: I think I figured out what's up, I left a comment on the bug, assigned myself, and will try a fix | |
| 17:27:43 | mriedem | cdent: ah yeah didn't see the retry decorator | |
| 17:27:44 | mriedem | nice catch | |
| 17:28:10 | cdent | I can't decide what to shake it at, my my tiny fist is shaking | |
| 17:28:16 | cdent | s/my my/but my/ | |
| 17:28:56 | mriedem | yeah not sure how to fix that | |
| 17:29:30 | mdbooth | Sorry in advance, had a merge conflict half way up :/ | |
| 17:29:58 | openstackgerrit | Matthew Booth proposed openstack/nova master: Make BlockDeviceMapping object support uuid https://review.openstack.org/242603 | |
| 17:29:58 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add uuid column to BlockDeviceMapping https://review.openstack.org/242602 | |
| 17:29:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: DriverBlockDevice: make subclasses inherit _proxy_as_attr https://review.openstack.org/524167 | |
| 17:29:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add an online migration for BDM.uuid https://review.openstack.org/525599 | |
| 17:30:00 | openstackgerrit | Matthew Booth proposed openstack/nova master: Give volume DriverBlockDevice classes a common prefix https://review.openstack.org/526346 | |
| 17:30:00 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose BDM uuid to drivers https://review.openstack.org/529037 | |
| 17:30:01 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | |
| 17:30:01 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347 | |
| 17:30:02 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | |
| 17:30:02 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | |
| 17:30:03 | openstackgerrit | Matthew Booth proposed openstack/nova master: Use real block_device_info data in libvirt tests https://review.openstack.org/527916 | |
| 17:30:03 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | |
| 17:30:04 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass disk_info dict to libvirt_info https://review.openstack.org/529329 | |
| 17:30:04 | openstackgerrit | Matthew Booth proposed openstack/nova master: Fix libvirt volume tests passing invalid disk_info https://review.openstack.org/529328 | |
| 17:30:05 | openstackgerrit | Matthew Booth proposed openstack/nova master: Local disk serial numbers for the libvirt driver https://review.openstack.org/529380 | |
| 17:36:46 | cdent | jaypipes: can you recall why/if it is important that AllocationList.create_all() is side effecty on the objects that it passes in? https://github.com/openstack/nova/blob/master/nova/tests/unit/objects/test_resource_provider.py#L355-L357 and https://github.com/openstack/nova/blob/609ddc2244bf9ba84d66cced52e7d4e2987ec669/nova/objects/resource_provider.py#L2061-L2086 and https://bugs.launchpad.net/nova/+bug/1739453 | |
| 17:36:47 | openstack | Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed] - Assigned to Chris Dent (cdent) | |
| 17:41:15 | melwitt | mriedem: FYI the consoles series is ready to go, complete with devstack patch sandwich around it for the tempest vnc tests. stephenfin has gone through it already https://review.openstack.org/#/q/topic:bp/convert-consoles-to-objects+(status:open+OR+status:merged) | |
| 17:41:57 | mriedem | devstack sandwich worked? | |
| 17:42:06 | melwitt | yes. I owe you a cream soda | |
| 17:42:11 | mriedem | awesome | |
| 17:43:17 | edleafe | cdent: if the allocation had an id, then it already exists, no? | |
| 17:44:07 | cdent | no, it's being assigned the .id, but then db deadlock happens and the transaction is reverted and started again, but the list of allocs now contains that alloc with an id | |
| 17:44:25 | mdbooth | artom: Do I recall you being involved in device tagging? | |
| 17:44:56 | edleafe | cdent: ah, so it should change the object, but it's missing the object revert? | |
| 17:46:17 | cdent | edleafe: that's one way to look at it. The other way is that we should be side effecty on on method parameters that are list or dict when we have an option to re-enter. So I'm trying to figure out how to change that in the best way | |
| 17:46:29 | cdent | dammit, should _not_ be side effecty | |
| 17:47:15 | mriedem | cdent: right if you copy the allocs list, then retry, you'll likely then hit a duplicate entry error | |
| 17:47:23 | edleafe | ok, I see what you're saying. It's modifying a mutable param | |
| 17:48:32 | cdent | i need to poke at what the db retry handler is actually doing to be sure | |
| 17:50:32 | mriedem | maybe we need to move the insert portion to another method and that method has the retry on deadlock handler | |
| 17:50:47 | mriedem | apart from the _set_allocations code that's checking the object id | |