| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-20 | |||
| 16:31:10 | mriedem | we do start auto-disabling computes that fail to build 10 times in a row | |
| 16:31:20 | mriedem | but in the scheduler code here, we include disabled services | |
| 16:31:54 | mnaser | mriedem: i think we disabled that because scheduling bugs would shut down our entire cloud lol | |
| 16:32:12 | mnaser | ex: failing to place numa on server will result in a build fail count being increased | |
| 16:38:57 | mriedem | mdbooth: ok i think we can still use part of your test and maintain the generator | |
| 16:39:02 | mriedem | i'm going to update your patch quick | |
| 16:39:18 | mdbooth | mriedem: on a call, but that's cool | |
| 16:42:46 | cdent | jaypipes, mriedem: if you're both still around, this is a good and relatively straightforward win for placement behaving in a version discovery world: https://review.openstack.org/#/c/522002/ | |
| 16:43:21 | mdbooth | mriedem: Did you see my unit test, btw? | |
| 16:43:38 | mnaser | mriedem: im not sure if this is relevant or not but do you think the fact we run 3x nova-schedulers could play a role in this .. somehow | |
| 16:44:10 | mriedem | mdbooth: yes i pulled part of the test and applied it to an existing test to recreate the bug | |
| 16:44:21 | mriedem | mnaser: separate processes? | |
| 16:44:30 | mnaser | yes, on seperate machines too | |
| 16:44:34 | mriedem | mnaser: should be ok if it's separate processes because the host_state_map is global to the single process | |
| 16:44:40 | mnaser | (i dont think it would but i figure i would mention it0 | |
| 16:44:50 | mnaser | oh you know what | |
| 16:44:58 | mnaser | let me see if i can grep the logs for "Removing dead compute node" | |
| 16:45:15 | mnaser | if that helps at all.. | |
| 16:46:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix an error in _get_host_states when deleting a compute node https://review.openstack.org/529352 | |
| 16:46:43 | mriedem | mnaser: btw with placement making claims via the filter scheduler in pike, we actually expect you to be able to run multile scheduler processes safely | |
| 16:46:56 | mriedem | i have a todo to update one of our CI jobs to do that (run with 2 schedulers) | |
| 16:47:03 | mriedem | mdbooth: jaypipes: ^ updated | |
| 16:47:07 | mnaser | mriedem: yeah before we have a little hacky trick of letting things reschedule more often | |
| 16:47:17 | mnaser | because in large volume schedulers all take the same decisions (before at least) | |
| 16:47:37 | mriedem | mnaser: yup, exactly why we're doing claims in the scheduler rather than rely on the compute to fail and reschedule | |
| 16:47:57 | mriedem | mnaser: so in pike, you should be good to remove the core/ram/disk filters, but you'll still need to rely on numa claims in the compute | |
| 16:48:52 | openstackgerrit | Ilya Shakhat proposed openstack/nova master: Initialize osprofiler in uWSGI application https://review.openstack.org/519664 | |
| 16:49:45 | mnaser | mriedem: ok i just found a really weird amount of compute nodes being marked as dead in logs | |
| 16:50:10 | mnaser | and i guess they're flapping for some reason and the state of self.host_state_map is always changing with the generator | |
| 16:50:26 | mriedem | mnaser: my guess would be the auto-disable thing if you don't have that disabled globally? | |
| 16:50:30 | mriedem | it's a per-compute config | |
| 16:50:32 | mnaser | or ntp | |
| 16:50:41 | gibi_away | I'm starting my vacation now. I will be back officially on 8th of Jan | |
| 16:50:43 | mriedem | yeah i guess service timeouts | |
| 16:50:48 | mriedem | gibi_away: NOOOO | |
| 16:51:02 | mnaser | gibi_away: have fun :-) | |
| 16:51:07 | gibi_away | mriedem: I'm trying to look at gerrit time to time but no promises | |
| 16:51:07 | mriedem | gibi_away: enjoy, see you in a few weeks | |
| 16:51:12 | mriedem | don't | |
| 16:51:18 | mnaser | mriedem: so logs from august till now in scheduler ... grep 'Removing dead compute node' /var/log/nova/nova-scheduler.log | wc -l => 102490. | |
| 16:51:23 | mnaser | so um | |
| 16:51:27 | gibi_away | mriedem: see you | |
| 16:51:43 | mnaser | something is wrong there, i guess with all those services going dead, it was creating the race condition | |
| 16:51:47 | mnaser | now why they are dead thats an exercise for me | |
| 16:52:16 | openstackgerrit | rahul bardia proposed openstack/python-novaclient master: nova limits ERROR (Exception): Field names must be unique https://review.openstack.org/527922 | |
| 16:52:21 | mnaser | some being removed 4 minutes ago.. i think this is ntp related | |
| 16:52:52 | mnaser | the conductor is the one that updates the service record.. or is it the compute directly? | |
| 16:53:04 | mnaser | (as in, who decides what time goes in updated_at) | |
| 16:54:37 | mriedem | that's the servicegroup api i think | |
| 16:55:05 | mriedem | https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L56 | |
| 16:55:39 | mnaser | so one of the compute nodes that went down 4 minutes ago is 4 seconds ahead | |
| 16:55:57 | mriedem | are you seeing any of the logging messages in here? https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L86 | |
| 16:57:26 | mnaser | mriedem: no | |
| 16:57:39 | melwitt | mnaser: I agree that it sounds ntp related if you're seeing some computes ahead in time of others | |
| 16:58:09 | mnaser | melwitt: this one that went 'down' was actually 4 seconds ahead though so if anything it would have updated with a time that is more ahead | |
| 16:58:21 | mriedem | oopsy doodle http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-n-super-cond.txt.gz#_Dec_20_15_15_20_744636 | |
| 16:58:27 | mriedem | cdent: ^ | |
| 17:00:40 | mriedem | http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-placement-api.txt.gz#_Dec_20_15_15_20_666337 | |
| 17:01:17 | mriedem | File "/opt/stack/new/nova/nova/objects/resource_provider.py", line 2029, in _set_allocations | |
| 17:01:22 | mriedem | nova.exception.ObjectActionError: Object action create failed because: already created | |
| 17:03:26 | mriedem | https://bugs.launchpad.net/nova/+bug/1739453 | |
| 17:03:27 | openstack | Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed] | |
| 17:06:14 | mriedem | not sure if using POST /allocations would fix this or not | |
| 17:06:38 | melwitt | mnaser, mriedem: it's the Service object that joins the servicegroup, so I think updated_at has to be the time of the host the service is running on, in this case the compute host | |
| 17:07:34 | openstackgerrit | Merged openstack/nova master: [placement] Separate API schemas (allocation_candidate) https://review.openstack.org/528635 | |
| 17:10:42 | cdent | mriedem: that's rather novel | |
| 17:14:24 | mriedem | yeah 2 hits in 7 days | |
| 17:16:27 | cdent | the code is claiming that that error should only happen when the incoming allocation objects already have an 'id' field | |
| 17:18:37 | cdent | which will take some digging to figure out how is even possible | |
| 17:19:57 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Move aggregates from report client to ProviderTree https://review.openstack.org/521685 | |
| 17:19:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Raise on API errors getting aggregates/traits https://review.openstack.org/526540 | |
| 17:19:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Track associated sharing RPs in report client https://review.openstack.org/526539 | |
| 17:19:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Track provider traits in report client https://review.openstack.org/521686 | |
| 17:19:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Track tree-associated providers in report client https://review.openstack.org/526541 | |
| 17:19:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756 | |
| 17:20:00 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add nested resources to server moving tests https://review.openstack.org/527728 | |
| 17:20:54 | stephenfin | jaypipes: Think there might be misunderstanding going on here https://review.openstack.org/#/c/527472/ | |
| 17:21:19 | stephenfin | Not saying I don't do overly clever things. Just that this isn't one of them ;) | |
| 17:21:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098 | |
| 17:21:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187 | |
| 17:22:04 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246 | |
| 17:26:46 | cdent | mriedem: I think I figured out what's up, I left a comment on the bug, assigned myself, and will try a fix | |
| 17:27:43 | mriedem | cdent: ah yeah didn't see the retry decorator | |
| 17:27:44 | mriedem | nice catch | |
| 17:28:10 | cdent | I can't decide what to shake it at, my my tiny fist is shaking | |
| 17:28:16 | cdent | s/my my/but my/ | |
| 17:28:56 | mriedem | yeah not sure how to fix that | |
| 17:29:30 | mdbooth | Sorry in advance, had a merge conflict half way up :/ | |
| 17:29:58 | openstackgerrit | Matthew Booth proposed openstack/nova master: Make BlockDeviceMapping object support uuid https://review.openstack.org/242603 | |
| 17:29:58 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add uuid column to BlockDeviceMapping https://review.openstack.org/242602 | |
| 17:29:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: DriverBlockDevice: make subclasses inherit _proxy_as_attr https://review.openstack.org/524167 | |
| 17:29:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add an online migration for BDM.uuid https://review.openstack.org/525599 | |
| 17:30:00 | openstackgerrit | Matthew Booth proposed openstack/nova master: Give volume DriverBlockDevice classes a common prefix https://review.openstack.org/526346 | |
| 17:30:00 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose BDM uuid to drivers https://review.openstack.org/529037 | |
| 17:30:01 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | |
| 17:30:01 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347 | |
| 17:30:02 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | |
| 17:30:02 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | |
| 17:30:03 | openstackgerrit | Matthew Booth proposed openstack/nova master: Use real block_device_info data in libvirt tests https://review.openstack.org/527916 | |
| 17:30:03 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | |