Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-20
16:29:03 mnaser happened with 9 different compute nodes in the KeyError
16:29:29 mnaser and i know for a fact some of thoser havent been deleted, especially those a few days ago too
16:29:31 mriedem do those computes have corresponding services table entries?
16:29:52 mnaser mriedem: i mean i see them in 'nova service-list'
16:29:58 mnaser and their updated_at is recent so they're reporting in
16:30:13 mnaser afaik the only way for that service record to go away is... if someone deleted it?
16:30:38 mriedem correct
16:31:10 mriedem we do start auto-disabling computes that fail to build 10 times in a row
16:31:20 mriedem but in the scheduler code here, we include disabled services
16:31:54 mnaser mriedem: i think we disabled that because scheduling bugs would shut down our entire cloud lol
16:32:12 mnaser ex: failing to place numa on server will result in a build fail count being increased
16:38:57 mriedem mdbooth: ok i think we can still use part of your test and maintain the generator
16:39:02 mriedem i'm going to update your patch quick
16:39:18 mdbooth mriedem: on a call, but that's cool
16:42:46 cdent jaypipes, mriedem: if you're both still around, this is a good and relatively straightforward win for placement behaving in a version discovery world: https://review.openstack.org/#/c/522002/
16:43:21 mdbooth mriedem: Did you see my unit test, btw?
16:43:38 mnaser mriedem: im not sure if this is relevant or not but do you think the fact we run 3x nova-schedulers could play a role in this .. somehow
16:44:10 mriedem mdbooth: yes i pulled part of the test and applied it to an existing test to recreate the bug
16:44:21 mriedem mnaser: separate processes?
16:44:30 mnaser yes, on seperate machines too
16:44:34 mriedem mnaser: should be ok if it's separate processes because the host_state_map is global to the single process
16:44:40 mnaser (i dont think it would but i figure i would mention it0
16:44:50 mnaser oh you know what
16:44:58 mnaser let me see if i can grep the logs for "Removing dead compute node"
16:45:15 mnaser if that helps at all..
16:46:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix an error in _get_host_states when deleting a compute node https://review.openstack.org/529352
16:46:43 mriedem mnaser: btw with placement making claims via the filter scheduler in pike, we actually expect you to be able to run multile scheduler processes safely
16:46:56 mriedem i have a todo to update one of our CI jobs to do that (run with 2 schedulers)
16:47:03 mriedem mdbooth: jaypipes: ^ updated
16:47:07 mnaser mriedem: yeah before we have a little hacky trick of letting things reschedule more often
16:47:17 mnaser because in large volume schedulers all take the same decisions (before at least)
16:47:37 mriedem mnaser: yup, exactly why we're doing claims in the scheduler rather than rely on the compute to fail and reschedule
16:47:57 mriedem mnaser: so in pike, you should be good to remove the core/ram/disk filters, but you'll still need to rely on numa claims in the compute
16:48:52 openstackgerrit Ilya Shakhat proposed openstack/nova master: Initialize osprofiler in uWSGI application https://review.openstack.org/519664
16:49:45 mnaser mriedem: ok i just found a really weird amount of compute nodes being marked as dead in logs
16:50:10 mnaser and i guess they're flapping for some reason and the state of self.host_state_map is always changing with the generator
16:50:26 mriedem mnaser: my guess would be the auto-disable thing if you don't have that disabled globally?
16:50:30 mriedem it's a per-compute config
16:50:32 mnaser or ntp
16:50:41 gibi_away I'm starting my vacation now. I will be back officially on 8th of Jan
16:50:43 mriedem yeah i guess service timeouts
16:50:48 mriedem gibi_away: NOOOO
16:51:02 mnaser gibi_away: have fun :-)
16:51:07 mriedem gibi_away: enjoy, see you in a few weeks
16:51:07 gibi_away mriedem: I'm trying to look at gerrit time to time but no promises
16:51:12 mriedem don't
16:51:18 mnaser mriedem: so logs from august till now in scheduler ... grep 'Removing dead compute node' /var/log/nova/nova-scheduler.log | wc -l => 102490.
16:51:23 mnaser so um
16:51:27 gibi_away mriedem: see you
16:51:43 mnaser something is wrong there, i guess with all those services going dead, it was creating the race condition
16:51:47 mnaser now why they are dead thats an exercise for me
16:52:16 openstackgerrit rahul bardia proposed openstack/python-novaclient master: nova limits ERROR (Exception): Field names must be unique https://review.openstack.org/527922
16:52:21 mnaser some being removed 4 minutes ago.. i think this is ntp related
16:52:52 mnaser the conductor is the one that updates the service record.. or is it the compute directly?
16:53:04 mnaser (as in, who decides what time goes in updated_at)
16:54:37 mriedem that's the servicegroup api i think
16:55:05 mriedem https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L56
16:55:39 mnaser so one of the compute nodes that went down 4 minutes ago is 4 seconds ahead
16:55:57 mriedem are you seeing any of the logging messages in here? https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L86
16:57:26 mnaser mriedem: no
16:57:39 melwitt mnaser: I agree that it sounds ntp related if you're seeing some computes ahead in time of others
16:58:09 mnaser melwitt: this one that went 'down' was actually 4 seconds ahead though so if anything it would have updated with a time that is more ahead
16:58:21 mriedem oopsy doodle http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-n-super-cond.txt.gz#_Dec_20_15_15_20_744636
16:58:27 mriedem cdent: ^
17:00:40 mriedem http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-placement-api.txt.gz#_Dec_20_15_15_20_666337
17:01:17 mriedem File "/opt/stack/new/nova/nova/objects/resource_provider.py", line 2029, in _set_allocations
17:01:22 mriedem nova.exception.ObjectActionError: Object action create failed because: already created
17:03:26 mriedem https://bugs.launchpad.net/nova/+bug/1739453
17:03:27 openstack Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed]
17:06:14 mriedem not sure if using POST /allocations would fix this or not
17:06:38 melwitt mnaser, mriedem: it's the Service object that joins the servicegroup, so I think updated_at has to be the time of the host the service is running on, in this case the compute host
17:07:34 openstackgerrit Merged openstack/nova master: [placement] Separate API schemas (allocation_candidate) https://review.openstack.org/528635
17:10:42 cdent mriedem: that's rather novel
17:14:24 mriedem yeah 2 hits in 7 days
17:16:27 cdent the code is claiming that that error should only happen when the incoming allocation objects already have an 'id' field
17:18:37 cdent which will take some digging to figure out how is even possible
17:19:57 openstackgerrit Stephen Finucane proposed openstack/nova master: Move aggregates from report client to ProviderTree https://review.openstack.org/521685
17:19:58 openstackgerrit Stephen Finucane proposed openstack/nova master: Track provider traits in report client https://review.openstack.org/521686
17:19:58 openstackgerrit Stephen Finucane proposed openstack/nova master: Track associated sharing RPs in report client https://review.openstack.org/526539
17:19:58 openstackgerrit Stephen Finucane proposed openstack/nova master: Raise on API errors getting aggregates/traits https://review.openstack.org/526540
17:19:59 openstackgerrit Stephen Finucane proposed openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756
17:19:59 openstackgerrit Stephen Finucane proposed openstack/nova master: Track tree-associated providers in report client https://review.openstack.org/526541
17:20:00 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Add nested resources to server moving tests https://review.openstack.org/527728
17:20:54 stephenfin jaypipes: Think there might be misunderstanding going on here https://review.openstack.org/#/c/527472/
17:21:19 stephenfin Not saying I don't do overly clever things. Just that this isn't one of them ;)
17:21:33 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098
17:21:53 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187
17:22:04 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246
17:26:46 cdent mriedem: I think I figured out what's up, I left a comment on the bug, assigned myself, and will try a fix
17:27:43 mriedem cdent: ah yeah didn't see the retry decorator
17:27:44 mriedem nice catch
17:28:10 cdent I can't decide what to shake it at, my my tiny fist is shaking
17:28:16 cdent s/my my/but my/
17:28:56 mriedem yeah not sure how to fix that
17:29:30 mdbooth Sorry in advance, had a merge conflict half way up :/
17:29:58 openstackgerrit Matthew Booth proposed openstack/nova master: Add uuid column to BlockDeviceMapping https://review.openstack.org/242602
17:29:58 openstackgerrit Matthew Booth proposed openstack/nova master: Make BlockDeviceMapping object support uuid https://review.openstack.org/242603
17:29:59 openstackgerrit Matthew Booth proposed openstack/nova master: Add an online migration for BDM.uuid https://review.openstack.org/525599
17:29:59 openstackgerrit Matthew Booth proposed openstack/nova master: DriverBlockDevice: make subclasses inherit _proxy_as_attr https://review.openstack.org/524167
17:30:00 openstackgerrit Matthew Booth proposed openstack/nova master: Expose BDM uuid to drivers https://review.openstack.org/529037

Earlier   Later