Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-20
16:44:10 mriedem mdbooth: yes i pulled part of the test and applied it to an existing test to recreate the bug
16:44:21 mriedem mnaser: separate processes?
16:44:30 mnaser yes, on seperate machines too
16:44:34 mriedem mnaser: should be ok if it's separate processes because the host_state_map is global to the single process
16:44:40 mnaser (i dont think it would but i figure i would mention it0
16:44:50 mnaser oh you know what
16:44:58 mnaser let me see if i can grep the logs for "Removing dead compute node"
16:45:15 mnaser if that helps at all..
16:46:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix an error in _get_host_states when deleting a compute node https://review.openstack.org/529352
16:46:43 mriedem mnaser: btw with placement making claims via the filter scheduler in pike, we actually expect you to be able to run multile scheduler processes safely
16:46:56 mriedem i have a todo to update one of our CI jobs to do that (run with 2 schedulers)
16:47:03 mriedem mdbooth: jaypipes: ^ updated
16:47:07 mnaser mriedem: yeah before we have a little hacky trick of letting things reschedule more often
16:47:17 mnaser because in large volume schedulers all take the same decisions (before at least)
16:47:37 mriedem mnaser: yup, exactly why we're doing claims in the scheduler rather than rely on the compute to fail and reschedule
16:47:57 mriedem mnaser: so in pike, you should be good to remove the core/ram/disk filters, but you'll still need to rely on numa claims in the compute
16:48:52 openstackgerrit Ilya Shakhat proposed openstack/nova master: Initialize osprofiler in uWSGI application https://review.openstack.org/519664
16:49:45 mnaser mriedem: ok i just found a really weird amount of compute nodes being marked as dead in logs
16:50:10 mnaser and i guess they're flapping for some reason and the state of self.host_state_map is always changing with the generator
16:50:26 mriedem mnaser: my guess would be the auto-disable thing if you don't have that disabled globally?
16:50:30 mriedem it's a per-compute config
16:50:32 mnaser or ntp
16:50:41 gibi_away I'm starting my vacation now. I will be back officially on 8th of Jan
16:50:43 mriedem yeah i guess service timeouts
16:50:48 mriedem gibi_away: NOOOO
16:51:02 mnaser gibi_away: have fun :-)
16:51:07 mriedem gibi_away: enjoy, see you in a few weeks
16:51:07 gibi_away mriedem: I'm trying to look at gerrit time to time but no promises
16:51:12 mriedem don't
16:51:18 mnaser mriedem: so logs from august till now in scheduler ... grep 'Removing dead compute node' /var/log/nova/nova-scheduler.log | wc -l => 102490.
16:51:23 mnaser so um
16:51:27 gibi_away mriedem: see you
16:51:43 mnaser something is wrong there, i guess with all those services going dead, it was creating the race condition
16:51:47 mnaser now why they are dead thats an exercise for me
16:52:16 openstackgerrit rahul bardia proposed openstack/python-novaclient master: nova limits ERROR (Exception): Field names must be unique https://review.openstack.org/527922
16:52:21 mnaser some being removed 4 minutes ago.. i think this is ntp related
16:52:52 mnaser the conductor is the one that updates the service record.. or is it the compute directly?
16:53:04 mnaser (as in, who decides what time goes in updated_at)
16:54:37 mriedem that's the servicegroup api i think
16:55:05 mriedem https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L56
16:55:39 mnaser so one of the compute nodes that went down 4 minutes ago is 4 seconds ahead
16:55:57 mriedem are you seeing any of the logging messages in here? https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L86
16:57:26 mnaser mriedem: no
16:57:39 melwitt mnaser: I agree that it sounds ntp related if you're seeing some computes ahead in time of others
16:58:09 mnaser melwitt: this one that went 'down' was actually 4 seconds ahead though so if anything it would have updated with a time that is more ahead
16:58:21 mriedem oopsy doodle http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-n-super-cond.txt.gz#_Dec_20_15_15_20_744636
16:58:27 mriedem cdent: ^
17:00:40 mriedem http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-placement-api.txt.gz#_Dec_20_15_15_20_666337
17:01:17 mriedem File "/opt/stack/new/nova/nova/objects/resource_provider.py", line 2029, in _set_allocations
17:01:22 mriedem nova.exception.ObjectActionError: Object action create failed because: already created
17:03:26 mriedem https://bugs.launchpad.net/nova/+bug/1739453
17:03:27 openstack Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed]
17:06:14 mriedem not sure if using POST /allocations would fix this or not
17:06:38 melwitt mnaser, mriedem: it's the Service object that joins the servicegroup, so I think updated_at has to be the time of the host the service is running on, in this case the compute host
17:07:34 openstackgerrit Merged openstack/nova master: [placement] Separate API schemas (allocation_candidate) https://review.openstack.org/528635
17:10:42 cdent mriedem: that's rather novel
17:14:24 mriedem yeah 2 hits in 7 days
17:16:27 cdent the code is claiming that that error should only happen when the incoming allocation objects already have an 'id' field
17:18:37 cdent which will take some digging to figure out how is even possible
17:19:57 openstackgerrit Stephen Finucane proposed openstack/nova master: Move aggregates from report client to ProviderTree https://review.openstack.org/521685
17:19:58 openstackgerrit Stephen Finucane proposed openstack/nova master: Track provider traits in report client https://review.openstack.org/521686
17:19:58 openstackgerrit Stephen Finucane proposed openstack/nova master: Track associated sharing RPs in report client https://review.openstack.org/526539
17:19:58 openstackgerrit Stephen Finucane proposed openstack/nova master: Raise on API errors getting aggregates/traits https://review.openstack.org/526540
17:19:59 openstackgerrit Stephen Finucane proposed openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756
17:19:59 openstackgerrit Stephen Finucane proposed openstack/nova master: Track tree-associated providers in report client https://review.openstack.org/526541
17:20:00 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Add nested resources to server moving tests https://review.openstack.org/527728
17:20:54 stephenfin jaypipes: Think there might be misunderstanding going on here https://review.openstack.org/#/c/527472/
17:21:19 stephenfin Not saying I don't do overly clever things. Just that this isn't one of them ;)
17:21:33 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098
17:21:53 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187
17:22:04 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246
17:26:46 cdent mriedem: I think I figured out what's up, I left a comment on the bug, assigned myself, and will try a fix
17:27:43 mriedem cdent: ah yeah didn't see the retry decorator
17:27:44 mriedem nice catch
17:28:10 cdent I can't decide what to shake it at, my my tiny fist is shaking
17:28:16 cdent s/my my/but my/
17:28:56 mriedem yeah not sure how to fix that
17:29:30 mdbooth Sorry in advance, had a merge conflict half way up :/
17:29:58 openstackgerrit Matthew Booth proposed openstack/nova master: Add uuid column to BlockDeviceMapping https://review.openstack.org/242602
17:29:58 openstackgerrit Matthew Booth proposed openstack/nova master: Make BlockDeviceMapping object support uuid https://review.openstack.org/242603
17:29:59 openstackgerrit Matthew Booth proposed openstack/nova master: Add an online migration for BDM.uuid https://review.openstack.org/525599
17:29:59 openstackgerrit Matthew Booth proposed openstack/nova master: DriverBlockDevice: make subclasses inherit _proxy_as_attr https://review.openstack.org/524167
17:30:00 openstackgerrit Matthew Booth proposed openstack/nova master: Expose BDM uuid to drivers https://review.openstack.org/529037
17:30:00 openstackgerrit Matthew Booth proposed openstack/nova master: Give volume DriverBlockDevice classes a common prefix https://review.openstack.org/526346
17:30:01 openstackgerrit Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347
17:30:01 openstackgerrit Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028
17:30:02 openstackgerrit Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029
17:30:02 openstackgerrit Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362
17:30:03 openstackgerrit Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363
17:30:03 openstackgerrit Matthew Booth proposed openstack/nova master: Use real block_device_info data in libvirt tests https://review.openstack.org/527916
17:30:04 openstackgerrit Matthew Booth proposed openstack/nova master: Fix libvirt volume tests passing invalid disk_info https://review.openstack.org/529328
17:30:04 openstackgerrit Matthew Booth proposed openstack/nova master: Pass disk_info dict to libvirt_info https://review.openstack.org/529329
17:30:05 openstackgerrit Matthew Booth proposed openstack/nova master: Local disk serial numbers for the libvirt driver https://review.openstack.org/529380
17:36:46 cdent jaypipes: can you recall why/if it is important that AllocationList.create_all() is side effecty on the objects that it passes in? https://github.com/openstack/nova/blob/master/nova/tests/unit/objects/test_resource_provider.py#L355-L357 and https://github.com/openstack/nova/blob/609ddc2244bf9ba84d66cced52e7d4e2987ec669/nova/objects/resource_provider.py#L2061-L2086 and https://bugs.launchpad.net/nova/+bug/1739453
17:36:47 openstack Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed] - Assigned to Chris Dent (cdent)
17:41:15 melwitt mriedem: FYI the consoles series is ready to go, complete with devstack patch sandwich around it for the tempest vnc tests. stephenfin has gone through it already https://review.openstack.org/#/q/topic:bp/convert-consoles-to-objects+(status:open+OR+status:merged)
17:41:57 mriedem devstack sandwich worked?
17:42:06 melwitt yes. I owe you a cream soda
17:42:11 mriedem awesome
17:43:17 edleafe cdent: if the allocation had an id, then it already exists, no?

Earlier   Later