Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-12
14:04:38 fried_rice cdent Is report client single threaded today?
14:04:48 fried_rice I thought it was, but started to think through it and realized it probably isn't.
14:05:07 cdent greenthreaded
14:05:12 fried_rice Because periodic tasks plus deploys
14:05:18 fried_rice I meant in practice rather than by design
14:05:19 cdent got somebody in the house, so gotta come back to it
14:08:52 bhagyashris sdague: Hi, Could you please review my devstack patch https://review.openstack.org/#/c/527907/
14:12:28 bhagyashris sdague: thank you :)
14:13:25 sdague bhagyashris: no prob
14:18:30 openstackgerrit Balazs Gibizer proposed openstack/nova master: Revert "Deduplicate service status notification samples" https://review.openstack.org/533190
14:19:12 gibi cores, the nova functional jobs are broken on master due to https://bugs.launchpad.net/nova/+bug/1742935 the revert unblocking the gate is https://review.openstack.org/533190
14:19:15 openstack Launchpad bug 1742935 in OpenStack Compute (nova) "TestServiceUpdateNotificationSample fails intermittently: u'host2' != u'host1': path: root.payload.nova_object.data.host " [Undecided,In progress] - Assigned to Balazs Gibizer (balazs-gibizer)
14:21:53 mriedem gibi: +W
14:24:30 gibi mriedem: thank
14:24:45 gibi mriedem: where do we store the functional test job definition? is it in the nova tree?
14:27:09 ildikov mriedem: updated the multi-attach api patch to microversion 2.60
14:30:34 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Fix comments in get_all_with_shared() https://review.openstack.org/533195
14:35:50 gibi mriedem: nvm, I found it https://github.com/openstack-infra/openstack-zuul-jobs/blob/21241aec13dd3d8eae2916518a5ad8933509a512/zuul.d/jobs.yaml#L352
14:38:46 mriedem gibi: that's the global one, there might be an override in project-config
14:39:27 mriedem gibi: https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L10600
14:39:37 mriedem that's the nova-specific blacklist
14:40:22 gibi mriedem: thanks, so we have to fix this line https://github.com/openstack-infra/project-config/blob/5017c2ba472bd0439392f037036dbf198e00c137/zuul.d/projects.yaml#L10604
14:40:48 mriedem ok - don't forget the gate queue definition for the same
14:40:49 gibi mriedem: hm, no
14:41:03 mriedem https://github.com/openstack-infra/project-config/blob/5017c2ba472bd0439392f037036dbf198e00c137/zuul.d/projects.yaml#L10797
14:41:14 mriedem ildikov: thanks
14:42:26 gibi mriedem: these definitions looks correct, according to this the doc/notification_samples
14:42:42 gibi mriedem: should trigger the functional test
14:43:47 mriedem gibi: the parent override maybe? https://github.com/openstack-infra/openstack-zuul-jobs/blob/21241aec13dd3d8eae2916518a5ad8933509a512/zuul.d/jobs.yaml#L361
14:44:16 mriedem i would think the one in project-config overrides the definition in the openstack-zuul-jobs repo though
14:44:20 mriedem you'd have to ask in infra
14:44:29 gibi OK, i will do that
14:53:55 mriedem need another core on the bottom 2 novaclient patches here for 2.57 and 2.58 https://review.openstack.org/#/c/528128/
14:57:43 bauzas mriedem: I was just about +Wing the 2.58 :-)
14:58:52 mriedem bauzas: don't let me distract you
14:58:54 mriedem ildikov: looky here logs.openstack.org/89/532689/1/experimental/nova-multiattach/720b3d4/
14:58:59 mriedem http://logs.openstack.org/89/532689/1/experimental/nova-multiattach/720b3d4/
14:59:01 mriedem finally ran
14:59:04 mriedem stvnoyes: ^
14:59:18 bauzas mriedem: the 2.58 is far quickier to review than the big 2.57
14:59:27 mriedem yeah i know 2.57 wasn't fun
14:59:29 mriedem wasn't fun to write it
14:59:47 ildikov mriedem: I saw \o/ :)
14:59:49 mriedem that's why it took me a year to do the file injection deprecation patch
14:59:56 stvnoyes cool. working on the swap test now
15:01:21 mriedem actually, weird, it ran a really small set of tempest tests http://logs.openstack.org/89/532689/1/experimental/nova-multiattach/720b3d4/job-output.txt.gz#_2018-01-12_03_10_33_860455
15:03:43 cdent fried_rice: I'm more available now
15:04:06 fried_rice cdent Trying to understand how urgent proper generation management is.
15:05:00 openstackgerrit Andreas Jaeger proposed openstack/nova master: DNM: Test trigger openstack-tox-functional https://review.openstack.org/533202
15:05:12 fried_rice cdent If we can't hit placement API calls for the same provider from multiple threads, not urgent at all.
15:06:39 mriedem ahh missing DEVSTACK_GATE_TEMPEST_FULL=1
15:08:41 fried_rice cdent So there's some kind of periodic task that updates inventory for a compute, right? And then if say a deploy is happening, is there a mutex for the periodic? (I seem to recall seeing something like that in the logs - "skipping such-and-such for instance X because it is in the state Y")
15:08:51 ildikov mriedem: I saw, but I didn't have time to think about what's missing :)
15:09:26 cdent fried_rice: I used to know, but that memory is archived
15:09:43 mriedem fried_rice: that's different
15:09:57 mriedem the compute node provider inventory update is the update_available_resource periodic in the compute manager,
15:10:04 mriedem the thing you're thinking of is the sync_power_state task
15:10:11 mriedem which skips instances that have a task_state set
15:10:18 openstackgerrit Andreas Jaeger proposed openstack/nova master: DNM: Further testing https://review.openstack.org/533206
15:10:29 mriedem *sync_instance_power_state, something like that
15:10:38 mriedem the task that auto-stops your instances
15:10:41 fried_rice ah, right, thanks mriedem. So *can* multiple report client threads try to hit placement at the same time for the same provider?
15:11:00 mriedem for inventory?
15:11:25 mriedem the update_available_resource task is in a semaphore lock
15:11:27 mriedem on the compute
15:11:37 fried_rice Well, for inventory as the code stands today, but we're working on getting traits in there, so now we have to coordinate RP generation between inventory and traits (and eventually aggregates etc.)
15:11:41 mriedem so when things like instance create/delete/move happens which require a claim or adjust inventory, those are in that shared lock
15:11:42 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
15:11:48 mriedem and the update_available_resource task is in the same lock
15:12:12 mriedem https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L51
15:12:14 mriedem "the lock"
15:12:35 elmaciej Hello! Maybe someone know how to achieve live-migration on libvirt lvm backend? I have shared lun between computes and using lvm as backend. But livemigration fails as it's saying it's not shared storage.
15:13:12 fried_rice nice. So if we were to use that same semaphore for whatever providery thing we're mucking with, we should be safe with the current (loose/lacking) generation handling.
15:15:39 mriedem i guess
15:15:46 mriedem it is just a big dumb lock though,
15:16:06 mriedem and there are known perf issues with holding that lock too long which starves other stuff
15:16:23 cdent does "too long" have a value?
15:16:33 mriedem at least 60 seconds yeah for default rpc timeout
15:16:35 fried_rice mriedem Yeah, I'll want to solve the generation thing properly eventually; I'm trying to determine the urgency of doing that.
15:16:47 mriedem we've had gate issues in the past where that lock was held for more than 60 seconds and we'd have messaging timeouts
15:17:19 mriedem otherwise "too long" is when some operator tells us in a year that their compute with 70 instances on it is holding the lock for 45 seconds and that blows up their server create time on that compute host
15:17:37 mriedem but of course that's hard to know ahead of time
15:17:41 mriedem or test for
15:21:45 cdent mriedem: is your vacation next week? back to mexico?
15:21:50 mriedem yup
15:21:53 mriedem ildikov: rebase looks good, nice work
15:22:00 mriedem gibi: thanks for reviewing the multiattach patches
15:22:06 mriedem gibi: just one more to go.... :)
15:23:03 ildikov mriedem: cool, thanks for double checking!
15:25:44 openstackgerrit Matt Riedemann proposed openstack/nova master: Add the nova-multiattach experimental queue job https://review.openstack.org/532689
15:25:50 mriedem put the new CI job on top of the series
15:26:31 mriedem wonder if i should make that non-voting check
15:26:37 openstackgerrit Balazs Gibizer proposed openstack/nova master: DNM: Testing if funct test is triggered https://review.openstack.org/533211
15:26:37 openstackgerrit Balazs Gibizer proposed openstack/nova master: DNM: Test how to override irrelevant-fiels in zuul jobs https://review.openstack.org/533210
15:27:56 gibi mriedem: I will have not time to dig into the last multi attach patch today but I will check back on Monday
15:28:04 mriedem cool, thanks
15:28:57 mriedem johnthetubaguy: i think this is probably something you as an ironic loving person should probably look at https://review.openstack.org/#/c/527541/ - i think we need that before we can remove the ironic flavor migration routine, and we also likely should backport that to pike
15:29:05 openstackgerrit Édouard Thuleau proposed openstack/nova master: Update plugs Contrail methods to work with prisep https://review.openstack.org/533212
15:30:24 gibi mriedem: based on what I learned today in infra, the project-config job definitions does not override the jobs.yaml at all so it is possible that we have more cases where the proper jobs are not triggered
15:31:01 mriedem gibi: so https://github.com/openstack-infra/project-config/blob/5017c2ba472bd0439392f037036dbf198e00c137/zuul.d/projects.yaml#L10604 makes no difference?

Earlier   Later