| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-12 | |||
| 13:52:28 | gnuoy | ack, ta | |
| 13:52:39 | kashyap | gnuoy: Someone can just merge it with a single +2 & a +W | |
| 13:52:51 | kashyap | 2 +2s is a 'good pratcise' thing | |
| 13:52:52 | fried_rice | Oh, right, it's not enforced by gerrit | |
| 13:53:01 | fried_rice | It's by policy | |
| 13:53:01 | cdent | fried_rice: it's always morning somewhere | |
| 13:53:08 | gnuoy | ok, understood. thanks | |
| 13:53:24 | bhagyashris | iawn: Hi, are you around? | |
| 13:53:51 | fried_rice | gnuoy Sometimes trivial fixes are quick-approved, or a trivial patch set on a change that previously had multiple +2s, that kind of thing. | |
| 13:54:09 | gnuoy | I see, makes sense | |
| 13:54:23 | fried_rice | ianw ^ misspelled ping :) | |
| 13:54:47 | bhagyashris | sorry sorry | |
| 13:54:59 | bhagyashris | ianw: Hi, are you around? | |
| 13:56:00 | gibi | I'm seening a lot of functional failures like on the gate like this: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22u'host2'%20!%3D%20u'host1':%20path:%20root.payload.nova_object.data.host%5C%22 | |
| 13:56:11 | gibi | I opened https://bugs.launchpad.net/nova/+bug/1742935 | |
| 13:56:12 | openstack | Launchpad bug 1742935 in OpenStack Compute (nova) "TestServiceUpdateNotificationSample fails intermittently: u'host2' != u'host1': path: root.payload.nova_object.data.host " [Undecided,New] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 14:04:38 | fried_rice | cdent Is report client single threaded today? | |
| 14:04:48 | fried_rice | I thought it was, but started to think through it and realized it probably isn't. | |
| 14:05:07 | cdent | greenthreaded | |
| 14:05:12 | fried_rice | Because periodic tasks plus deploys | |
| 14:05:18 | fried_rice | I meant in practice rather than by design | |
| 14:05:19 | cdent | got somebody in the house, so gotta come back to it | |
| 14:08:52 | bhagyashris | sdague: Hi, Could you please review my devstack patch https://review.openstack.org/#/c/527907/ | |
| 14:12:28 | bhagyashris | sdague: thank you :) | |
| 14:13:25 | sdague | bhagyashris: no prob | |
| 14:18:30 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Revert "Deduplicate service status notification samples" https://review.openstack.org/533190 | |
| 14:19:12 | gibi | cores, the nova functional jobs are broken on master due to https://bugs.launchpad.net/nova/+bug/1742935 the revert unblocking the gate is https://review.openstack.org/533190 | |
| 14:19:15 | openstack | Launchpad bug 1742935 in OpenStack Compute (nova) "TestServiceUpdateNotificationSample fails intermittently: u'host2' != u'host1': path: root.payload.nova_object.data.host " [Undecided,In progress] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 14:21:53 | mriedem | gibi: +W | |
| 14:24:30 | gibi | mriedem: thank | |
| 14:24:45 | gibi | mriedem: where do we store the functional test job definition? is it in the nova tree? | |
| 14:27:09 | ildikov | mriedem: updated the multi-attach api patch to microversion 2.60 | |
| 14:30:34 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix comments in get_all_with_shared() https://review.openstack.org/533195 | |
| 14:35:50 | gibi | mriedem: nvm, I found it https://github.com/openstack-infra/openstack-zuul-jobs/blob/21241aec13dd3d8eae2916518a5ad8933509a512/zuul.d/jobs.yaml#L352 | |
| 14:38:46 | mriedem | gibi: that's the global one, there might be an override in project-config | |
| 14:39:27 | mriedem | gibi: https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L10600 | |
| 14:39:37 | mriedem | that's the nova-specific blacklist | |
| 14:40:22 | gibi | mriedem: thanks, so we have to fix this line https://github.com/openstack-infra/project-config/blob/5017c2ba472bd0439392f037036dbf198e00c137/zuul.d/projects.yaml#L10604 | |
| 14:40:48 | mriedem | ok - don't forget the gate queue definition for the same | |
| 14:40:49 | gibi | mriedem: hm, no | |
| 14:41:03 | mriedem | https://github.com/openstack-infra/project-config/blob/5017c2ba472bd0439392f037036dbf198e00c137/zuul.d/projects.yaml#L10797 | |
| 14:41:14 | mriedem | ildikov: thanks | |
| 14:42:26 | gibi | mriedem: these definitions looks correct, according to this the doc/notification_samples | |
| 14:42:42 | gibi | mriedem: should trigger the functional test | |
| 14:43:47 | mriedem | gibi: the parent override maybe? https://github.com/openstack-infra/openstack-zuul-jobs/blob/21241aec13dd3d8eae2916518a5ad8933509a512/zuul.d/jobs.yaml#L361 | |
| 14:44:16 | mriedem | i would think the one in project-config overrides the definition in the openstack-zuul-jobs repo though | |
| 14:44:20 | mriedem | you'd have to ask in infra | |
| 14:44:29 | gibi | OK, i will do that | |
| 14:53:55 | mriedem | need another core on the bottom 2 novaclient patches here for 2.57 and 2.58 https://review.openstack.org/#/c/528128/ | |
| 14:57:43 | bauzas | mriedem: I was just about +Wing the 2.58 :-) | |
| 14:58:52 | mriedem | bauzas: don't let me distract you | |
| 14:58:54 | mriedem | ildikov: looky here logs.openstack.org/89/532689/1/experimental/nova-multiattach/720b3d4/ | |
| 14:58:59 | mriedem | http://logs.openstack.org/89/532689/1/experimental/nova-multiattach/720b3d4/ | |
| 14:59:01 | mriedem | finally ran | |
| 14:59:04 | mriedem | stvnoyes: ^ | |
| 14:59:18 | bauzas | mriedem: the 2.58 is far quickier to review than the big 2.57 | |
| 14:59:27 | mriedem | yeah i know 2.57 wasn't fun | |
| 14:59:29 | mriedem | wasn't fun to write it | |
| 14:59:47 | ildikov | mriedem: I saw \o/ :) | |
| 14:59:49 | mriedem | that's why it took me a year to do the file injection deprecation patch | |
| 14:59:56 | stvnoyes | cool. working on the swap test now | |
| 15:01:21 | mriedem | actually, weird, it ran a really small set of tempest tests http://logs.openstack.org/89/532689/1/experimental/nova-multiattach/720b3d4/job-output.txt.gz#_2018-01-12_03_10_33_860455 | |
| 15:03:43 | cdent | fried_rice: I'm more available now | |
| 15:04:06 | fried_rice | cdent Trying to understand how urgent proper generation management is. | |
| 15:05:00 | openstackgerrit | Andreas Jaeger proposed openstack/nova master: DNM: Test trigger openstack-tox-functional https://review.openstack.org/533202 | |
| 15:05:12 | fried_rice | cdent If we can't hit placement API calls for the same provider from multiple threads, not urgent at all. | |
| 15:06:39 | mriedem | ahh missing DEVSTACK_GATE_TEMPEST_FULL=1 | |
| 15:08:41 | fried_rice | cdent So there's some kind of periodic task that updates inventory for a compute, right? And then if say a deploy is happening, is there a mutex for the periodic? (I seem to recall seeing something like that in the logs - "skipping such-and-such for instance X because it is in the state Y") | |
| 15:08:51 | ildikov | mriedem: I saw, but I didn't have time to think about what's missing :) | |
| 15:09:26 | cdent | fried_rice: I used to know, but that memory is archived | |
| 15:09:43 | mriedem | fried_rice: that's different | |
| 15:09:57 | mriedem | the compute node provider inventory update is the update_available_resource periodic in the compute manager, | |
| 15:10:04 | mriedem | the thing you're thinking of is the sync_power_state task | |
| 15:10:11 | mriedem | which skips instances that have a task_state set | |
| 15:10:18 | openstackgerrit | Andreas Jaeger proposed openstack/nova master: DNM: Further testing https://review.openstack.org/533206 | |
| 15:10:29 | mriedem | *sync_instance_power_state, something like that | |
| 15:10:38 | mriedem | the task that auto-stops your instances | |
| 15:10:41 | fried_rice | ah, right, thanks mriedem. So *can* multiple report client threads try to hit placement at the same time for the same provider? | |
| 15:11:00 | mriedem | for inventory? | |
| 15:11:25 | mriedem | the update_available_resource task is in a semaphore lock | |
| 15:11:27 | mriedem | on the compute | |
| 15:11:37 | fried_rice | Well, for inventory as the code stands today, but we're working on getting traits in there, so now we have to coordinate RP generation between inventory and traits (and eventually aggregates etc.) | |
| 15:11:41 | mriedem | so when things like instance create/delete/move happens which require a claim or adjust inventory, those are in that shared lock | |
| 15:11:42 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 15:11:48 | mriedem | and the update_available_resource task is in the same lock | |
| 15:12:12 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L51 | |
| 15:12:14 | mriedem | "the lock" | |
| 15:12:35 | elmaciej | Hello! Maybe someone know how to achieve live-migration on libvirt lvm backend? I have shared lun between computes and using lvm as backend. But livemigration fails as it's saying it's not shared storage. | |
| 15:13:12 | fried_rice | nice. So if we were to use that same semaphore for whatever providery thing we're mucking with, we should be safe with the current (loose/lacking) generation handling. | |
| 15:15:39 | mriedem | i guess | |
| 15:15:46 | mriedem | it is just a big dumb lock though, | |
| 15:16:06 | mriedem | and there are known perf issues with holding that lock too long which starves other stuff | |
| 15:16:23 | cdent | does "too long" have a value? | |
| 15:16:33 | mriedem | at least 60 seconds yeah for default rpc timeout | |
| 15:16:35 | fried_rice | mriedem Yeah, I'll want to solve the generation thing properly eventually; I'm trying to determine the urgency of doing that. | |
| 15:16:47 | mriedem | we've had gate issues in the past where that lock was held for more than 60 seconds and we'd have messaging timeouts | |
| 15:17:19 | mriedem | otherwise "too long" is when some operator tells us in a year that their compute with 70 instances on it is holding the lock for 45 seconds and that blows up their server create time on that compute host | |
| 15:17:37 | mriedem | but of course that's hard to know ahead of time | |
| 15:17:41 | mriedem | or test for | |
| 15:21:45 | cdent | mriedem: is your vacation next week? back to mexico? | |