| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-21 | |||
| 17:54:16 | mriedem | these are the 2 as of today https://bugs.launchpad.net/nova/+bugs?field.tag=rocky-rc-potential | |
| 17:54:21 | mriedem | looks like final rc is thursday | |
| 17:54:31 | mriedem | that first one has a fix in thegate | |
| 17:54:44 | mriedem | i wouldn't mind bouncing of a few of you on the 2nd one | |
| 17:54:47 | mriedem | *off a few | |
| 17:55:35 | melwitt | ack | |
| 17:58:23 | mriedem | dansmith: melwitt: so tl;dr, the issue in https://bugs.launchpad.net/nova/+bug/1788014 is that live migration fails and we trigger a lifecycle event which activates the port binding on the dest host incorrectly, it shouldn't do that, | |
| 17:58:23 | openstack | Launchpad bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 17:58:37 | mriedem | we're getting an event from libvirt but don't know if it's success or failure for the job, | |
| 17:59:14 | mriedem | so what i could do if we want to be low risk with the fix for rocky GA is just not listen on that event and we'll activate port bindings on success like we always did before the change, and we'll still do the early port activating on post-copy events if the live migration is successful | |
| 17:59:35 | mriedem | long-term we could check the actual job status and if failed, don't trigger our lifecycle event, but that's riskier for rocky GA at this point IMO | |
| 18:00:08 | mriedem | so i'd propose a 2-part fix, one partial that we backport and one for just stein | |
| 18:03:57 | melwitt | ok | |
| 18:04:28 | openstackgerrit | Dan Smith proposed openstack/nova stable/queens: Fix cancel_all_events event name parsing https://review.openstack.org/592086 | |
| 18:04:29 | openstackgerrit | Dan Smith proposed openstack/nova stable/queens: Wait for network-vif-plugged before starting live migration https://review.openstack.org/591735 | |
| 18:04:30 | openstackgerrit | Dan Smith proposed openstack/nova stable/queens: DNM: Debug patch to test live migration waiting https://review.openstack.org/591775 | |
| 18:05:23 | melwitt | I guess I don't understand what the lifecycle event gives us if we already know success or failure | |
| 18:05:55 | melwitt | without listening for it | |
| 18:07:06 | mriedem | in rocky we started listening on 2 new events, | |
| 18:07:12 | mriedem | one is post-copy and one is migration completed | |
| 18:07:31 | mriedem | the idea is that as soon as we switch we activate the port bindings on the dest host for minimal downtime | |
| 18:07:50 | mriedem | the problem is we get the latter event even if live migration fails | |
| 18:08:01 | mriedem | and we're not doing any conditional logic in that one to see if the job failed or not | |
| 18:09:26 | melwitt | I think I understand that part, it's just when you said "we'll activate port bindings on success like we always did before the change" it makes me not understand what gain the lifecycle event was supposed to give, if we already know success or failure without it | |
| 18:11:55 | mriedem | because if we can activate the network on the dest at the point the guest is paused to complete the transfer, it makes the network downtime window shorter | |
| 18:14:16 | melwitt | and without the lifecycle event, we activate the network later on after the guest is paused | |
| 18:14:19 | mriedem | we might still need my other fix for rollback, i'm not sure; what sean is hitting isn't a failure after post-copy | |
| 18:14:32 | mriedem | we activate the network after the guest transfer is complete and resumed on the dest | |
| 18:16:49 | melwitt | ok, got it | |
| 18:30:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Don't react to VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED events https://review.openstack.org/594508 | |
| 18:33:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Don't react to VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED events https://review.openstack.org/594508 | |
| 19:36:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event https://review.openstack.org/594527 | |
| 19:37:36 | mriedem | melwitt: dansmith: alright fyi ^ would need sean-k-mooney to test out the 2nd more complicated fix since he has the env that recreates the bug | |
| 19:37:43 | mriedem | and i need to run to an appt | |
| 19:39:36 | mriedem | also threw that stuff in the rc todo etherpad | |
| 19:43:27 | melwitt | ack | |
| 19:54:56 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 19:54:57 | openstackgerrit | Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033 | |
| 19:54:58 | openstackgerrit | Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider raise https://review.openstack.org/584598 | |
| 19:54:59 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 19:55:00 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648 | |
| 19:55:01 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 19:55:02 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049 | |
| 19:55:03 | openstackgerrit | Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 20:14:31 | dansmith | so I | |
| 20:14:48 | dansmith | am pretty sure my batching breaks tssurya's down cell in its current form | |
| 20:15:02 | dansmith | more specifically, it'll cause scatter gather to never notice failues | |
| 20:24:16 | melwitt | dansmith: it's specific to the batching? it looks like pre-batch the code still removes the "did not respond" and "raised exception" results | |
| 20:24:29 | dansmith | still removes? | |
| 20:25:21 | melwitt | dansmith: this part looks like it's removing "down cells" from results? https://review.openstack.org/#/c/592698/5/nova/compute/multi_cell_list.py@311 | |
| 20:26:00 | dansmith | right but with the batching we never hit that because we don't start executing the queries until the heapq | |
| 20:27:10 | melwitt | oh, I see | |
| 20:27:45 | melwitt | hmm | |
| 20:29:21 | dansmith | I'll just have to get a little more into the middle of that process and we just won't get the standard handlers from scatter gather | |
| 21:13:09 | mriedem | huh, you don't see instance.save() messaging timeouts in the gate very often http://logs.openstack.org/98/591898/3/check/nova-next/2d5e60c/logs/screen-n-cpu.txt.gz#_Aug_21_17_10_17_732263 | |
| 21:14:24 | mriedem | guessing this isn't good http://logs.openstack.org/98/591898/3/check/nova-next/2d5e60c/logs/screen-n-cpu.txt.gz#_Aug_21_17_10_17_426420 | |
| 21:16:54 | mriedem | seen here too http://logs.openstack.org/85/567785/7/check/nova-tox-functional-py35/d5a8036/job-output.txt#_2018-08-17_09_06_20_241427 | |
| 21:36:54 | mriedem | melwitt: what do you think is missing from placement for shared storage providers support? as far as i know, it's the nova stuff that's lacking as we identified ~2 weeks ago | |
| 21:37:05 | mriedem | for sure move operations are not ready for shared storage providers | |
| 21:37:18 | mriedem | the placement side of shared storage is pretty simple though, and has been done for a long time | |
| 21:40:28 | openstackgerrit | Dmitry Sutyagin proposed openstack/nova-specs master: Allow disabling KSM / mem-merge via extra spec https://review.openstack.org/593197 | |
| 21:41:23 | melwitt | mriedem: it's not that I think anything is missing, I'm pragmatically thinking of the integration work and if there is something unforeseen we need to fix. I expect bugs to shake out when we integrate something for the first time. I think not having bugs shake out will be the rarer case | |
| 21:41:38 | dansmith | mriedem: I think we were saying the same about aggregates being done in placement before we added the placement filter stuff and realized we needed tweaks | |
| 21:41:56 | dansmith | and surely thought what was being done in placement for NRPs was going to be usable by nova until we thought about it | |
| 21:46:01 | mriedem | dansmith: the member_of thing with aggs right? | |
| 21:46:17 | dansmith | we needed member_of with and and or | |
| 21:46:33 | dansmith | and I meant prefilter above | |
| 21:46:37 | dansmith | or request filter | |
| 21:46:40 | dansmith | or whatever that guy called it | |
| 21:47:29 | dansmith | the granular request stuff is another similar example, where we try to use what we think is just clean resource requests for actual nova stuff and realize we need this giant complex syntax instead | |
| 21:47:54 | dansmith | all that could be developed in two separate rooms for sure, just like multiattach or multiple host bindings | |
| 21:48:05 | dansmith | and hot damn, in a few years we'll be golden | |
| 21:49:31 | dansmith | mriedem: by the way, I wonder if the object action rpc methods to conductor ought to be long_rpc_timeouts | |
| 21:49:31 | melwitt | yeah. in case it wasn't clear, the thing I care about is delivering stuff that operators and users need, that we know they need, and I don't see how becoming two separate groups helps that | |
| 21:49:56 | dansmith | mriedem: they really should never hang for a long time, but just piling up more because we time out, run the periodic again and generate more traffic is probably worse | |
| 21:50:18 | dansmith | mriedem: re: your save timeout | |
| 21:52:00 | mriedem | dansmith: it seems something weird happened with the servicegroup in that failure | |
| 21:52:23 | dansmith | yeah, not related to your actual thing, but just thinking of what that reminds me of, which is conductor is overwhelmed | |
| 21:58:56 | mriedem | dansmith: also, i was talking with efried about kevin's exclusive trait thing, and guess what https://review.openstack.org/#/c/593475/ | |
| 21:58:59 | mriedem | it's already been proposed :) | |
| 22:00:21 | dansmith | well, having not read it and skimmed the -1, I'm assuming it's for the same reason I think it's a non-starter | |
| 22:03:41 | mriedem | encoding metadata in a trait name | |
| 22:05:56 | dansmith | and it's only one special prefix, | |
| 22:05:57 | dansmith | which means one class | |
| 22:06:46 | mriedem | CUSTOM_INTEL_FOR_SERIOUS_WORKLOADS | |
| 22:06:52 | mriedem | i can see it now | |
| 22:08:39 | efried | POST /traits/CUSTOM_INTEL_FOR_SERIOUS_WORKLOADS | |
| 22:08:39 | efried | { 'name': 'CUSTOM_INTEL_FOR_SERIOUS_WORKLOADS, | |
| 22:08:40 | efried | 'required': true, | |
| 22:08:40 | efried | 'allowed_user_ids': [...], | |
| 22:08:40 | efried | 'allowed_project_ids': [...], | |
| 22:08:40 | efried | ... | |
| 22:08:40 | efried | } | |
| 22:08:57 | mriedem | queue jay vomit | |
| 22:09:01 | mriedem | *cue | |
| 22:09:07 | efried | we could do the same thing with aggregates | |
| 22:09:27 | mriedem | dansmith: re granular, we could still do POST queries.... | |
| 22:09:29 | mriedem | just saying | |
| 22:10:49 | mriedem | granular request group syntax is likely something that could benefit from some kind of flavor extra specs validate api | |
| 22:13:26 | efried | no argument there | |