Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-05
16:14:27 mriedem blasphemy
16:31:02 dansmith not blasphemy, plus it means bacon more often
16:31:06 dansmith BACON AT EVERY MEAL (tm)
16:32:31 efried Okay, I updated the meeting agenda. I'm going to go be miserable somewhere for a while.
16:34:59 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Improve metadata server performance with large security groups https://review.opendev.org/695925
16:40:30 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: Improve metadata server performance with large security groups https://review.opendev.org/697517
16:41:31 melwitt mriedem: I went out on a limb and proposed a backport to queens that landed in rocky, for cleaning up placement consumer records that no longer have allocations https://review.opendev.org/697398 I had to modify it a lot because the code was in different files in queens
16:42:46 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Improve metadata server performance with large security groups https://review.opendev.org/697518
16:43:04 melwitt not sure whether that's going to fly but I figured I'd try because it's a pretty big unbounded table grow that otherwise has to be cleaned up manually
16:45:48 mriedem queens is EM so that's pretty low priority to address but fine to post if you're doing it for downstream anyway and if others want to use it
16:46:37 melwitt ah, I forgot it's EM. ok
16:46:49 melwitt understood
16:52:13 mriedem dansmith: how do you feel about backporting this? https://review.opendev.org/#/c/696953/
16:52:28 mriedem seems ok since it's tied to a long-standing bug but wanted a 2nd opinion
16:53:04 dansmith mriedem: yeah, I mean... as long as the old version has a sufficient oslo.messaging and the config knob, I can't really think of any reason not to do tose
16:53:05 dansmith *those
16:53:47 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Improve metadata server performance with large security groups https://review.opendev.org/697523
16:56:25 mriedem lyarwood: i'd be +2 on this https://review.opendev.org/#/c/696953/ but i'm not sure how pedantic you want to be about using the correct source commit hash
16:57:21 mriedem given train and stein had conflicts which you've already resolved i'm assuming you don't want to redo all of that
16:59:35 mriedem nvm i just +2ed
17:01:05 mriedem dansmith: you want to get this https://review.opendev.org/#/c/696339/ and this then https://review.opendev.org/696953
17:03:22 dansmith yar
17:06:31 lyarwood mriedem: sorry was on a call, I really don't care about the source commit hash with backports given the Change IDs tbh. That said I have no idea how I ended up using the wrong one for this.
17:06:34 lyarwood mriedem: and thanks
17:33:14 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: rt: only map compute node if we created it https://review.opendev.org/676463
17:54:48 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Add functional regression test for bug 1849409 https://review.opendev.org/690746
17:54:48 openstack bug 1849409 in OpenStack Compute (nova) queens "openstack server list --deleted --limit -1 hangs" [High,In progress] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem)
17:54:48 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Fix listing deleted servers with a marker https://review.opendev.org/690747
17:57:21 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Make '_IntegratedTestBase' subclass 'InstanceHelperMixin' https://review.opendev.org/689182
17:57:21 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Remove 'get_invalid_image' https://review.opendev.org/697454
17:57:22 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add unified '_build_server' helper function https://review.opendev.org/697537
17:57:22 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add unified '_(build|create)_flavor' helper functions https://review.opendev.org/697538
17:57:23 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add '_delete_server' to 'InstanceHelperMixin' https://review.opendev.org/697539
17:57:23 openstackgerrit Stephen Finucane proposed openstack/nova master: functional: Add '_create_server' helper https://review.opendev.org/697540
18:05:58 stephenfin sean-k-mooney: One comment on https://review.opendev.org/#/c/674072/
18:16:35 stephenfin sean-k-mooney: also on https://review.opendev.org/#/c/687957/
18:16:59 stephenfin and with that...
18:35:49 sean-k-mooney stephenfin: yep ill take a look and adress them today. just back form dinner so ill work on that shortly
18:50:42 openstackgerrit Matt Riedemann proposed openstack/nova master: Add resource provider allocation unset example to troubleshooting doc https://review.opendev.org/696582
19:17:00 efried mriedem, smcginnis, jungleboyj: Any ideas on how to make sure we've tested https://review.opendev.org/697134 through os-brick?
19:19:39 mriedem efried: i commented but tl;dr i can't think of anything
19:20:01 mriedem this is assuming that all of os-brick is using privsep and not any rootwrap helpers for some of the backends in os-brick somewhere
19:20:13 mriedem we can see that it's fine for lvm and rbd though
19:20:38 mriedem the libvirt driver uses os-brick, the ceph job uses the rbd volume type and backend, and all other jobs are using lvm
19:21:09 mriedem if there is something else in brick not using privsep then we likely won't see it here, but once we land this i'd think 3rd party CIs in cinder would be failing and alerting to cinder team
19:21:34 mriedem though if cinder's 3rd party ci's are anything like ours, there are probably many that fail often so people don't pay attention
19:22:43 mriedem the quobyte ci might be a decent indication and it failed but it also didn't report anything
19:22:50 mriedem kaisers: ^
19:24:12 efried thanks for the look mriedem
19:25:51 mriedem looking at the hyperv logs, and i noticed this the other day as well, they seem to hit https://review.opendev.org/#/q/topic:bug/1748697+(status:open+OR+status:merged) quite a bit
19:26:06 mriedem doing cold migration testing and the scheduler picks the current host which triggers a reschedule
19:26:41 efried ttx: If you concur, my philosophy is that we should merge this now to give some soak time for 3pCIs to uncover issues. We can always revert if we need to.
19:26:51 mriedem 2019-12-05 11:43:22.672 2556 2225548250976 MainThread ERROR nova.compute.manager [req-3e316bf9-f63f-4388-8da2-108d7781d6f8 14ffcc57bb934290aa4ec7ba0ff79a98 6fec721e534d443f9aa4123acb1d9d20 - default default] [instance: ebcd3e84-9ee7-455d-be7a-849338e9781e] Error: Instance rollback performed due to: Unable to migrate instance (ebcd3e84-9ee7-455d-be7a-849338e9781e) to current host (n-h1-697134-2).: nova.exception.InstanceFaultR
19:26:51 mriedem ack: Instance rollback performed due to: Unable to migrate instance (ebcd3e84-9ee7-455d-be7a-849338e9781e) to current host (n-h1-697134-2).
19:28:24 jungleboyj efried: I concur with mriedem 's analysis.
19:28:37 efried thanks jungleboyj
19:28:46 jungleboyj Unfortunately the CIs aren't super reliable so what kind of flags it will send up I don't know.
19:29:04 jungleboyj It should be covered by the check for RBD and LVM though.
19:29:08 jungleboyj So, that is a good start.
19:29:34 mriedem hemna_ should be able to just tell us if there is anything in os-brick that isn't using privsep
19:30:16 jungleboyj I feel like we got things moved over.
19:31:24 hemna_ I don't think any of the connectors were using rootwrap, but I'd have to look closely.
19:37:09 jungleboyj hemna_: Have I mentioned it is good to have you back?
19:39:02 efried mriedem: I don't know if we're talking about different things that could (n)ever work by filtering GET /a_c with (required or forbidden) COMPUTE_SAME_HOST_COLD_MIGRATE. I tried to explain myself again.
19:40:36 efried We don't have "conditional traits". We can't say "exclude if rp_uuid=$self and !COMPUTE_SAME_HOST_COLD_MIGRATE". Without that conditional, saying required=!COMPUTE_SAME_HOST_COLD_MIGRATE will never, ever make sense. Ever.
19:43:30 mriedem i replied, think i got what you're saying
19:43:36 hemna_ :) thanks man
19:46:23 sean-k-mooney efried: well required=!COMPUTE_SAME_HOST_COLD_MIGRATE litrally would mean a host that cant support same host cold migration but ya we can tmake that require ment condionaly only if rp_uuid is a specific value and ignore it otherwsie for the same allocation candiate
19:46:39 efried right
19:46:39 hemna_ from what I can tell, most stuff works through the Executor, which defaults to privsep, or the privileged/rootwrap.execute, which also defaults to privsep
19:47:15 efried mriedem: before I dive back into the nova side, is it using the trait?
19:47:26 efried (for the post-filter, that is)
19:48:17 mriedem https://review.opendev.org/#/c/676022/ is a backportable workaround that is not, but if we don't care about backportability for the fix, then i will abandon. the patch that comes after that https://review.opendev.org/#/c/695220/ uses the trait yes and has a functional test to show it working.
19:48:56 mriedem well, it's using a custom trait for now since the os-traits patch isn't released
19:48:58 mriedem same idea
19:49:14 efried okay, I didn't see a Depends-On at the top so that's what I was asking. Will dig in.
19:49:46 mriedem i left a draft proposed comment for the os-traits change if it will work for you
19:50:04 efried I'm responding. But maybe easier to discuss here...
19:50:17 efried That's a reasonable walkthrough of a specific scenario, but my reasoning was much simpler than that.
19:50:38 mriedem just tell me what to write
19:51:05 efried let me talk this out first
19:51:49 efried If I'm on a host and doing a cold migrate, the condition I want to express is "if SHCM is not supported by $me, exclude $me from the results".
19:52:06 mriedem correct
19:52:10 efried but we don't have any placement way to say that
19:52:57 mriedem not unless there is some !in_tree logic in GET /a_c i'm not aware of
19:52:59 sean-k-mooney right we would have to do too quires. one to determin if we support X and then a second with either this host included or excluded
19:53:34 efried Exactly, point being that the GET /a_c query wouldn't involve the SHCM trait; it would involve the host (RP (tree)) we want to exclude.
19:54:19 efried ...assuming !in_tree was a thing
19:54:47 sean-k-mooney i think that was aded but the in paramter of traits was not
19:54:54 efried we were going to do something like that for CERNs server groups case, right?
19:55:05 mriedem i don't remember
19:55:06 sean-k-mooney for anti affinity i think
19:55:23 mriedem there is no !in_tree semantics documented in the GET /a_c reference
19:55:56 efried nor in the gabbits, so yeah, I don't think that's a thing.
19:56:12 sean-k-mooney we may have drop the propsoal since we did not need it in train
19:57:24 efried regardless, the point is that we don't expect that trait to ever make sense in a GET /a_c query because we have no way to express the SAME_HOST part of it in GET /a_c
19:57:29 mriedem as for anti-affinity....there is a member_of=!in, so for server groups you could create a provider aggregate for any host a member of that group is on and then when scheduling others you'd say !in for any other provider aggregate that represents other hosts in the group that has members, maybe - or not, it's all confusing and a tangent to what i'm trying to solve
19:57:45 efried it's, like, a tautological impossibility.
19:58:46 mriedem well, saying "no never because we can't ever" and "because GET /a_c doesn't have some semantics today" is a conflict, but sure
19:58:50 mriedem i agree today it's not possible

Earlier   Later