| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-05 | |||
| 18:05:58 | stephenfin | sean-k-mooney: One comment on https://review.opendev.org/#/c/674072/ | |
| 18:16:35 | stephenfin | sean-k-mooney: also on https://review.opendev.org/#/c/687957/ | |
| 18:16:59 | stephenfin | and with that... | |
| 18:35:49 | sean-k-mooney | stephenfin: yep ill take a look and adress them today. just back form dinner so ill work on that shortly | |
| 18:50:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add resource provider allocation unset example to troubleshooting doc https://review.opendev.org/696582 | |
| 19:17:00 | efried | mriedem, smcginnis, jungleboyj: Any ideas on how to make sure we've tested https://review.opendev.org/697134 through os-brick? | |
| 19:19:39 | mriedem | efried: i commented but tl;dr i can't think of anything | |
| 19:20:01 | mriedem | this is assuming that all of os-brick is using privsep and not any rootwrap helpers for some of the backends in os-brick somewhere | |
| 19:20:13 | mriedem | we can see that it's fine for lvm and rbd though | |
| 19:20:38 | mriedem | the libvirt driver uses os-brick, the ceph job uses the rbd volume type and backend, and all other jobs are using lvm | |
| 19:21:09 | mriedem | if there is something else in brick not using privsep then we likely won't see it here, but once we land this i'd think 3rd party CIs in cinder would be failing and alerting to cinder team | |
| 19:21:34 | mriedem | though if cinder's 3rd party ci's are anything like ours, there are probably many that fail often so people don't pay attention | |
| 19:22:43 | mriedem | the quobyte ci might be a decent indication and it failed but it also didn't report anything | |
| 19:22:50 | mriedem | kaisers: ^ | |
| 19:24:12 | efried | thanks for the look mriedem | |
| 19:25:51 | mriedem | looking at the hyperv logs, and i noticed this the other day as well, they seem to hit https://review.opendev.org/#/q/topic:bug/1748697+(status:open+OR+status:merged) quite a bit | |
| 19:26:06 | mriedem | doing cold migration testing and the scheduler picks the current host which triggers a reschedule | |
| 19:26:41 | efried | ttx: If you concur, my philosophy is that we should merge this now to give some soak time for 3pCIs to uncover issues. We can always revert if we need to. | |
| 19:26:51 | mriedem | ack: Instance rollback performed due to: Unable to migrate instance (ebcd3e84-9ee7-455d-be7a-849338e9781e) to current host (n-h1-697134-2). | |
| 19:26:51 | mriedem | 2019-12-05 11:43:22.672 2556 2225548250976 MainThread ERROR nova.compute.manager [req-3e316bf9-f63f-4388-8da2-108d7781d6f8 14ffcc57bb934290aa4ec7ba0ff79a98 6fec721e534d443f9aa4123acb1d9d20 - default default] [instance: ebcd3e84-9ee7-455d-be7a-849338e9781e] Error: Instance rollback performed due to: Unable to migrate instance (ebcd3e84-9ee7-455d-be7a-849338e9781e) to current host (n-h1-697134-2).: nova.exception.InstanceFaultR | |
| 19:28:24 | jungleboyj | efried: I concur with mriedem 's analysis. | |
| 19:28:37 | efried | thanks jungleboyj | |
| 19:28:46 | jungleboyj | Unfortunately the CIs aren't super reliable so what kind of flags it will send up I don't know. | |
| 19:29:04 | jungleboyj | It should be covered by the check for RBD and LVM though. | |
| 19:29:08 | jungleboyj | So, that is a good start. | |
| 19:29:34 | mriedem | hemna_ should be able to just tell us if there is anything in os-brick that isn't using privsep | |
| 19:30:16 | jungleboyj | I feel like we got things moved over. | |
| 19:31:24 | hemna_ | I don't think any of the connectors were using rootwrap, but I'd have to look closely. | |
| 19:37:09 | jungleboyj | hemna_: Have I mentioned it is good to have you back? | |
| 19:39:02 | efried | mriedem: I don't know if we're talking about different things that could (n)ever work by filtering GET /a_c with (required or forbidden) COMPUTE_SAME_HOST_COLD_MIGRATE. I tried to explain myself again. | |
| 19:40:36 | efried | We don't have "conditional traits". We can't say "exclude if rp_uuid=$self and !COMPUTE_SAME_HOST_COLD_MIGRATE". Without that conditional, saying required=!COMPUTE_SAME_HOST_COLD_MIGRATE will never, ever make sense. Ever. | |
| 19:43:30 | mriedem | i replied, think i got what you're saying | |
| 19:43:36 | hemna_ | :) thanks man | |
| 19:46:23 | sean-k-mooney | efried: well required=!COMPUTE_SAME_HOST_COLD_MIGRATE litrally would mean a host that cant support same host cold migration but ya we can tmake that require ment condionaly only if rp_uuid is a specific value and ignore it otherwsie for the same allocation candiate | |
| 19:46:39 | hemna_ | from what I can tell, most stuff works through the Executor, which defaults to privsep, or the privileged/rootwrap.execute, which also defaults to privsep | |
| 19:46:39 | efried | right | |
| 19:47:15 | efried | mriedem: before I dive back into the nova side, is it using the trait? | |
| 19:47:26 | efried | (for the post-filter, that is) | |
| 19:48:17 | mriedem | https://review.opendev.org/#/c/676022/ is a backportable workaround that is not, but if we don't care about backportability for the fix, then i will abandon. the patch that comes after that https://review.opendev.org/#/c/695220/ uses the trait yes and has a functional test to show it working. | |
| 19:48:56 | mriedem | well, it's using a custom trait for now since the os-traits patch isn't released | |
| 19:48:58 | mriedem | same idea | |
| 19:49:14 | efried | okay, I didn't see a Depends-On at the top so that's what I was asking. Will dig in. | |
| 19:49:46 | mriedem | i left a draft proposed comment for the os-traits change if it will work for you | |
| 19:50:04 | efried | I'm responding. But maybe easier to discuss here... | |
| 19:50:17 | efried | That's a reasonable walkthrough of a specific scenario, but my reasoning was much simpler than that. | |
| 19:50:38 | mriedem | just tell me what to write | |
| 19:51:05 | efried | let me talk this out first | |
| 19:51:49 | efried | If I'm on a host and doing a cold migrate, the condition I want to express is "if SHCM is not supported by $me, exclude $me from the results". | |
| 19:52:06 | mriedem | correct | |
| 19:52:10 | efried | but we don't have any placement way to say that | |
| 19:52:57 | mriedem | not unless there is some !in_tree logic in GET /a_c i'm not aware of | |
| 19:52:59 | sean-k-mooney | right we would have to do too quires. one to determin if we support X and then a second with either this host included or excluded | |
| 19:53:34 | efried | Exactly, point being that the GET /a_c query wouldn't involve the SHCM trait; it would involve the host (RP (tree)) we want to exclude. | |
| 19:54:19 | efried | ...assuming !in_tree was a thing | |
| 19:54:47 | sean-k-mooney | i think that was aded but the in paramter of traits was not | |
| 19:54:54 | efried | we were going to do something like that for CERNs server groups case, right? | |
| 19:55:05 | mriedem | i don't remember | |
| 19:55:06 | sean-k-mooney | for anti affinity i think | |
| 19:55:23 | mriedem | there is no !in_tree semantics documented in the GET /a_c reference | |
| 19:55:56 | efried | nor in the gabbits, so yeah, I don't think that's a thing. | |
| 19:56:12 | sean-k-mooney | we may have drop the propsoal since we did not need it in train | |
| 19:57:24 | efried | regardless, the point is that we don't expect that trait to ever make sense in a GET /a_c query because we have no way to express the SAME_HOST part of it in GET /a_c | |
| 19:57:29 | mriedem | as for anti-affinity....there is a member_of=!in, so for server groups you could create a provider aggregate for any host a member of that group is on and then when scheduling others you'd say !in for any other provider aggregate that represents other hosts in the group that has members, maybe - or not, it's all confusing and a tangent to what i'm trying to solve | |
| 19:57:45 | efried | it's, like, a tautological impossibility. | |
| 19:58:46 | mriedem | well, saying "no never because we can't ever" and "because GET /a_c doesn't have some semantics today" is a conflict, but sure | |
| 19:58:50 | mriedem | i agree today it's not possible | |
| 19:59:33 | mriedem | anyway, that's kind of why i didn't really want to put a bunch of words in a comment in os-traits about the state of how things are done between nova and placement today | |
| 19:59:38 | mriedem | it's super duper tight coupling | |
| 20:00:04 | efried | it's tight coupling between os-traits and placement, not anything and nova. | |
| 20:00:25 | mriedem | efried: if you're asking for a comment about why the trait can't be used in GET /a_c then that seems like something that should go in the nova change that's using the trait | |
| 20:00:40 | mriedem | because that is the code that would eventually be changed, not where the trait lives in os-traits | |
| 20:00:55 | mriedem | i mean, if you're asking about a reminder to yourself if someone tries to do this some day | |
| 20:01:00 | efried | that would also be good. | |
| 20:02:08 | eandersson | This RabbitMQ issue is so weird. | |
| 20:02:22 | eandersson | The compute queues only are getting messages, but they are staying as un'ackd. | |
| 20:02:57 | sean-k-mooney | eandersson: have you restarted the nova compute agent | |
| 20:03:00 | eandersson | We suspect RabbitMQ still, but it's weird that it is only affectin compute. | |
| 20:03:21 | eandersson | Yea sean-k-mooney they are kept as unac'kd even after restarting the agent. So for sure a RabbitMQ issue, but don't know what triggers it. | |
| 20:03:29 | efried | mriedem: responded x2 | |
| 20:03:33 | mriedem | efried: so to recap, what i'd like to do is (1) remove the mention from the os-traits commit message about eventually doing GET /a_c or whatever, (2) *not* leave a coupled comment next to the trait, and instead (3) put a comment about that in the nova patch https://review.opendev.org/#/c/695220/ | |
| 20:03:57 | efried | mriedem: (2) do leave a comment, (3) do put a comment. | |
| 20:04:00 | efried | They're different comments | |
| 20:04:27 | sean-k-mooney | eandersson: oh ok so its not the issue where the compute agent does not recreate the topic queue if the rabbmq server goes offline | |
| 20:05:11 | sean-k-mooney | eandersson: im not sure what that issue is then | |
| 20:05:55 | efried | (2) I suggested in the os-traits patch, just talks about GET /a_c. (3) should have your specific libvirt scenario described | |
| 20:07:56 | openstackgerrit | Matt Riedemann proposed openstack/os-traits master: Add COMPUTE_SAME_HOST_COLD_MIGRATE trait https://review.opendev.org/666604 | |
| 20:08:06 | mriedem | done | |
| 20:09:35 | mriedem | artom: dansmith: shall i just abandon the backportable workaround option that is https://review.opendev.org/#/c/676022/ and focus on the trait-based master-only solution to the extremely latent bug? | |
| 20:09:47 | mriedem | if it's this latent people obviously don't care enough so backports are a meh to me at this point | |
| 20:09:57 | efried | mriedem: +2, thanks for bearing with me. | |
| 20:10:05 | mriedem | same | |
| 20:12:35 | dansmith | mriedem: sounds like that would sidestep the whole argument, so.. yes? | |
| 20:13:33 | efried | It's got a bug heat of 32 fwiw. | |
| 20:13:50 | efried | Multiple people commented on it over the almost-2y it's been open. | |
| 20:14:02 | efried | so having a backportable solution doesn't seem like a terrible idea to me. | |
| 20:14:16 | efried | but I need to relook at how ugly it is. | |
| 20:14:45 | mriedem | i've abandoned | |
| 20:14:51 | mriedem | people can cherry pick it downstream if they really need it | |
| 20:14:58 | efried | okay | |
| 20:15:38 | efried | mriedem: I guess you're going to rebase and clean up the other one now, so I should wait to review? | |