| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-05 | |||
| 19:26:41 | efried | ttx: If you concur, my philosophy is that we should merge this now to give some soak time for 3pCIs to uncover issues. We can always revert if we need to. | |
| 19:26:51 | mriedem | ack: Instance rollback performed due to: Unable to migrate instance (ebcd3e84-9ee7-455d-be7a-849338e9781e) to current host (n-h1-697134-2). | |
| 19:26:51 | mriedem | 2019-12-05 11:43:22.672 2556 2225548250976 MainThread ERROR nova.compute.manager [req-3e316bf9-f63f-4388-8da2-108d7781d6f8 14ffcc57bb934290aa4ec7ba0ff79a98 6fec721e534d443f9aa4123acb1d9d20 - default default] [instance: ebcd3e84-9ee7-455d-be7a-849338e9781e] Error: Instance rollback performed due to: Unable to migrate instance (ebcd3e84-9ee7-455d-be7a-849338e9781e) to current host (n-h1-697134-2).: nova.exception.InstanceFaultR | |
| 19:28:24 | jungleboyj | efried: I concur with mriedem 's analysis. | |
| 19:28:37 | efried | thanks jungleboyj | |
| 19:28:46 | jungleboyj | Unfortunately the CIs aren't super reliable so what kind of flags it will send up I don't know. | |
| 19:29:04 | jungleboyj | It should be covered by the check for RBD and LVM though. | |
| 19:29:08 | jungleboyj | So, that is a good start. | |
| 19:29:34 | mriedem | hemna_ should be able to just tell us if there is anything in os-brick that isn't using privsep | |
| 19:30:16 | jungleboyj | I feel like we got things moved over. | |
| 19:31:24 | hemna_ | I don't think any of the connectors were using rootwrap, but I'd have to look closely. | |
| 19:37:09 | jungleboyj | hemna_: Have I mentioned it is good to have you back? | |
| 19:39:02 | efried | mriedem: I don't know if we're talking about different things that could (n)ever work by filtering GET /a_c with (required or forbidden) COMPUTE_SAME_HOST_COLD_MIGRATE. I tried to explain myself again. | |
| 19:40:36 | efried | We don't have "conditional traits". We can't say "exclude if rp_uuid=$self and !COMPUTE_SAME_HOST_COLD_MIGRATE". Without that conditional, saying required=!COMPUTE_SAME_HOST_COLD_MIGRATE will never, ever make sense. Ever. | |
| 19:43:30 | mriedem | i replied, think i got what you're saying | |
| 19:43:36 | hemna_ | :) thanks man | |
| 19:46:23 | sean-k-mooney | efried: well required=!COMPUTE_SAME_HOST_COLD_MIGRATE litrally would mean a host that cant support same host cold migration but ya we can tmake that require ment condionaly only if rp_uuid is a specific value and ignore it otherwsie for the same allocation candiate | |
| 19:46:39 | hemna_ | from what I can tell, most stuff works through the Executor, which defaults to privsep, or the privileged/rootwrap.execute, which also defaults to privsep | |
| 19:46:39 | efried | right | |
| 19:47:15 | efried | mriedem: before I dive back into the nova side, is it using the trait? | |
| 19:47:26 | efried | (for the post-filter, that is) | |
| 19:48:17 | mriedem | https://review.opendev.org/#/c/676022/ is a backportable workaround that is not, but if we don't care about backportability for the fix, then i will abandon. the patch that comes after that https://review.opendev.org/#/c/695220/ uses the trait yes and has a functional test to show it working. | |
| 19:48:56 | mriedem | well, it's using a custom trait for now since the os-traits patch isn't released | |
| 19:48:58 | mriedem | same idea | |
| 19:49:14 | efried | okay, I didn't see a Depends-On at the top so that's what I was asking. Will dig in. | |
| 19:49:46 | mriedem | i left a draft proposed comment for the os-traits change if it will work for you | |
| 19:50:04 | efried | I'm responding. But maybe easier to discuss here... | |
| 19:50:17 | efried | That's a reasonable walkthrough of a specific scenario, but my reasoning was much simpler than that. | |
| 19:50:38 | mriedem | just tell me what to write | |
| 19:51:05 | efried | let me talk this out first | |
| 19:51:49 | efried | If I'm on a host and doing a cold migrate, the condition I want to express is "if SHCM is not supported by $me, exclude $me from the results". | |
| 19:52:06 | mriedem | correct | |
| 19:52:10 | efried | but we don't have any placement way to say that | |
| 19:52:57 | mriedem | not unless there is some !in_tree logic in GET /a_c i'm not aware of | |
| 19:52:59 | sean-k-mooney | right we would have to do too quires. one to determin if we support X and then a second with either this host included or excluded | |
| 19:53:34 | efried | Exactly, point being that the GET /a_c query wouldn't involve the SHCM trait; it would involve the host (RP (tree)) we want to exclude. | |
| 19:54:19 | efried | ...assuming !in_tree was a thing | |
| 19:54:47 | sean-k-mooney | i think that was aded but the in paramter of traits was not | |
| 19:54:54 | efried | we were going to do something like that for CERNs server groups case, right? | |
| 19:55:05 | mriedem | i don't remember | |
| 19:55:06 | sean-k-mooney | for anti affinity i think | |
| 19:55:23 | mriedem | there is no !in_tree semantics documented in the GET /a_c reference | |
| 19:55:56 | efried | nor in the gabbits, so yeah, I don't think that's a thing. | |
| 19:56:12 | sean-k-mooney | we may have drop the propsoal since we did not need it in train | |
| 19:57:24 | efried | regardless, the point is that we don't expect that trait to ever make sense in a GET /a_c query because we have no way to express the SAME_HOST part of it in GET /a_c | |
| 19:57:29 | mriedem | as for anti-affinity....there is a member_of=!in, so for server groups you could create a provider aggregate for any host a member of that group is on and then when scheduling others you'd say !in for any other provider aggregate that represents other hosts in the group that has members, maybe - or not, it's all confusing and a tangent to what i'm trying to solve | |
| 19:57:45 | efried | it's, like, a tautological impossibility. | |
| 19:58:46 | mriedem | well, saying "no never because we can't ever" and "because GET /a_c doesn't have some semantics today" is a conflict, but sure | |
| 19:58:50 | mriedem | i agree today it's not possible | |
| 19:59:33 | mriedem | anyway, that's kind of why i didn't really want to put a bunch of words in a comment in os-traits about the state of how things are done between nova and placement today | |
| 19:59:38 | mriedem | it's super duper tight coupling | |
| 20:00:04 | efried | it's tight coupling between os-traits and placement, not anything and nova. | |
| 20:00:25 | mriedem | efried: if you're asking for a comment about why the trait can't be used in GET /a_c then that seems like something that should go in the nova change that's using the trait | |
| 20:00:40 | mriedem | because that is the code that would eventually be changed, not where the trait lives in os-traits | |
| 20:00:55 | mriedem | i mean, if you're asking about a reminder to yourself if someone tries to do this some day | |
| 20:01:00 | efried | that would also be good. | |
| 20:02:08 | eandersson | This RabbitMQ issue is so weird. | |
| 20:02:22 | eandersson | The compute queues only are getting messages, but they are staying as un'ackd. | |
| 20:02:57 | sean-k-mooney | eandersson: have you restarted the nova compute agent | |
| 20:03:00 | eandersson | We suspect RabbitMQ still, but it's weird that it is only affectin compute. | |
| 20:03:21 | eandersson | Yea sean-k-mooney they are kept as unac'kd even after restarting the agent. So for sure a RabbitMQ issue, but don't know what triggers it. | |
| 20:03:29 | efried | mriedem: responded x2 | |
| 20:03:33 | mriedem | efried: so to recap, what i'd like to do is (1) remove the mention from the os-traits commit message about eventually doing GET /a_c or whatever, (2) *not* leave a coupled comment next to the trait, and instead (3) put a comment about that in the nova patch https://review.opendev.org/#/c/695220/ | |
| 20:03:57 | efried | mriedem: (2) do leave a comment, (3) do put a comment. | |
| 20:04:00 | efried | They're different comments | |
| 20:04:27 | sean-k-mooney | eandersson: oh ok so its not the issue where the compute agent does not recreate the topic queue if the rabbmq server goes offline | |
| 20:05:11 | sean-k-mooney | eandersson: im not sure what that issue is then | |
| 20:05:55 | efried | (2) I suggested in the os-traits patch, just talks about GET /a_c. (3) should have your specific libvirt scenario described | |
| 20:07:56 | openstackgerrit | Matt Riedemann proposed openstack/os-traits master: Add COMPUTE_SAME_HOST_COLD_MIGRATE trait https://review.opendev.org/666604 | |
| 20:08:06 | mriedem | done | |
| 20:09:35 | mriedem | artom: dansmith: shall i just abandon the backportable workaround option that is https://review.opendev.org/#/c/676022/ and focus on the trait-based master-only solution to the extremely latent bug? | |
| 20:09:47 | mriedem | if it's this latent people obviously don't care enough so backports are a meh to me at this point | |
| 20:09:57 | efried | mriedem: +2, thanks for bearing with me. | |
| 20:10:05 | mriedem | same | |
| 20:12:35 | dansmith | mriedem: sounds like that would sidestep the whole argument, so.. yes? | |
| 20:13:33 | efried | It's got a bug heat of 32 fwiw. | |
| 20:13:50 | efried | Multiple people commented on it over the almost-2y it's been open. | |
| 20:14:02 | efried | so having a backportable solution doesn't seem like a terrible idea to me. | |
| 20:14:16 | efried | but I need to relook at how ugly it is. | |
| 20:14:45 | mriedem | i've abandoned | |
| 20:14:51 | mriedem | people can cherry pick it downstream if they really need it | |
| 20:14:58 | efried | okay | |
| 20:15:38 | efried | mriedem: I guess you're going to rebase and clean up the other one now, so I should wait to review? | |
| 20:16:01 | efried | yeah | |
| 20:16:14 | mriedem | i'm not going to get to that today but it won't be much different | |
| 20:16:25 | mriedem | probably crank it out in the wee morn | |
| 20:16:34 | efried | I didn't open any of the files, but the commit message at least needs an overhaul. | |
| 20:17:12 | mriedem | ubetcha | |
| 20:17:48 | efried | okay, let's see if I can think clearly enough to get back to rg/rp mappings... | |
| 20:18:46 | efried | mriedem: if you're feeling magnanimous, you could hit the bottom three patches here, fairly simple: https://review.opendev.org/#/c/696418/ | |
| 20:19:34 | mriedem | oh god request groups | |
| 20:19:45 | mriedem | i think melwitt said she wanted to learn more about this part of the code | |
| 20:19:49 | efried | hahaha | |
| 20:20:17 | efried | In my IRC client, melwitt shows up a nice grasshopper green. Coincidence? | |
| 20:21:23 | mriedem | as in the kung fu grasshopper pupil green? | |
| 20:21:45 | mriedem | or the "oh god request groups, i think i'm going to be sick" green | |
| 20:32:21 | mriedem | gibi: a few nits in https://review.opendev.org/#/c/696245/ if you want to start a FUP patch for that series | |
| 20:34:17 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use provider mappings from Placement (mostly) https://review.opendev.org/696992 | |
| 21:00:27 | efried | meeting now | |
| 21:14:10 | efried | mriedem: The vote in the meeting was unanimous: do a spec for the ironic owner filter | |