| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-06 | |||
| 15:43:20 | mriedem | due to some timing issue | |
| 15:43:34 | dansmith | and maybe not at all at gate-level scale | |
| 15:44:39 | mriedem | would be nice to have some company with a stress test lab kick around either of those to see what falls out | |
| 15:44:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete instance_group_member records from API DB during archive https://review.openstack.org/588943 | |
| 15:45:07 | mriedem | but i'm pretty sure at this point in my time working on openstack, no company has a stress test lab that shares results publicly :) | |
| 15:45:32 | mriedem | if we don't see it in the gate, | |
| 15:45:44 | mriedem | we'll see it 18-24 months from now when someone actually upgrades to use it and finds problems | |
| 15:45:50 | efried | right | |
| 15:46:02 | jaypipes | mriedem: 36-64 months from now. | |
| 15:46:11 | mriedem | well, cern would be the first to hit it probably | |
| 15:46:13 | efried | But even that is a big "maybe". | |
| 15:46:56 | mriedem | i also haven't had the time to read cdent's write up on this either yet | |
| 15:47:09 | efried | mriedem: TL;DR we're calling _update twice every periodic. | |
| 15:47:16 | mriedem | yeah i knew that much :) | |
| 15:47:24 | efried | which calls all the placement things to refresh the cache etc. | |
| 15:47:33 | efried | that's really all there is to it. | |
| 15:47:42 | cdent | mriedem: there's not much more than that what efried says, must more words to indicate the code path | |
| 15:47:43 | efried | Do we need both calls for some reason? | |
| 15:48:05 | mriedem | also https://review.openstack.org/#/c/520024/ | |
| 15:48:20 | mriedem | given ^ you should probably see if the ovh gang is interested in testing out either of those changes | |
| 15:48:50 | mriedem | when did we start doing the 2 calls? | |
| 15:48:56 | mriedem | b/c ovh might not have a region running that yet | |
| 15:49:30 | cdent | >18 months ago | |
| 15:49:39 | cdent | it was there when I did the first version of the post | |
| 15:50:26 | efried | oh look, https://review.openstack.org/#/c/520024 is exactly the same as https://review.openstack.org/#/c/588091/ | |
| 15:50:29 | mriedem | looking at ^ from ovh they opted to leave in the _update call at the end | |
| 15:50:55 | mriedem | the +1 from minho on that was also because they had done a duplicate of the same patch | |
| 15:51:03 | openstackgerrit | Eric Fried proposed openstack/nova master: Update resources once in update_available_resource https://review.openstack.org/520024 | |
| 15:52:44 | mriedem | ok i have voted | |
| 15:52:54 | openstackgerrit | Chen proposed openstack/nova master: Trivial fix on migration doc https://review.openstack.org/589028 | |
| 15:52:55 | mriedem | if it's a coin toss, go with what ovh is already using | |
| 15:53:13 | mriedem | and drop the duplicate change from efried and coalesce on the ovh patch | |
| 15:53:26 | efried | Note that there's also https://review.openstack.org/#/c/588094/ which is failing tox (expected because I haven't updated those tests yet) but also live migration consistently. | |
| 15:53:53 | efried | mriedem: The coalesce is a no-op; the extras in mine are only there because of the reshaper series. I've abandoned it. | |
| 15:54:05 | mriedem | maciejjozefczyk: are you guys running with this in production now? https://review.openstack.org/#/c/520024/ | |
| 15:54:16 | mriedem | maciejjozefczyk: any side effects or issues with that patch? | |
| 16:00:06 | mriedem | jaypipes: has cfriesen talked with you about how they account for hosting shared and pinned cpus on the same host by making VCPU inventory a fraction? | |
| 16:00:37 | mriedem | i assume it would have come up during https://review.openstack.org/#/c/555081/ | |
| 16:01:25 | jaypipes | mriedem: nope. | |
| 16:01:46 | jaypipes | mriedem: and I'm 100% against making amount a non-integer value. | |
| 16:03:55 | cdent | It was briefly glossed over in Dublin, but the response then was "placement can't do that" | |
| 16:04:05 | cdent | or maybe s/can't/won't/ | |
| 16:04:18 | dansmith | I'm also 100% against that :) | |
| 16:04:24 | mriedem | i'm not suggesting we do that, | |
| 16:04:35 | mriedem | but i thought there was an alternative way to model it via nested providers, or some other kind of inventory | |
| 16:04:38 | mriedem | PCPUs? | |
| 16:05:37 | mriedem | looks like that is the proposal in the spec | |
| 16:05:51 | dansmith | we talked about making dedicated cpus a different inventory item in placement | |
| 16:05:53 | dansmith | if that's what youmean | |
| 16:05:55 | mriedem | yes | |
| 16:06:14 | mriedem | there is just a shit load of code in starlingx to deal with shared and pinned cpus on the same host, | |
| 16:06:26 | mriedem | and i'm mostly at the point of glossing over all of it and just saying, we have a spec for this | |
| 16:07:07 | mriedem | at the end of my diff dive i want to be able to give a sort of tl;dr on the major changes | |
| 16:07:33 | mriedem | which at this point is i think just shared/pinned on same host, live resize (cpu only), and l3 cache partitioning | |
| 16:07:47 | mriedem | there are a lot of other things, but those are the big ones that affect the entire stack | |
| 16:08:57 | mriedem | if you're all lucky, i might even make some m'fing charts! | |
| 16:17:59 | mriedem | surprisingly i don't see the tpm stuff in her | |
| 16:18:00 | mriedem | *here | |
| 16:19:22 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add descriptions for rebuild https://review.openstack.org/588931 | |
| 16:21:24 | mdbooth | lyarwood: https://review.openstack.org/#/c/587071/ I think that's the wrong build | |
| 16:22:18 | mdbooth | lyarwood: I won't be bowled over with surprise if I'm wrong, though. | |
| 16:29:44 | mdbooth | lyarwood: Hmm, looks like compute calls build_instances directly on reschedule? So... I could be wrong. | |
| 16:46:54 | mdbooth | mriedem: I assume it's not possible to 'demote' an attachment to a reservation? | |
| 16:50:11 | mriedem | first, that sounds borderline genocidal | |
| 16:50:22 | mriedem | second, i don't know what that means | |
| 16:50:32 | mriedem | an empty volume attachment reserves the volume | |
| 16:50:54 | mriedem | actually i think even a volume attachment that has a host connector is still not considering the volume as in-use, | |
| 16:51:02 | mriedem | that's why we have to call the 'complete' action on the attachment | |
| 16:51:06 | mriedem | to make the volume to in-use | |
| 16:51:20 | mdbooth | mriedem: Right. I'm just thinking of lyarwood's bug. The issue, IIUC, is that we create a 'reservation', which is an empty attachment, then the compute turns it into a real attachment, right? | |
| 16:51:29 | mdbooth | And the only way to get rid of that is to delete it. | |
| 16:51:46 | mriedem | depends on where we fail, | |
| 16:52:01 | mriedem | if we didn't get to the point of calling the complete action on the attachment, the volume is not in-use | |
| 16:52:02 | mdbooth | Sure, but if it's after we hydrated the attachment, this is how to rollback. | |
| 16:52:35 | openstackgerrit | Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598 | |
| 16:52:35 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 16:52:36 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648 | |
| 16:52:37 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 16:52:39 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: update_from_provider_tree w/reshape https://review.openstack.org/585049 | |
| 16:52:40 | openstackgerrit | Eric Fried proposed openstack/nova master: Compute: Handle reshaped provider trees https://review.openstack.org/576236 | |
| 16:52:46 | mriedem | i believe i talked to myself at length on an earlier PS on the review about how we could just simply not delete the attachment before rescheduling | |
| 16:53:08 | mdbooth | I'm pretty sure the answer's no, but I'm just wondering if there's any direct opposite to the compute's action which would cause the attachment to go back to being just a 'reservation' | |
| 16:53:21 | mdbooth | Ok, sounds like you already did that dance. | |
| 16:54:03 | mriedem | https://review.openstack.org/#/c/587071/3/nova/compute/manager.py@1631 | |
| 16:55:41 | mriedem | tl;dr is i think this is no worse than what happened during reschedule *before* the attachments stuff, | |
| 16:55:57 | mriedem | we could go either way and there are pros/cons both ways | |
| 16:56:38 | mriedem | e.g. if compute doesn't delete the attachment, conductor would have to when we exhausted retries | |
| 16:58:23 | mdbooth | mriedem: Nice writeup, thanks. | |
| 17:00:04 | mriedem | if anyone knows about or cares about uefi instances, this might be a bug fix we need in nova https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-f4019782d93a196a0d026479e6aa61b1R8514 | |
| 17:00:28 | mriedem | ah looky here https://bugs.launchpad.net/nova/+bug/1785123 | |
| 17:00:28 | openstack | Launchpad bug 1785123 in OpenStack Compute (nova) "UEFI NVRAM lost on cold migration or resize" [Undecided,New] | |
| 17:23:33 | melwitt | . | |
| 17:24:02 | mdbooth | mriedem: I wonder if live migration handles that. | |
| 17:24:24 | mdbooth | mriedem: Pretty sure we care, btw. | |
| 17:53:00 | mriedem | mdbooth: they also have lvm thin pools; i looked up the patch to add that to nova and you nacked it saying we already have sparse lvms, | |
| 17:53:08 | mriedem | but sparse_logical_volumes was deprecated in rocky | |
| 17:55:38 | mriedem | they have quite a bit of support for lvm, not sure if that's just for their 1-2 node configs, and then use ceph for their 100 node deployment or what | |
| 17:57:09 | mriedem | and i'm done with the diff \o/ | |
| 18:04:42 | penick | I saw the diff and backed away | |
| 18:16:58 | openstackgerrit | Radoslav Gerganov proposed openstack/nova stable/queens: Reload oslo_context after calling monkey_patch() https://review.openstack.org/589249 | |