| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 18:52:51 | donnyd | LOL mriedem | |
| 18:54:15 | artom | sean-k-mooney, AFAICT that calls down to https://github.com/openstack/nova/blob/master/nova/objects/pci_device.py#L373 | |
| 18:54:24 | artom | sean-k-mooney, which saves pci_devices, but not pci_requests | |
| 18:56:08 | artom | Does it even make sense for pci_requests to change for a live migration? | |
| 18:56:58 | sean-k-mooney | well the number and type of request wont change | |
| 18:57:21 | artom | The spec might? | |
| 18:58:04 | mriedem | i would hope that pci requests don't change for a live migration | |
| 18:58:09 | mriedem | otherwise we f'ed up somewhere in the data modeling | |
| 18:58:11 | sean-k-mooney | what do you mean | |
| 18:58:22 | mriedem | pci devices are the per-compute/instance thing that rack inventory and what's allocated | |
| 18:58:33 | mriedem | the pci requests should just be the host agnostic request | |
| 18:58:48 | sean-k-mooney | ya they more or less are | |
| 18:58:55 | sean-k-mooney | the can have an alisa | |
| 18:59:16 | sean-k-mooney | or my have tags like a neutorn physnet | |
| 18:59:23 | melwitt | donnyd, mriedem: I just asked penick and he said they've only done image cache warming once or twice on an ad hoc basis, for example for a 40G windows image that was needed on certain hypervisors. he said even still, downloads don't take too long (couple minutes at most) bc they always have their glance in the same datacenter. he also said they disable image checksums to avoid that slowness | |
| 18:59:26 | sean-k-mooney | or device type e.g. type-VF | |
| 18:59:36 | artom | sean-k-mooney, right, but that can't change during a live migration, right? | |
| 18:59:50 | sean-k-mooney | but the request does not have any info about the specifc device | |
| 18:59:58 | sean-k-mooney | artom: correct it cant | |
| 19:00:42 | melwitt | donnyd: I didn't notice if you mentioned how long is a long time in your scenario. is it a few minutes or like 20 minutes | |
| 19:00:42 | donnyd | melwitt: My downloads are in the same rack on a 10G network (compute side) and they are painfully slow for the gear that underpins them | |
| 19:02:50 | artom | So we seem fairly confident pci_requests can't change during a live migration. So back to our original problem, it should be find for a Claim to write them back to the instance | |
| 19:03:06 | donnyd | well the drive glance is hosted on is an nvme device with 4G/s in read speed and the machine its on has 40G networking... so I expected to get somewhere around 1/4 of that or about 1G/s (ish) in download speed | |
| 19:03:06 | artom | In the sense that, the SRIOV live migration code won't have changed them | |
| 19:03:10 | dansmith | right, which was my original thing.. why not just keep them? | |
| 19:03:21 | donnyd | the reality is 100M/s | |
| 19:03:25 | artom | dansmith, I was worried about them changing under us | |
| 19:04:23 | artom | So by keeping them we'd end up clobbering the new ones set by the SRIOV live migration code | |
| 19:04:33 | donnyd | If I could speed up the downloading part, i wouldn't even notice | |
| 19:04:53 | artom | Which is why I wanted to strongly to avoid touching them altogether | |
| 19:04:54 | donnyd | because a new image would be downloaded in 30-40 seconds | |
| 19:05:04 | dansmith | artom: we stash a copy of the requests to be applied int he migration context with other things.. if the sriov migration code is not playing nice with that, then it's wrong | |
| 19:05:08 | sean-k-mooney | artom: we wont clobber anything | |
| 19:05:18 | sean-k-mooney | but we will end up trying to claim pci device twice | |
| 19:05:42 | artom | sean-k-mooney, well no, dansmith's point was that we still skip the actual claiming (right?) | |
| 19:05:49 | artom | Just don't mess with any DB stuff | |
| 19:05:54 | sean-k-mooney | and then whe might not move them form claimed to allocated correctly and leak pci deivces | |
| 19:06:03 | dansmith | artom: I don't think I made that claim | |
| 19:06:05 | sean-k-mooney | e.g. once that are claimed for the instance but not allocated to it | |
| 19:06:07 | dansmith | artom: I might have asked that | |
| 19:06:19 | mriedem | "I don't think I made that claim" | |
| 19:06:25 | artom | We need to stop overloading "claim" >_< | |
| 19:06:39 | dansmith | I still don't understand why this needs to be different for cold and live migration, with respect to the accounting | |
| 19:06:57 | melwitt | donnyd: right... that does sound strange, but I'm definitely not that knowledgable about what is reasonable to expect there. I'll run those details by penick and find out if it rings a bell | |
| 19:06:58 | sean-k-mooney | claimed and allocated are states in the pci resouce tracker. | |
| 19:07:08 | sean-k-mooney | and claimes generally refers to the RT | |
| 19:07:14 | artom | dansmith, it doesn't :) But SRIOV live migration was implemented without it, so now we're in this mess | |
| 19:07:18 | sean-k-mooney | we have moved to using allocaiton to refer to placmeent | |
| 19:07:30 | dansmith | artom: yeah, sounds like that is the real problem here | |
| 19:08:04 | donnyd | I think the last time we dug into it, it had something to do with the requests library only being able to use one core, which in turn make sense because my controllers cores aren't super fast | |
| 19:08:24 | sean-k-mooney | i can proably make the sriov stuff work with move claims. but i would prefer to keep move claimes for cold migration | |
| 19:08:52 | sean-k-mooney | sriov migration did not use claims because we did not need to and live migration never used them before | |
| 19:09:32 | mriedem | so something was bolted on instead and now we have a mess | |
| 19:09:33 | donnyd | I thought the http.store option in glance would allow compute to grab from a http server (like apache or something), but i have no idea how to configure it | |
| 19:09:36 | mriedem | is the summary yeah? | |
| 19:09:51 | dansmith | rightm | |
| 19:09:52 | dansmith | that's the problem | |
| 19:09:53 | dansmith | "live migration is different so I can be more different" | |
| 19:10:05 | artom | mriedem, I think sean-k-mooney would object to the "bolted on" wording, but yeah | |
| 19:10:13 | sean-k-mooney | when we were first proposing sirov migration we were not planning to use move claime for numa migration | |
| 19:10:39 | artom | It's also on me for not having reviewed that spec/code | |
| 19:11:01 | sean-k-mooney | i am testing your code right now by the way | |
| 19:11:03 | artom | Could have said "hey let's use claims since we'll need them for NUMA LM anyways" | |
| 19:11:58 | sean-k-mooney | well we dont you could alos do the calimes the way we do but anyway should i start looking at how to convert the sriov code | |
| 19:12:08 | dansmith | which cores were reviewing that? | |
| 19:12:11 | dansmith | I don't think it was me, | |
| 19:12:17 | mriedem | jay and stephen | |
| 19:12:21 | dansmith | okay | |
| 19:12:46 | melwitt | donnyd: the last comment confuses me because compute downloads images over glance API (http), so what is the difference between that and the http.store option you mention, I wonder? | |
| 19:15:49 | dansmith | Seems like jay was actually in favor of managing the pci stuff like the rest of it: https://review.opendev.org/#/c/620115/ | |
| 19:16:00 | dansmith | "If we managed PCI devices using the same system we do for CPU, memory and local disk, then we could do the resource *claim* in the scheduler's claim_resources() " | |
| 19:16:29 | dansmith | actually, | |
| 19:16:37 | dansmith | maybe he means more placement-like by that | |
| 19:16:41 | sean-k-mooney | yes | |
| 19:17:24 | sean-k-mooney | he wanted to avoid the move cliams becuae he wante use to use placment as much as possible | |
| 19:17:59 | dansmith | I don't really see anywhere that he said that, but I don't doubt it | |
| 19:19:47 | artom | I really didn't want to start a witch hunt :( | |
| 19:19:52 | sean-k-mooney | that was my preference too by the way so im biased in recolection but i think if we need too i can adapt the sriov code to get the device form the move claim if needed | |
| 19:20:00 | dansmith | artom: no, just looking for context | |
| 19:20:10 | artom | Choices were made, I'm sure at the time with the available info they were optimal | |
| 19:20:14 | artom | dansmith, ack | |
| 19:20:43 | dansmith | artom: this is the problem with bolting on incremental "okay we'll justsupport live migration with pci if they're like this" kind of features | |
| 19:20:48 | dansmith | after you do that a couple times, you end up here | |
| 19:20:51 | donnyd | melwitt: well I am really just guessing because I don't understand how the http.store option works... I was hopeful that I could configure the filestore to save images in a particular location, and then have that exposed directly by apache (or something of the like) https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_drivers/http.py | |
| 19:21:58 | sean-k-mooney | well we evenually wanted to use the multiple port binding workflow for could migration and then move the sriov port claiming to always use the live migration flow when we did that | |
| 19:22:05 | artom | dansmith, isn't there a "law" about that? | |
| 19:22:15 | mriedem | ooo i'm scheduled for that tomorrow | |
| 19:22:19 | mriedem | we're sympatico | |
| 19:22:37 | artom | Products reproduce the communication structures of their organisations, or something like that? | |
| 19:22:53 | donnyd | melwitt: but i honestly don't know because I have never had to get that far into glance... in my normal uses cases... it just works | |
| 19:22:55 | artom | So us, being distributed, loose and non-cohesive, will produce features that are the same? | |
| 19:23:48 | artom | Aha, https://en.wikipedia.org/wiki/Conway%27s_law | |
| 19:24:10 | mriedem | "organizations that are a clusterfuck will produce software that is a clusterfuck" | |
| 19:24:11 | mriedem | got it | |
| 19:24:32 | artom | I mean, yes. :P | |
| 19:25:28 | sean-k-mooney | fyi i just migrated an instace with a numa toployg and an sriov macvtap port | |
| 19:25:33 | melwitt | donnyd: understood. I'm grepping around and also can't find the config option in glance about how to disable checksum verification for the image_cache. I'll need to chat with penick and get back to you. I'll ask him if he knows anything about http.store while I'm at it | |
| 19:25:38 | sean-k-mooney | i need to now look at the xml and see what happend | |
| 19:25:44 | sean-k-mooney | but it succeded | |
| 19:26:08 | donnyd | melwitt: much appreciate... or just any possible way to make it faster would be a massive help | |