| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 18:48:25 | artom | sean-k-mooney, do you recall if the SRIOV live migration code ever updated instance.pci_requests? | |
| 18:48:38 | artom | I don't see anything quickly glancing through your patches | |
| 18:48:56 | mriedem | donnyd: this is what i'm thinking of https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/imagebackend.py#L257 | |
| 18:49:18 | sean-k-mooney | we do update them when we allocate the claimed pci deivce i belive | |
| 18:50:14 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#compute.max_concurrent_disk_ops was the thing starlingx added | |
| 18:51:18 | donnyd | well I should give it a spin.. probably cut the "new image loaded so i will fail a bunch" down to almost nothing | |
| 18:51:25 | openstackgerrit | Merged openstack/python-novaclient master: Follow up for microversion 2.75 https://review.opendev.org/678473 | |
| 18:51:33 | sean-k-mooney | artom: i think it get saved here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7229-L7256 | |
| 18:51:42 | mriedem | donnyd: if you do let us know how it goes, | |
| 18:51:45 | sean-k-mooney | in post_live_migration_at_destination | |
| 18:51:53 | sean-k-mooney | so we only update it if the migration succeeds | |
| 18:51:54 | mriedem | donnyd: i also threw it in https://bugs.launchpad.net/nova/+bug/1838819/comments/7 so we don't forget docs about it | |
| 18:51:55 | openstack | Launchpad bug 1838819 in OpenStack Compute (nova) "Docs needed for tunables at large scale" [Undecided,Confirmed] | |
| 18:52:14 | sean-k-mooney | if it was reverted we do not modify the instace pci_requests | |
| 18:52:20 | mriedem | image cache docs in general would be great but stephenfin just refuses to support us documenting things | |
| 18:52:25 | mriedem | o-) | |
| 18:52:51 | donnyd | LOL mriedem | |
| 18:54:15 | artom | sean-k-mooney, AFAICT that calls down to https://github.com/openstack/nova/blob/master/nova/objects/pci_device.py#L373 | |
| 18:54:24 | artom | sean-k-mooney, which saves pci_devices, but not pci_requests | |
| 18:56:08 | artom | Does it even make sense for pci_requests to change for a live migration? | |
| 18:56:58 | sean-k-mooney | well the number and type of request wont change | |
| 18:57:21 | artom | The spec might? | |
| 18:58:04 | mriedem | i would hope that pci requests don't change for a live migration | |
| 18:58:09 | mriedem | otherwise we f'ed up somewhere in the data modeling | |
| 18:58:11 | sean-k-mooney | what do you mean | |
| 18:58:22 | mriedem | pci devices are the per-compute/instance thing that rack inventory and what's allocated | |
| 18:58:33 | mriedem | the pci requests should just be the host agnostic request | |
| 18:58:48 | sean-k-mooney | ya they more or less are | |
| 18:58:55 | sean-k-mooney | the can have an alisa | |
| 18:59:16 | sean-k-mooney | or my have tags like a neutorn physnet | |
| 18:59:23 | melwitt | donnyd, mriedem: I just asked penick and he said they've only done image cache warming once or twice on an ad hoc basis, for example for a 40G windows image that was needed on certain hypervisors. he said even still, downloads don't take too long (couple minutes at most) bc they always have their glance in the same datacenter. he also said they disable image checksums to avoid that slowness | |
| 18:59:26 | sean-k-mooney | or device type e.g. type-VF | |
| 18:59:36 | artom | sean-k-mooney, right, but that can't change during a live migration, right? | |
| 18:59:50 | sean-k-mooney | but the request does not have any info about the specifc device | |
| 18:59:58 | sean-k-mooney | artom: correct it cant | |
| 19:00:42 | donnyd | melwitt: My downloads are in the same rack on a 10G network (compute side) and they are painfully slow for the gear that underpins them | |
| 19:00:42 | melwitt | donnyd: I didn't notice if you mentioned how long is a long time in your scenario. is it a few minutes or like 20 minutes | |
| 19:02:50 | artom | So we seem fairly confident pci_requests can't change during a live migration. So back to our original problem, it should be find for a Claim to write them back to the instance | |
| 19:03:06 | artom | In the sense that, the SRIOV live migration code won't have changed them | |
| 19:03:06 | donnyd | well the drive glance is hosted on is an nvme device with 4G/s in read speed and the machine its on has 40G networking... so I expected to get somewhere around 1/4 of that or about 1G/s (ish) in download speed | |
| 19:03:10 | dansmith | right, which was my original thing.. why not just keep them? | |
| 19:03:21 | donnyd | the reality is 100M/s | |
| 19:03:25 | artom | dansmith, I was worried about them changing under us | |
| 19:04:23 | artom | So by keeping them we'd end up clobbering the new ones set by the SRIOV live migration code | |
| 19:04:33 | donnyd | If I could speed up the downloading part, i wouldn't even notice | |
| 19:04:53 | artom | Which is why I wanted to strongly to avoid touching them altogether | |
| 19:04:54 | donnyd | because a new image would be downloaded in 30-40 seconds | |
| 19:05:04 | dansmith | artom: we stash a copy of the requests to be applied int he migration context with other things.. if the sriov migration code is not playing nice with that, then it's wrong | |
| 19:05:08 | sean-k-mooney | artom: we wont clobber anything | |
| 19:05:18 | sean-k-mooney | but we will end up trying to claim pci device twice | |
| 19:05:42 | artom | sean-k-mooney, well no, dansmith's point was that we still skip the actual claiming (right?) | |
| 19:05:49 | artom | Just don't mess with any DB stuff | |
| 19:05:54 | sean-k-mooney | and then whe might not move them form claimed to allocated correctly and leak pci deivces | |
| 19:06:03 | dansmith | artom: I don't think I made that claim | |
| 19:06:05 | sean-k-mooney | e.g. once that are claimed for the instance but not allocated to it | |
| 19:06:07 | dansmith | artom: I might have asked that | |
| 19:06:19 | mriedem | "I don't think I made that claim" | |
| 19:06:25 | artom | We need to stop overloading "claim" >_< | |
| 19:06:39 | dansmith | I still don't understand why this needs to be different for cold and live migration, with respect to the accounting | |
| 19:06:57 | melwitt | donnyd: right... that does sound strange, but I'm definitely not that knowledgable about what is reasonable to expect there. I'll run those details by penick and find out if it rings a bell | |
| 19:06:58 | sean-k-mooney | claimed and allocated are states in the pci resouce tracker. | |
| 19:07:08 | sean-k-mooney | and claimes generally refers to the RT | |
| 19:07:14 | artom | dansmith, it doesn't :) But SRIOV live migration was implemented without it, so now we're in this mess | |
| 19:07:18 | sean-k-mooney | we have moved to using allocaiton to refer to placmeent | |
| 19:07:30 | dansmith | artom: yeah, sounds like that is the real problem here | |
| 19:08:04 | donnyd | I think the last time we dug into it, it had something to do with the requests library only being able to use one core, which in turn make sense because my controllers cores aren't super fast | |
| 19:08:24 | sean-k-mooney | i can proably make the sriov stuff work with move claims. but i would prefer to keep move claimes for cold migration | |
| 19:08:52 | sean-k-mooney | sriov migration did not use claims because we did not need to and live migration never used them before | |
| 19:09:32 | mriedem | so something was bolted on instead and now we have a mess | |
| 19:09:33 | donnyd | I thought the http.store option in glance would allow compute to grab from a http server (like apache or something), but i have no idea how to configure it | |
| 19:09:36 | mriedem | is the summary yeah? | |
| 19:09:51 | dansmith | rightm | |
| 19:09:52 | dansmith | that's the problem | |
| 19:09:53 | dansmith | "live migration is different so I can be more different" | |
| 19:10:05 | artom | mriedem, I think sean-k-mooney would object to the "bolted on" wording, but yeah | |
| 19:10:13 | sean-k-mooney | when we were first proposing sirov migration we were not planning to use move claime for numa migration | |
| 19:10:39 | artom | It's also on me for not having reviewed that spec/code | |
| 19:11:01 | sean-k-mooney | i am testing your code right now by the way | |
| 19:11:03 | artom | Could have said "hey let's use claims since we'll need them for NUMA LM anyways" | |
| 19:11:58 | sean-k-mooney | well we dont you could alos do the calimes the way we do but anyway should i start looking at how to convert the sriov code | |
| 19:12:08 | dansmith | which cores were reviewing that? | |
| 19:12:11 | dansmith | I don't think it was me, | |
| 19:12:17 | mriedem | jay and stephen | |
| 19:12:21 | dansmith | okay | |
| 19:12:46 | melwitt | donnyd: the last comment confuses me because compute downloads images over glance API (http), so what is the difference between that and the http.store option you mention, I wonder? | |
| 19:15:49 | dansmith | Seems like jay was actually in favor of managing the pci stuff like the rest of it: https://review.opendev.org/#/c/620115/ | |
| 19:16:00 | dansmith | "If we managed PCI devices using the same system we do for CPU, memory and local disk, then we could do the resource *claim* in the scheduler's claim_resources() " | |
| 19:16:29 | dansmith | actually, | |
| 19:16:37 | dansmith | maybe he means more placement-like by that | |
| 19:16:41 | sean-k-mooney | yes | |
| 19:17:24 | sean-k-mooney | he wanted to avoid the move cliams becuae he wante use to use placment as much as possible | |
| 19:17:59 | dansmith | I don't really see anywhere that he said that, but I don't doubt it | |
| 19:19:47 | artom | I really didn't want to start a witch hunt :( | |
| 19:19:52 | sean-k-mooney | that was my preference too by the way so im biased in recolection but i think if we need too i can adapt the sriov code to get the device form the move claim if needed | |
| 19:20:00 | dansmith | artom: no, just looking for context | |
| 19:20:10 | artom | Choices were made, I'm sure at the time with the available info they were optimal | |
| 19:20:14 | artom | dansmith, ack | |
| 19:20:43 | dansmith | artom: this is the problem with bolting on incremental "okay we'll justsupport live migration with pci if they're like this" kind of features | |
| 19:20:48 | dansmith | after you do that a couple times, you end up here | |
| 19:20:51 | donnyd | melwitt: well I am really just guessing because I don't understand how the http.store option works... I was hopeful that I could configure the filestore to save images in a particular location, and then have that exposed directly by apache (or something of the like) https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_drivers/http.py | |