| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-27 | |||
| 18:40:24 | mriedem | ok yeah that ^ | |
| 18:40:25 | jroll | if we do it idk about it | |
| 18:40:26 | melwitt | mriedem: haha, my slow message composition strikes again | |
| 18:40:28 | mriedem | was going to ask penick but he's not here | |
| 18:40:31 | jroll | rax did some of that too | |
| 18:40:34 | donnyd | I thought about setting up a share from my block image server that would share _base across hypervisors | |
| 18:40:44 | donnyd | making it only have to download once | |
| 18:40:48 | donnyd | which is acceptable | |
| 18:41:05 | donnyd | but i am not sure of the other performance implications of doing such a thing | |
| 18:41:11 | jroll | mriedem: is there something I can help with around that? | |
| 18:41:13 | mriedem | yeah, people do that https://bugs.launchpad.net/nova/+bug/1804262 | |
| 18:41:15 | openstack | Launchpad bug 1804262 in OpenStack Compute (nova) "ComputeManager._run_image_cache_manager_pass times out when running on NFS" [Medium,In progress] - Assigned to Matthew Booth (mbooth-9) | |
| 18:41:27 | mriedem | jroll: not really | |
| 18:41:33 | jroll | k :) | |
| 18:41:43 | mriedem | i thought i saw something (blog/talk?) where penick was talking about doing this | |
| 18:41:54 | jroll | possibly | |
| 18:42:08 | mriedem | i was probably surprised because it was some non-baremetal thing | |
| 18:42:11 | donnyd | I was only going to mount _base so the first hypervisor to download the image would speed it up for the rest | |
| 18:42:15 | jroll | just not sure if there's a question to be answered or if you're just curious | |
| 18:42:39 | jroll | rax did bare metal image caching and it was awesomesauce | |
| 18:42:50 | mriedem | donnyd: yeah i'm pretty sure i was talking with someone in here awhile back that had the same idea | |
| 18:42:53 | melwitt | yeah sharing _base/ makes sense | |
| 18:43:36 | donnyd | jroll: Well when new images are loaded from nodepool, it takes a while for compute to download from glance X # of hypervisors | |
| 18:43:55 | donnyd | so some pre-caching or sharing of _base would likely speed things up | |
| 18:44:27 | donnyd | Glance is seemingly limited to one core's worth of power when downloading | |
| 18:45:18 | donnyd | which for my controllers is unfortunately not fast | |
| 18:46:24 | donnyd | melwitt: my only concern in mounting _base on a shared drive is that what happens when multiple hypervisors are all trying to download the same image at the same time | |
| 18:47:05 | mriedem | there might be a lock in the code, but would need to verify | |
| 18:47:08 | melwitt | hm, yeah | |
| 18:47:20 | mriedem | in fact, i think starlingx people added a configurable lock there | |
| 18:47:32 | jroll | donnyd: gotcha, thanks for context | |
| 18:47:37 | donnyd | would they all download it anyways, or as mriedem just said would they see another hypervisor already has that in-flight | |
| 18:48:25 | artom | sean-k-mooney, do you recall if the SRIOV live migration code ever updated instance.pci_requests? | |
| 18:48:38 | artom | I don't see anything quickly glancing through your patches | |
| 18:48:56 | mriedem | donnyd: this is what i'm thinking of https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/imagebackend.py#L257 | |
| 18:49:18 | sean-k-mooney | we do update them when we allocate the claimed pci deivce i belive | |
| 18:50:14 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#compute.max_concurrent_disk_ops was the thing starlingx added | |
| 18:51:18 | donnyd | well I should give it a spin.. probably cut the "new image loaded so i will fail a bunch" down to almost nothing | |
| 18:51:25 | openstackgerrit | Merged openstack/python-novaclient master: Follow up for microversion 2.75 https://review.opendev.org/678473 | |
| 18:51:33 | sean-k-mooney | artom: i think it get saved here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7229-L7256 | |
| 18:51:42 | mriedem | donnyd: if you do let us know how it goes, | |
| 18:51:45 | sean-k-mooney | in post_live_migration_at_destination | |
| 18:51:53 | sean-k-mooney | so we only update it if the migration succeeds | |
| 18:51:54 | mriedem | donnyd: i also threw it in https://bugs.launchpad.net/nova/+bug/1838819/comments/7 so we don't forget docs about it | |
| 18:51:55 | openstack | Launchpad bug 1838819 in OpenStack Compute (nova) "Docs needed for tunables at large scale" [Undecided,Confirmed] | |
| 18:52:14 | sean-k-mooney | if it was reverted we do not modify the instace pci_requests | |
| 18:52:20 | mriedem | image cache docs in general would be great but stephenfin just refuses to support us documenting things | |
| 18:52:25 | mriedem | o-) | |
| 18:52:51 | donnyd | LOL mriedem | |
| 18:54:15 | artom | sean-k-mooney, AFAICT that calls down to https://github.com/openstack/nova/blob/master/nova/objects/pci_device.py#L373 | |
| 18:54:24 | artom | sean-k-mooney, which saves pci_devices, but not pci_requests | |
| 18:56:08 | artom | Does it even make sense for pci_requests to change for a live migration? | |
| 18:56:58 | sean-k-mooney | well the number and type of request wont change | |
| 18:57:21 | artom | The spec might? | |
| 18:58:04 | mriedem | i would hope that pci requests don't change for a live migration | |
| 18:58:09 | mriedem | otherwise we f'ed up somewhere in the data modeling | |
| 18:58:11 | sean-k-mooney | what do you mean | |
| 18:58:22 | mriedem | pci devices are the per-compute/instance thing that rack inventory and what's allocated | |
| 18:58:33 | mriedem | the pci requests should just be the host agnostic request | |
| 18:58:48 | sean-k-mooney | ya they more or less are | |
| 18:58:55 | sean-k-mooney | the can have an alisa | |
| 18:59:16 | sean-k-mooney | or my have tags like a neutorn physnet | |
| 18:59:23 | melwitt | donnyd, mriedem: I just asked penick and he said they've only done image cache warming once or twice on an ad hoc basis, for example for a 40G windows image that was needed on certain hypervisors. he said even still, downloads don't take too long (couple minutes at most) bc they always have their glance in the same datacenter. he also said they disable image checksums to avoid that slowness | |
| 18:59:26 | sean-k-mooney | or device type e.g. type-VF | |
| 18:59:36 | artom | sean-k-mooney, right, but that can't change during a live migration, right? | |
| 18:59:50 | sean-k-mooney | but the request does not have any info about the specifc device | |
| 18:59:58 | sean-k-mooney | artom: correct it cant | |
| 19:00:42 | melwitt | donnyd: I didn't notice if you mentioned how long is a long time in your scenario. is it a few minutes or like 20 minutes | |
| 19:00:42 | donnyd | melwitt: My downloads are in the same rack on a 10G network (compute side) and they are painfully slow for the gear that underpins them | |
| 19:02:50 | artom | So we seem fairly confident pci_requests can't change during a live migration. So back to our original problem, it should be find for a Claim to write them back to the instance | |
| 19:03:06 | donnyd | well the drive glance is hosted on is an nvme device with 4G/s in read speed and the machine its on has 40G networking... so I expected to get somewhere around 1/4 of that or about 1G/s (ish) in download speed | |
| 19:03:06 | artom | In the sense that, the SRIOV live migration code won't have changed them | |
| 19:03:10 | dansmith | right, which was my original thing.. why not just keep them? | |
| 19:03:21 | donnyd | the reality is 100M/s | |
| 19:03:25 | artom | dansmith, I was worried about them changing under us | |
| 19:04:23 | artom | So by keeping them we'd end up clobbering the new ones set by the SRIOV live migration code | |
| 19:04:33 | donnyd | If I could speed up the downloading part, i wouldn't even notice | |
| 19:04:53 | artom | Which is why I wanted to strongly to avoid touching them altogether | |
| 19:04:54 | donnyd | because a new image would be downloaded in 30-40 seconds | |
| 19:05:04 | dansmith | artom: we stash a copy of the requests to be applied int he migration context with other things.. if the sriov migration code is not playing nice with that, then it's wrong | |
| 19:05:08 | sean-k-mooney | artom: we wont clobber anything | |
| 19:05:18 | sean-k-mooney | but we will end up trying to claim pci device twice | |
| 19:05:42 | artom | sean-k-mooney, well no, dansmith's point was that we still skip the actual claiming (right?) | |
| 19:05:49 | artom | Just don't mess with any DB stuff | |
| 19:05:54 | sean-k-mooney | and then whe might not move them form claimed to allocated correctly and leak pci deivces | |
| 19:06:03 | dansmith | artom: I don't think I made that claim | |
| 19:06:05 | sean-k-mooney | e.g. once that are claimed for the instance but not allocated to it | |
| 19:06:07 | dansmith | artom: I might have asked that | |
| 19:06:19 | mriedem | "I don't think I made that claim" | |
| 19:06:25 | artom | We need to stop overloading "claim" >_< | |
| 19:06:39 | dansmith | I still don't understand why this needs to be different for cold and live migration, with respect to the accounting | |
| 19:06:57 | melwitt | donnyd: right... that does sound strange, but I'm definitely not that knowledgable about what is reasonable to expect there. I'll run those details by penick and find out if it rings a bell | |
| 19:06:58 | sean-k-mooney | claimed and allocated are states in the pci resouce tracker. | |
| 19:07:08 | sean-k-mooney | and claimes generally refers to the RT | |
| 19:07:14 | artom | dansmith, it doesn't :) But SRIOV live migration was implemented without it, so now we're in this mess | |
| 19:07:18 | sean-k-mooney | we have moved to using allocaiton to refer to placmeent | |
| 19:07:30 | dansmith | artom: yeah, sounds like that is the real problem here | |
| 19:08:04 | donnyd | I think the last time we dug into it, it had something to do with the requests library only being able to use one core, which in turn make sense because my controllers cores aren't super fast | |
| 19:08:24 | sean-k-mooney | i can proably make the sriov stuff work with move claims. but i would prefer to keep move claimes for cold migration | |
| 19:08:52 | sean-k-mooney | sriov migration did not use claims because we did not need to and live migration never used them before | |