| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-12 | |||
| 09:21:10 | sean-k-mooney | but i dont think that is a resonable long term fix | |
| 09:22:45 | gibi | simply counting it as reserved does not help when a VM is booted with a new image to a compute. There we would need to make sure that both the new cached image and the VM root fs fits the compute | |
| 09:22:58 | gibi | we discussed these options with dansmith yesterday | |
| 09:23:09 | gibi | see the summary in the bug and the link to the IRC discussion | |
| 09:24:14 | openstackgerrit | Merged openstack/nova-specs master: move implemented specs in ussuri https://review.opendev.org/721278 | |
| 09:25:35 | sean-k-mooney | gibi: are we sure we use the cache image directly for the backing file by the way | |
| 09:26:13 | sean-k-mooney | gibi: i tought that was how it worked but i remeber someone telling me its not how it worked and that we chage two copies one in the caceh and one for the backing file | |
| 09:26:59 | gibi | let me double check | |
| 09:27:10 | ikla | any info on gpu passthrough w/ rtx 8000's ? | |
| 09:27:27 | huaqiang | stephenfin: I am not sure how to keep exisiting +2, will that be kept if I do not change the | |
| 09:27:38 | sean-k-mooney | ikla: are you trying to do full passthough or are you trying to use its vgpu capablity | |
| 09:27:46 | aarents | I think it is still the same directory | |
| 09:28:14 | huaqiang | will the +2 be kepet if I donot change the 'Change-ID'? | |
| 09:28:53 | gibi | sean-k-mooney: http://paste.openstack.org/show/793424/ | |
| 09:29:35 | gibi | sean-k-mooney: based on this the image in the _base dir is the backing store for the instance's image | |
| 09:29:58 | gibi | I have two instances from the same image, and have one backing image in the _base dir | |
| 09:30:22 | sean-k-mooney | gibi: yep but is that the cached image or is it a second copy | |
| 09:30:41 | sean-k-mooney | depenidn gon how we set the force cow and force raw values this might change | |
| 09:31:03 | sean-k-mooney | in that case we are using a raw backing file | |
| 09:31:15 | sean-k-mooney | but waht formate was the image originally? | |
| 09:31:26 | sean-k-mooney | i assume qcow? | |
| 09:31:27 | gibi | sean-k-mooney: a) it doesn't matter. I can reword the bug that the bacing file size is not counted as used b) I think this is the cache as if I delete the instances it does not deleted imediately | |
| 09:31:59 | bauzas | ikla: not sure I understand your question about gpu passthrough | |
| 09:33:19 | gibi | sean-k-mooney: I will try to look into the case when the VM uses a raw file to see if then the image in _base is deleted, or if I can make a small change to be deleted | |
| 09:34:46 | sean-k-mooney | gibi: there is a periodic that deletes the image when no vms is using it on the host as far as i am aware | |
| 09:35:02 | gibi | sean-k-mooney: yes, that is the cache manager :) | |
| 09:35:05 | sean-k-mooney | so until that runs the cached copy will be there | |
| 09:35:48 | gibi | sean-k-mooney: anyhow the core of the problem is that nova uses more disk that it counts as used and today there is no way to avoid that except putting the _base dir on a separate partition | |
| 09:36:46 | gibi | when we had the DiskFilter it had the disk_available_least information from the compute to prevent overallocating the disks but placement does not have such info | |
| 09:37:19 | sean-k-mooney | well not entirely | |
| 09:37:32 | ikla | grid vgpu | |
| 09:37:46 | sean-k-mooney | the disk avaiable least behavior was still affected by the disk allocation ratio | |
| 09:38:26 | bauzas | ikla: gpu passthrough != grid vgpu , you know ? | |
| 09:38:36 | sean-k-mooney | gibi: i still go back to https://gist.github.com/JCallicoat/43505cab0535057ca4fb every time i want to figure that out | |
| 09:39:03 | bauzas | in one case, you're litterally giving up the gpu device to the guest | |
| 09:39:18 | ikla | sorry I didn't explain that correctly | |
| 09:39:26 | ikla | you could do a full passthrough | |
| 09:39:27 | bauzas | in the other case, you're asking the nvidia driver to slice your gpu into pieces that can be provided to the guests | |
| 09:39:28 | ikla | or do vgpu | |
| 09:39:34 | ikla | correct | |
| 09:39:42 | sean-k-mooney | ikla: yes both are supportred | |
| 09:39:53 | bauzas | so, again, what's your question ? | |
| 09:40:00 | ikla | is there docs on it | |
| 09:40:09 | sean-k-mooney | you can use mdev based vgpus or you can do direct passhtough of the gpu | |
| 09:40:14 | bauzas | ikla: indeed | |
| 09:40:24 | bauzas | https://docs.openstack.org/nova/latest/admin/virtual-gpu.html | |
| 09:40:28 | ikla | I just found it | |
| 09:40:31 | ikla | oh... thanks. :) | |
| 09:40:34 | bauzas | https://docs.openstack.org/nova/latest/admin/pci-passthrough.html | |
| 09:40:58 | bauzas | we don't provide special bits of nvidia installing and OS preparation | |
| 09:41:22 | bauzas | but you can get'em from the nvidia grid docs | |
| 09:42:18 | bauzas | eg. https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#red-hat-el-kvm-install-configure-vgpu | |
| 09:42:21 | gibi | sean-k-mooney: I can reword. Assuming disk_allocation_ratio = 1.0. DiskFilter detected if you overallocated the disk and nova does that with the image cache. placement does not know if you overallocate the disk | |
| 09:43:17 | sean-k-mooney | gibi: placement has the allication raitio in the inventory recored too fyi | |
| 09:43:33 | sean-k-mooney | gibi: im not that worried about the wording to be honest | |
| 09:43:45 | sean-k-mooney | gibi: im more interested in how you plan to adress it | |
| 09:43:51 | gibi | sean-k-mooney: this is not the intentional disk allocaiton ratio see my assumption above. It is the unintentional disk overallocation due to image cache | |
| 09:44:18 | gibi | sean-k-mooney: have you seen the options in the bug comments? | |
| 09:44:24 | sean-k-mooney | yes | |
| 09:44:26 | sean-k-mooney | i have | |
| 09:45:00 | sean-k-mooney | options A and b we have said wont work for different reasons | |
| 09:45:18 | sean-k-mooney | so the only viable one would be disabel cache or one of the workarounds | |
| 09:45:52 | sean-k-mooney | i guess B could work if instead of purgin the image when the cache is full we just dont cache new iamges | |
| 09:46:24 | sean-k-mooney | A wont work becasue we could race with concurrent boot requests | |
| 09:46:53 | sean-k-mooney | gibi: i have an option d | |
| 09:46:58 | gibi | shoot | |
| 09:47:25 | sean-k-mooney | gibi: if we have consumer types we could create allocations for cached images against the RP | |
| 09:47:37 | sean-k-mooney | using a nova consumer type | |
| 09:47:53 | sean-k-mooney | if we cant create the allcoation because there is not enough sapce then we dont cache it | |
| 09:48:12 | sean-k-mooney | if we can the it will prevent the issue as all usage will be tracked in placment | |
| 09:49:26 | gibi | we migt not need consumer types, we can simple create an allocation where the consumer_id is not an instance or a migration but the cache itself. | |
| 09:49:56 | sean-k-mooney | ya i was thinking we could use the image uuid as the consumer uuid and a cache consumer type | |
| 09:50:15 | sean-k-mooney | that way we would know which image are cached on each node | |
| 09:50:33 | gibi | in the allocation candidate query we have to either include 1x disk space if the host already caching the image or 2x disk space if the host will cache the image due to the current request | |
| 09:51:00 | sean-k-mooney | gibi: not if we change the logic so that the cacheing is optional | |
| 09:51:24 | sean-k-mooney | e.g. have the compute node make the allcoation after the fact when teh node is about to be spawned | |
| 09:52:14 | sean-k-mooney | so if the image is not already cached, chech can you create an allcoation for it, if so cache it and proceed with the boot if not dont cache it and just create a copy | |
| 09:52:20 | sean-k-mooney | for the vm | |
| 09:52:20 | ikla | can I request I set multiple names in pci_passthrough for the flavor? | |
| 09:53:10 | sean-k-mooney | gibi: the only time that would not cache is when the disk is almost full | |
| 09:53:28 | ikla | what would be the syntax or is it multiple pci_passthrough lines for the flavor? | |
| 09:54:14 | gibi | sean-k-mooney: do you see the complexity of making the cache optional in nova code? yesterday we conculded with dansmith that it could be pretty hairy change | |
| 09:54:32 | gibi | but I'm not an expert on the image backends | |
| 09:55:04 | sean-k-mooney | gibi: i know that part of the code makes heavy use of functools.partil and is really hard to follow | |
| 09:55:21 | sean-k-mooney | so i would guess it would be non trivial | |
| 09:55:30 | sean-k-mooney | but mainly due to techdebt | |
| 09:55:48 | openstackgerrit | Jiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD https://review.opendev.org/572805 | |
| 09:56:27 | ikla | something like: {"pci_passthrough:alias"="name1:1, name2:1, name3:1"} | |
| 09:57:44 | sean-k-mooney | ikla: yes its a comma seperate list | |
| 09:57:46 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/pci/request.py#L234-L237 | |
| 09:58:24 | sean-k-mooney | so openstack flavor set --property "pci_passthrough:alias"="name1:1,name2:1,name3:1" my-flavor | |
| 09:58:24 | gibi | sean-k-mooney: thanks. I will look into this direction as well | |
| 10:01:39 | ikla | Insufficient compute resources: Claim pci failed. | |
| 10:01:41 | ikla | hmm | |
| 10:02:15 | sean-k-mooney | ikla: you might need to weaken the default numa affintiy policy | |
| 10:02:35 | sean-k-mooney | e.g. if you dont have all the device on the same numa node | |
| 10:03:17 | sean-k-mooney | ikla: you can set numa_policy=perfer in the alisa | |
| 10:03:31 | sean-k-mooney | i just need to check that value is correct but i think its prefer or prefered | |
| 10:05:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Address issues with flake8 3.8.0 https://review.opendev.org/727140 | |
| 10:05:25 | sean-k-mooney | ikla: its "preferred" | |