| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-12 | |||
| 09:10:52 | gibi | bauzas, stephenfin: easy +2 https://review.opendev.org/#/c/721278 (ussuri spec move) | |
| 09:10:57 | ikla | my flavor keeps failing when trying to turn on a instance | |
| 09:11:03 | sean-k-mooney | ikla: the alias need to be on the computes and contoler | |
| 09:11:14 | jkulik | sean-k-mooney, yes. definitely pets | |
| 09:11:18 | sean-k-mooney | ikla: the pci whitelist is only needed on the compute | |
| 09:11:27 | ikla | alias yes, device only on compute.. | |
| 09:11:28 | ikla | k | |
| 09:11:52 | sean-k-mooney | ikla: so the contole just uses it to translate the value in the flavor into pci request for schduling | |
| 09:12:20 | gibi | bauzas: I agree with WontFix https://bugs.launchpad.net/nova/+bug/1838309 | |
| 09:12:20 | openstack | Launchpad bug 1838309 in OpenStack Compute (nova) "Live migration might fail when run after revert of previous live migration" [Undecided,Won't fix] - Assigned to Vladyslav Drok (vdrok) | |
| 09:12:29 | bauzas | cool, moving on | |
| 09:13:03 | gibi | tony_su: thanks for taking that feature over. I will look at the spec soon | |
| 09:15:27 | gibi | bauzas: thanks for looking at the new bugs. You have the bug lock, I have yet another downstream originated issue to look at https://bugs.launchpad.net/nova/+bug/1878024 | |
| 09:15:27 | openstack | Launchpad bug 1878024 in OpenStack Compute (nova) "disk usage of the nova image cache is not counted as used disk space" [Undecided,Confirmed] | |
| 09:16:12 | sean-k-mooney | gibi: thats a tricky one | |
| 09:16:40 | sean-k-mooney | in that really we should never fail to boot a vm becasue of a cached image taking up space | |
| 09:17:10 | sean-k-mooney | i.e. i would expect as an operator for the cache to be perurged first | |
| 09:17:25 | sean-k-mooney | so im not sure i would want it to be counted as used | |
| 09:17:47 | sean-k-mooney | that siad we dont purge the cache today as far as i know at least not while the image is used on the host | |
| 09:17:54 | sean-k-mooney | so i can see this causeing issues too | |
| 09:18:18 | sean-k-mooney | could you just workaround this using the host reserved disk paramater | |
| 09:18:47 | sean-k-mooney | or perhaps we need a config option to limit the size of the cache | |
| 09:19:37 | gibi | sean-k-mooney: the cache is used as a backing file for qcow root fs so it cannot be really purged | |
| 09:19:40 | sean-k-mooney | the easy option is just to add the size of the cache to the reserved value in placment but that will reduce the amount of vms that can be spawned | |
| 09:19:54 | sean-k-mooney | gibi: i tought we had two copies | |
| 09:20:01 | sean-k-mooney | on in the cache and a second for the backing file | |
| 09:20:20 | sean-k-mooney | isnt the cach module indpentend of the image backend | |
| 09:20:22 | gibi | sean-k-mooney: reserved_host_disk_mb would only be useful if we could limit the size of the cache | |
| 09:21:02 | sean-k-mooney | yep the two line "fix" is to jsut count the cache size and added it to the reserved value in placment | |
| 09:21:10 | sean-k-mooney | but i dont think that is a resonable long term fix | |
| 09:22:45 | gibi | simply counting it as reserved does not help when a VM is booted with a new image to a compute. There we would need to make sure that both the new cached image and the VM root fs fits the compute | |
| 09:22:58 | gibi | we discussed these options with dansmith yesterday | |
| 09:23:09 | gibi | see the summary in the bug and the link to the IRC discussion | |
| 09:24:14 | openstackgerrit | Merged openstack/nova-specs master: move implemented specs in ussuri https://review.opendev.org/721278 | |
| 09:25:35 | sean-k-mooney | gibi: are we sure we use the cache image directly for the backing file by the way | |
| 09:26:13 | sean-k-mooney | gibi: i tought that was how it worked but i remeber someone telling me its not how it worked and that we chage two copies one in the caceh and one for the backing file | |
| 09:26:59 | gibi | let me double check | |
| 09:27:10 | ikla | any info on gpu passthrough w/ rtx 8000's ? | |
| 09:27:27 | huaqiang | stephenfin: I am not sure how to keep exisiting +2, will that be kept if I do not change the | |
| 09:27:38 | sean-k-mooney | ikla: are you trying to do full passthough or are you trying to use its vgpu capablity | |
| 09:27:46 | aarents | I think it is still the same directory | |
| 09:28:14 | huaqiang | will the +2 be kepet if I donot change the 'Change-ID'? | |
| 09:28:53 | gibi | sean-k-mooney: http://paste.openstack.org/show/793424/ | |
| 09:29:35 | gibi | sean-k-mooney: based on this the image in the _base dir is the backing store for the instance's image | |
| 09:29:58 | gibi | I have two instances from the same image, and have one backing image in the _base dir | |
| 09:30:22 | sean-k-mooney | gibi: yep but is that the cached image or is it a second copy | |
| 09:30:41 | sean-k-mooney | depenidn gon how we set the force cow and force raw values this might change | |
| 09:31:03 | sean-k-mooney | in that case we are using a raw backing file | |
| 09:31:15 | sean-k-mooney | but waht formate was the image originally? | |
| 09:31:26 | sean-k-mooney | i assume qcow? | |
| 09:31:27 | gibi | sean-k-mooney: a) it doesn't matter. I can reword the bug that the bacing file size is not counted as used b) I think this is the cache as if I delete the instances it does not deleted imediately | |
| 09:31:59 | bauzas | ikla: not sure I understand your question about gpu passthrough | |
| 09:33:19 | gibi | sean-k-mooney: I will try to look into the case when the VM uses a raw file to see if then the image in _base is deleted, or if I can make a small change to be deleted | |
| 09:34:46 | sean-k-mooney | gibi: there is a periodic that deletes the image when no vms is using it on the host as far as i am aware | |
| 09:35:02 | gibi | sean-k-mooney: yes, that is the cache manager :) | |
| 09:35:05 | sean-k-mooney | so until that runs the cached copy will be there | |
| 09:35:48 | gibi | sean-k-mooney: anyhow the core of the problem is that nova uses more disk that it counts as used and today there is no way to avoid that except putting the _base dir on a separate partition | |
| 09:36:46 | gibi | when we had the DiskFilter it had the disk_available_least information from the compute to prevent overallocating the disks but placement does not have such info | |
| 09:37:19 | sean-k-mooney | well not entirely | |
| 09:37:32 | ikla | grid vgpu | |
| 09:37:46 | sean-k-mooney | the disk avaiable least behavior was still affected by the disk allocation ratio | |
| 09:38:26 | bauzas | ikla: gpu passthrough != grid vgpu , you know ? | |
| 09:38:36 | sean-k-mooney | gibi: i still go back to https://gist.github.com/JCallicoat/43505cab0535057ca4fb every time i want to figure that out | |
| 09:39:03 | bauzas | in one case, you're litterally giving up the gpu device to the guest | |
| 09:39:18 | ikla | sorry I didn't explain that correctly | |
| 09:39:26 | ikla | you could do a full passthrough | |
| 09:39:27 | bauzas | in the other case, you're asking the nvidia driver to slice your gpu into pieces that can be provided to the guests | |
| 09:39:28 | ikla | or do vgpu | |
| 09:39:34 | ikla | correct | |
| 09:39:42 | sean-k-mooney | ikla: yes both are supportred | |
| 09:39:53 | bauzas | so, again, what's your question ? | |
| 09:40:00 | ikla | is there docs on it | |
| 09:40:09 | sean-k-mooney | you can use mdev based vgpus or you can do direct passhtough of the gpu | |
| 09:40:14 | bauzas | ikla: indeed | |
| 09:40:24 | bauzas | https://docs.openstack.org/nova/latest/admin/virtual-gpu.html | |
| 09:40:28 | ikla | I just found it | |
| 09:40:31 | ikla | oh... thanks. :) | |
| 09:40:34 | bauzas | https://docs.openstack.org/nova/latest/admin/pci-passthrough.html | |
| 09:40:58 | bauzas | we don't provide special bits of nvidia installing and OS preparation | |
| 09:41:22 | bauzas | but you can get'em from the nvidia grid docs | |
| 09:42:18 | bauzas | eg. https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#red-hat-el-kvm-install-configure-vgpu | |
| 09:42:21 | gibi | sean-k-mooney: I can reword. Assuming disk_allocation_ratio = 1.0. DiskFilter detected if you overallocated the disk and nova does that with the image cache. placement does not know if you overallocate the disk | |
| 09:43:17 | sean-k-mooney | gibi: placement has the allication raitio in the inventory recored too fyi | |
| 09:43:33 | sean-k-mooney | gibi: im not that worried about the wording to be honest | |
| 09:43:45 | sean-k-mooney | gibi: im more interested in how you plan to adress it | |
| 09:43:51 | gibi | sean-k-mooney: this is not the intentional disk allocaiton ratio see my assumption above. It is the unintentional disk overallocation due to image cache | |
| 09:44:18 | gibi | sean-k-mooney: have you seen the options in the bug comments? | |
| 09:44:24 | sean-k-mooney | yes | |
| 09:44:26 | sean-k-mooney | i have | |
| 09:45:00 | sean-k-mooney | options A and b we have said wont work for different reasons | |
| 09:45:18 | sean-k-mooney | so the only viable one would be disabel cache or one of the workarounds | |
| 09:45:52 | sean-k-mooney | i guess B could work if instead of purgin the image when the cache is full we just dont cache new iamges | |
| 09:46:24 | sean-k-mooney | A wont work becasue we could race with concurrent boot requests | |
| 09:46:53 | sean-k-mooney | gibi: i have an option d | |
| 09:46:58 | gibi | shoot | |
| 09:47:25 | sean-k-mooney | gibi: if we have consumer types we could create allocations for cached images against the RP | |
| 09:47:37 | sean-k-mooney | using a nova consumer type | |
| 09:47:53 | sean-k-mooney | if we cant create the allcoation because there is not enough sapce then we dont cache it | |
| 09:48:12 | sean-k-mooney | if we can the it will prevent the issue as all usage will be tracked in placment | |
| 09:49:26 | gibi | we migt not need consumer types, we can simple create an allocation where the consumer_id is not an instance or a migration but the cache itself. | |