| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-12 | |||
| 09:35:05 | sean-k-mooney | so until that runs the cached copy will be there | |
| 09:35:48 | gibi | sean-k-mooney: anyhow the core of the problem is that nova uses more disk that it counts as used and today there is no way to avoid that except putting the _base dir on a separate partition | |
| 09:36:46 | gibi | when we had the DiskFilter it had the disk_available_least information from the compute to prevent overallocating the disks but placement does not have such info | |
| 09:37:19 | sean-k-mooney | well not entirely | |
| 09:37:32 | ikla | grid vgpu | |
| 09:37:46 | sean-k-mooney | the disk avaiable least behavior was still affected by the disk allocation ratio | |
| 09:38:26 | bauzas | ikla: gpu passthrough != grid vgpu , you know ? | |
| 09:38:36 | sean-k-mooney | gibi: i still go back to https://gist.github.com/JCallicoat/43505cab0535057ca4fb every time i want to figure that out | |
| 09:39:03 | bauzas | in one case, you're litterally giving up the gpu device to the guest | |
| 09:39:18 | ikla | sorry I didn't explain that correctly | |
| 09:39:26 | ikla | you could do a full passthrough | |
| 09:39:27 | bauzas | in the other case, you're asking the nvidia driver to slice your gpu into pieces that can be provided to the guests | |
| 09:39:28 | ikla | or do vgpu | |
| 09:39:34 | ikla | correct | |
| 09:39:42 | sean-k-mooney | ikla: yes both are supportred | |
| 09:39:53 | bauzas | so, again, what's your question ? | |
| 09:40:00 | ikla | is there docs on it | |
| 09:40:09 | sean-k-mooney | you can use mdev based vgpus or you can do direct passhtough of the gpu | |
| 09:40:14 | bauzas | ikla: indeed | |
| 09:40:24 | bauzas | https://docs.openstack.org/nova/latest/admin/virtual-gpu.html | |
| 09:40:28 | ikla | I just found it | |
| 09:40:31 | ikla | oh... thanks. :) | |
| 09:40:34 | bauzas | https://docs.openstack.org/nova/latest/admin/pci-passthrough.html | |
| 09:40:58 | bauzas | we don't provide special bits of nvidia installing and OS preparation | |
| 09:41:22 | bauzas | but you can get'em from the nvidia grid docs | |
| 09:42:18 | bauzas | eg. https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#red-hat-el-kvm-install-configure-vgpu | |
| 09:42:21 | gibi | sean-k-mooney: I can reword. Assuming disk_allocation_ratio = 1.0. DiskFilter detected if you overallocated the disk and nova does that with the image cache. placement does not know if you overallocate the disk | |
| 09:43:17 | sean-k-mooney | gibi: placement has the allication raitio in the inventory recored too fyi | |
| 09:43:33 | sean-k-mooney | gibi: im not that worried about the wording to be honest | |
| 09:43:45 | sean-k-mooney | gibi: im more interested in how you plan to adress it | |
| 09:43:51 | gibi | sean-k-mooney: this is not the intentional disk allocaiton ratio see my assumption above. It is the unintentional disk overallocation due to image cache | |
| 09:44:18 | gibi | sean-k-mooney: have you seen the options in the bug comments? | |
| 09:44:24 | sean-k-mooney | yes | |
| 09:44:26 | sean-k-mooney | i have | |
| 09:45:00 | sean-k-mooney | options A and b we have said wont work for different reasons | |
| 09:45:18 | sean-k-mooney | so the only viable one would be disabel cache or one of the workarounds | |
| 09:45:52 | sean-k-mooney | i guess B could work if instead of purgin the image when the cache is full we just dont cache new iamges | |
| 09:46:24 | sean-k-mooney | A wont work becasue we could race with concurrent boot requests | |
| 09:46:53 | sean-k-mooney | gibi: i have an option d | |
| 09:46:58 | gibi | shoot | |
| 09:47:25 | sean-k-mooney | gibi: if we have consumer types we could create allocations for cached images against the RP | |
| 09:47:37 | sean-k-mooney | using a nova consumer type | |
| 09:47:53 | sean-k-mooney | if we cant create the allcoation because there is not enough sapce then we dont cache it | |
| 09:48:12 | sean-k-mooney | if we can the it will prevent the issue as all usage will be tracked in placment | |
| 09:49:26 | gibi | we migt not need consumer types, we can simple create an allocation where the consumer_id is not an instance or a migration but the cache itself. | |
| 09:49:56 | sean-k-mooney | ya i was thinking we could use the image uuid as the consumer uuid and a cache consumer type | |
| 09:50:15 | sean-k-mooney | that way we would know which image are cached on each node | |
| 09:50:33 | gibi | in the allocation candidate query we have to either include 1x disk space if the host already caching the image or 2x disk space if the host will cache the image due to the current request | |
| 09:51:00 | sean-k-mooney | gibi: not if we change the logic so that the cacheing is optional | |
| 09:51:24 | sean-k-mooney | e.g. have the compute node make the allcoation after the fact when teh node is about to be spawned | |
| 09:52:14 | sean-k-mooney | so if the image is not already cached, chech can you create an allcoation for it, if so cache it and proceed with the boot if not dont cache it and just create a copy | |
| 09:52:20 | sean-k-mooney | for the vm | |
| 09:52:20 | ikla | can I request I set multiple names in pci_passthrough for the flavor? | |
| 09:53:10 | sean-k-mooney | gibi: the only time that would not cache is when the disk is almost full | |
| 09:53:28 | ikla | what would be the syntax or is it multiple pci_passthrough lines for the flavor? | |
| 09:54:14 | gibi | sean-k-mooney: do you see the complexity of making the cache optional in nova code? yesterday we conculded with dansmith that it could be pretty hairy change | |
| 09:54:32 | gibi | but I'm not an expert on the image backends | |
| 09:55:04 | sean-k-mooney | gibi: i know that part of the code makes heavy use of functools.partil and is really hard to follow | |
| 09:55:21 | sean-k-mooney | so i would guess it would be non trivial | |
| 09:55:30 | sean-k-mooney | but mainly due to techdebt | |
| 09:55:48 | openstackgerrit | Jiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD https://review.opendev.org/572805 | |
| 09:56:27 | ikla | something like: {"pci_passthrough:alias"="name1:1, name2:1, name3:1"} | |
| 09:57:44 | sean-k-mooney | ikla: yes its a comma seperate list | |
| 09:57:46 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/pci/request.py#L234-L237 | |
| 09:58:24 | sean-k-mooney | so openstack flavor set --property "pci_passthrough:alias"="name1:1,name2:1,name3:1" my-flavor | |
| 09:58:24 | gibi | sean-k-mooney: thanks. I will look into this direction as well | |
| 10:01:39 | ikla | Insufficient compute resources: Claim pci failed. | |
| 10:01:41 | ikla | hmm | |
| 10:02:15 | sean-k-mooney | ikla: you might need to weaken the default numa affintiy policy | |
| 10:02:35 | sean-k-mooney | e.g. if you dont have all the device on the same numa node | |
| 10:03:17 | sean-k-mooney | ikla: you can set numa_policy=perfer in the alisa | |
| 10:03:31 | sean-k-mooney | i just need to check that value is correct but i think its prefer or prefered | |
| 10:05:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Address issues with flake8 3.8.0 https://review.opendev.org/727140 | |
| 10:05:25 | sean-k-mooney | ikla: its "preferred" | |
| 10:05:28 | stephenfin | gibi, bauzas: if that isn't failing our gate yet, it's going to start soon ^ | |
| 10:05:32 | sean-k-mooney | ikla: https://github.com/openstack/nova/blob/master/nova/objects/fields.py#L734 | |
| 10:05:42 | sean-k-mooney | ikla: it could have failed for other reasons too | |
| 10:05:52 | stephenfin | https://review.opendev.org/727133 will fix it but I don't know how long until that's released | |
| 10:05:58 | stephenfin | *how long it will be | |
| 10:06:08 | bauzas | wait | |
| 10:06:35 | bauzas | stephenfin: doesn't that provide a new exception ? | |
| 10:07:23 | bauzas | stephenfin: do you have more context ? | |
| 10:07:26 | stephenfin | yes, E741 and F522 seem to be new | |
| 10:07:36 | bauzas | so we could ignore them first ? | |
| 10:07:46 | stephenfin | but...why? | |
| 10:07:50 | ikla | set in flavor? | |
| 10:08:03 | ikla | oh, in the alias? | |
| 10:08:17 | ikla | I get the same issue with one device | |
| 10:08:29 | stephenfin | the correct fix is for hacking to limit flake8 to a given minor version instead of the major version range it's using, but we need a new release of hacking for that | |
| 10:09:58 | sean-k-mooney | ikla: are you testing with a gpu | |
| 10:10:23 | sean-k-mooney | ikla: or do you have any specific errors int eh nova compute agent log | |
| 10:11:36 | ikla | no, these are network cards | |
| 10:12:37 | ikla | nothing in logs | |
| 10:12:39 | sean-k-mooney | ok then likely you if its not the numa issue the next thing to check woudl be the pci device type | |
| 10:12:52 | sean-k-mooney | do the nics support sriov? | |
| 10:13:13 | ikla | yes | |
| 10:13:30 | sean-k-mooney | if they dont have the capablity of SRIOV then the type will be type-PCI | |
| 10:13:30 | ikla | they are setup and I can see them in the pci list with lspci -nnn | |
| 10:13:35 | ikla | Virtual x 4 | |
| 10:13:46 | sean-k-mooney | if they do then the PF will be type-PF and the VFs will be type-VF | |