| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-14 | |||
| 21:41:19 | tonyb | I'll go googling ;p | |
| 21:41:50 | sean-k-mooney | oh looks like you can force it too now https://wiki.qemu.org/Features/tcg-multithread | |
| 21:43:00 | sean-k-mooney | looks like aarch64 was in 2.9 | |
| 21:43:11 | sean-k-mooney | not sure if power has been added | |
| 21:48:42 | sean-k-mooney | look like mips,arm,aarch64,x86_64 and aplha have support in the latest version but no power supprot at elast form the release notes | |
| 21:48:57 | tonyb | Pfft | |
| 21:57:51 | openstackgerrit | Merged openstack/nova master: Join migration_context and flavor in Migration.instance https://review.opendev.org/689846 | |
| 21:57:57 | openstackgerrit | Merged openstack/nova master: Improve metadata server performance with large security groups https://review.opendev.org/656084 | |
| 22:04:34 | sean-k-mooney | ... | |
| 22:04:51 | sean-k-mooney | i was just about to push and then call it a day and now i have a merg conflcit | |
| 22:06:48 | openstackgerrit | sean mooney proposed openstack/nova master: support pci numa affinity policies in flavor and image https://review.opendev.org/674072 | |
| 22:06:54 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/train: Join migration_context and flavor in Migration.instance https://review.opendev.org/694408 | |
| 22:07:33 | sean-k-mooney | ill fix that tomorrow but it now has the func tests and artoms nits adressed | |
| 22:07:39 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/train: Improve metadata server performance with large security groups https://review.opendev.org/694409 | |
| 22:18:47 | sean-k-mooney | actully it looks trivial ill fix it now | |
| 22:19:37 | openstackgerrit | sean mooney proposed openstack/nova master: support pci numa affinity policies in flavor and image https://review.opendev.org/674072 | |
| 23:30:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Store instance action event exc_val fault details https://review.opendev.org/694428 | |
| #openstack-nova - 2019-11-15 | |||
| 00:43:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Expose instance action event details out of the API https://review.opendev.org/694430 | |
| 02:20:29 | openstackgerrit | Merged openstack/nova master: Move compute_node_to_inventory_dict to test-only code https://review.opendev.org/693438 | |
| 02:20:35 | openstackgerrit | Merged openstack/nova master: Remove get_minimum_version mocks from test_resource_tracker https://review.opendev.org/693439 | |
| 02:20:43 | openstackgerrit | Merged openstack/nova stable/queens: doc: fix and clarify --block-device usage in user docs https://review.opendev.org/694357 | |
| 02:59:56 | openstackgerrit | Merged openstack/nova master: Stop using NoAuthMiddleware in tests https://review.opendev.org/687416 | |
| 03:10:17 | openstackgerrit | Merged openstack/nova master: Remove TODO from ComputeTaskManager._live_migrate https://review.opendev.org/693696 | |
| 04:31:02 | openstackgerrit | Archit Modi proposed openstack/nova stable/pike: doc: fix and clarify --block-device usage in user docs https://review.opendev.org/694450 | |
| 05:41:05 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Add resources metadata of instance https://review.opendev.org/663563 | |
| 06:45:33 | openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Add resources metadata of instance https://review.opendev.org/663563 | |
| 07:18:39 | openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Handle new is_volume_backend join column query https://review.opendev.org/694462 | |
| 07:18:39 | openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Ignore root_gb if instance is booted from volume https://review.opendev.org/612626 | |
| 07:18:40 | openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Instance object changes for the new 'is_volume_backed' expected_attr https://review.opendev.org/694463 | |
| 08:08:32 | bauzas | good morning Nova | |
| 08:39:17 | awalende | Hello, I have trouble passing through an nvidia t4 card to my guests on Rocky. nova-compute log reports the final resource as: pci_stats=[PciDevicePool(count=1,numa_node=1,product_id='1eb8',tags={dev_type='type-PF'},vendor_id='10de')] | |
| 08:40:57 | awalende | However, when I set the alias as alias={"name":"T4","vendor_id":"10de","product_id":"1eb8"} on the controller and alias={"name":"T4","vendor_id":"10de", "product_id":"1eb8"} on compute node does not work | |
| 08:41:30 | awalende | Also I wonder, why it is declared as a "PF". I clearly inserted an PCIe card | |
| 08:42:16 | stephenfin | awalende: How have you configured the PCI whitelist in nova.conf? | |
| 08:42:40 | awalende | passthrough_whitelist={"vendor_id":"10de"} | |
| 08:45:16 | stephenfin | awalende: So this is interesting. The Tesla T4 supports SR-IOV https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-t4/t4-tensor-core-product-brief.pdf | |
| 08:46:21 | stephenfin | and because we're detecting that, we're flagging it as a PF https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L6562-L6565 | |
| 08:46:44 | awalende | So there is probably a difference passing them through as other regular gpus? | |
| 08:47:45 | awalende | Can we pass them even at all? | |
| 08:47:54 | stephenfin | Possibly. I could be wrong, but I do recall that type-PF devices were disabled by default because you typically wouldn't want to use them when you talk about NICs | |
| 08:47:59 | stephenfin | but it can be enabled | |
| 08:48:08 | stephenfin | do you want to pass through the PF or the VF? | |
| 08:48:23 | awalende | the whole hardware | |
| 08:48:53 | awalende | I guess probably the PF....so the usecase is, we want to pass the whole card to the guest. so that they have to install nvidia drivers by themselfs on the guest machine | |
| 08:49:09 | awalende | like we did with our older Tesla V100 cards | |
| 08:50:06 | stephenfin | Seems like you could get better resource utilization if you passed through the VFs though (since apparently that would give you up to 16 devices) | |
| 08:50:11 | stephenfin | That doesn't matter for now though | |
| 08:51:03 | stephenfin | If you want to use type-PF devices, I _think_ you need to either (a) set the 'device_type' flag in the 'alias' config option or (b) make sure the VFs are not whitelisted | |
| 08:51:46 | awalende | ye we already tried to set 'device_type' but it didn't make a difference :( | |
| 08:52:06 | stephenfin | have you tried (b)? | |
| 08:52:16 | stephenfin | to do that, you'll need to also set the 'product_id' in the 'passthrough_whitelist' | |
| 08:52:44 | stephenfin | to whatever the product_id of the PF is, assuming the PF and VF have different product_id's (they should) | |
| 08:52:54 | awalende | my colleague means he "maybe" tried it as well. We try it again and let you know | |
| 08:53:23 | stephenfin | If they don't, you're going to need to use the 'address' field, I'm afraid | |
| 08:53:31 | stephenfin | okay, do please | |
| 08:53:59 | stephenfin | sean-k-mooney might be able to help too but they're definitely fast asleep atm :) | |
| 08:54:13 | awalende | How do you make sure that "VFs are not whitelisted"? | |
| 08:55:08 | bauzas | the T4 ? | |
| 08:55:19 | bauzas | awalende: which grid driver are you using ? | |
| 08:55:25 | bauzas | GRID8 ? | |
| 08:55:53 | awalende | no driver at all, we plugged the T4 into our hypervisor nodes and want to pass them through to guests | |
| 08:56:05 | stephenfin | awalende: If SR-IOV is enabled on the device, you will have additional PCI devices appearing at e.g. 'lspci' | |
| 08:56:19 | awalende | one moment | |
| 08:57:06 | stephenfin | 'lspci | grep -i nvidia' should flag them, I'd imagine | |
| 08:57:07 | bauzas | oh right, then that's normal | |
| 08:57:26 | bauzas | Nvidia plans to deliver virtual GPUs as VFs later | |
| 08:57:36 | bauzas | I guess they already started to expose the GPU as a PF | |
| 08:57:44 | awalende | stephenfin, it only shows me one card as af:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1) | |
| 08:57:45 | stephenfin | I'm not sure how you'd turn them on/off. Probably by echoing stuff to sysfs like you do for SR-IOV NICs, I guess | |
| 08:57:48 | bauzas | but I thought the Tesla models wouldn't get it | |
| 08:57:55 | bauzas | just the Volta ones | |
| 08:58:20 | bauzas | I'm not a nVidia salesperson, (un?)fornutately :p | |
| 08:58:23 | stephenfin | okay, it's probably disabled so | |
| 08:59:02 | bauzas | awalende: just do something like 'lspci -vv | grep 10de | |
| 08:59:06 | stephenfin | awalende: So there is definitely _something_ specific you need to do in order to use type-PF devices. I'm just trying to figure out what it is | |
| 08:59:53 | awalende | bauzas, returns empty | |
| 08:59:59 | bauzas | mmm | |
| 09:00:04 | stephenfin | awalende: This is what I'm thinking of https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L289-L297 | |
| 09:00:19 | bauzas | awalende: are you OK with pasting your whole lspci output ? | |
| 09:00:49 | stephenfin | so if I'm reading that correctly, you *must* set 'dev_type': 'type-PF' in the alias | |
| 09:01:22 | bauzas | I agree | |
| 09:01:22 | awalende | which lspci command to you want pasted? 'lspci -vv'? | |
| 09:01:33 | bauzas | awalende: yes please | |
| 09:02:09 | bauzas | stephenfin: looks like you need to expose your type-PF this way | |
| 09:02:22 | stephenfin | wdym? | |
| 09:02:27 | bauzas | but I'm amazed that nvidia provides a PF and not a straight PCI ID | |
| 09:03:02 | bauzas | stephenfin: you have to expose the PCI devices in nova.conf in order to allow Nova to use them, right? | |
| 09:03:22 | stephenfin | yeah, awalende has done that mostly correctly I think | |
| 09:03:51 | awalende | https://paste.ubuntu.com/p/xXkRfFMRYh/ | |
| 09:04:12 | stephenfin | the '[pci] passthrough_whitelist' on the compute node is correct, but the '[pci] alias' on the controller and compute node doesn't appear to be (it needs 'dev_type') | |
| 09:04:43 | awalende | stephenfin, we will try soon! | |
| 09:05:09 | awalende | does dev_type needs to be specified on controller and compute? | |
| 09:05:48 | stephenfin | sorry, 'device_type' | |
| 09:06:01 | stephenfin | in the 'alias' option on both | |
| 09:06:22 | awalende | ok, we'll try it | |
| 09:06:38 | stephenfin | alias={"name":"T4","vendor_id":"10de","product_id":"1eb8","device_type":"type-PF"} | |
| 09:06:40 | stephenfin | or something like that | |
| 09:07:56 | bauzas | very interesting lspci | |
| 09:08:11 | bauzas | https://paste.ubuntu.com/p/xXkRfFMRYh/ L3241 | |
| 09:08:28 | bauzas | it shows a SR-IOV capability | |