Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-15
08:59:02 bauzas awalende: just do something like 'lspci -vv | grep 10de
08:59:06 stephenfin awalende: So there is definitely _something_ specific you need to do in order to use type-PF devices. I'm just trying to figure out what it is
08:59:53 awalende bauzas, returns empty
08:59:59 bauzas mmm
09:00:04 stephenfin awalende: This is what I'm thinking of https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L289-L297
09:00:19 bauzas awalende: are you OK with pasting your whole lspci output ?
09:00:49 stephenfin so if I'm reading that correctly, you *must* set 'dev_type': 'type-PF' in the alias
09:01:22 awalende which lspci command to you want pasted? 'lspci -vv'?
09:01:22 bauzas I agree
09:01:33 bauzas awalende: yes please
09:02:09 bauzas stephenfin: looks like you need to expose your type-PF this way
09:02:22 stephenfin wdym?
09:02:27 bauzas but I'm amazed that nvidia provides a PF and not a straight PCI ID
09:03:02 bauzas stephenfin: you have to expose the PCI devices in nova.conf in order to allow Nova to use them, right?
09:03:22 stephenfin yeah, awalende has done that mostly correctly I think
09:03:51 awalende https://paste.ubuntu.com/p/xXkRfFMRYh/
09:04:12 stephenfin the '[pci] passthrough_whitelist' on the compute node is correct, but the '[pci] alias' on the controller and compute node doesn't appear to be (it needs 'dev_type')
09:04:43 awalende stephenfin, we will try soon!
09:05:09 awalende does dev_type needs to be specified on controller and compute?
09:05:48 stephenfin sorry, 'device_type'
09:06:01 stephenfin in the 'alias' option on both
09:06:22 awalende ok, we'll try it
09:06:38 stephenfin alias={"name":"T4","vendor_id":"10de","product_id":"1eb8","device_type":"type-PF"}
09:06:40 stephenfin or something like that
09:07:56 bauzas very interesting lspci
09:08:11 bauzas https://paste.ubuntu.com/p/xXkRfFMRYh/ L3241
09:08:28 bauzas it shows a SR-IOV capability
09:08:35 bauzas but no VGA capability (!)
09:08:56 bauzas that's beyond my comfort zone
09:09:02 stephenfin yeah, it's an accelerator, not a GPU
09:09:10 bauzas my bad then
09:09:18 bauzas I'm not familiar with the product line
09:09:45 bauzas so we're just talking of PCI passthrugh, not VGA passthru, my bad
09:10:08 bauzas anyway, it doesn't expose a standard PCI address
09:12:49 bauzas I just feel that stephenfin is absolutely right, nova punts the address because of the internal conditional he said
09:15:21 awalende someone rebooted the server =.= one more minute
09:17:19 awalende ok, its spawning :D fingers crossed
09:20:53 awalende Ok, the device shows up in the guest
09:20:54 awalende hooray
09:21:29 awalende But why do we explicitly have to specify device_type. In all of our other devices it wasn't really needed.....is it because pci is default?
09:21:52 stephenfin awalende: https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L289-L297
09:22:43 stephenfin If you have a device that supports SR-IOV, the assumption we've made is that you'll generally want to use that capability because of the scalability it offers
09:23:33 awalende Uhhh just a minor complaint....this isn't documented :D :D. But nice, the T4 is now there in the guest. Thanks all for helping us!
09:23:44 stephenfin So by default we don't allow you to use the parent/root/PF device because if you do, you won't be able to use the child/VF device(s)
09:24:11 stephenfin Agreed. I'm working on a patch to do that now. I'll stick you on the review once it's up, if you like
09:24:24 awalende sure thing :D
09:27:09 awalende irc name is launchpad name
09:33:30 huaqiang sean-k-mooney: about the per-vm-pci-NUMA-policy spec, I want to confirm that it defines a policy applied to all PCI devices for whole VM, right?
09:43:51 huaqiang sean-k-mooney: seems I have sent my questions we have drawn a conclusion agian by mistake, pls ignore it.
10:01:53 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP block_device: Use original volume_type when creating volumes from snapshots https://review.opendev.org/694497
10:21:26 openstackgerrit Liang Fang proposed openstack/nova-specs master: Support volume local cache https://review.opendev.org/689070
11:01:22 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.opendev.org/693849
11:01:23 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: Boot from volume instance rescue https://review.opendev.org/694063
11:46:01 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Change order of PCI configuration steps https://review.opendev.org/694521
11:46:01 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522
11:46:04 stephenfin awalende: ^
11:46:14 stephenfin Also sean-k-mooney, adrianc: ^
11:46:30 stephenfin I'd like to backport that as far as we can go
11:48:18 openstackgerrit Merged openstack/nova master: Reset vm_state to original value if rebuild claim fails https://review.opendev.org/692185
11:48:51 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522
11:53:52 openstackgerrit Merged openstack/nova master: FUP for Ib62ac0b692eb92a2ed364ec9f486ded05def39ad https://review.opendev.org/693556
12:19:02 sean-k-mooney stephenfin: im not sure if that is correct
12:19:23 sean-k-mooney if you dont set the device_type in the alias at all i think it will work with type-PF
12:19:44 sean-k-mooney the device_type is not required in the alias as far as i remember
12:19:56 sean-k-mooney if it is present it definetly needs to match
12:20:24 sean-k-mooney i need to test some sriov stuff later today so ill try this if i have time otherwise ill do it on monday
12:21:25 shilpasd gibi: hi, can you please elaborate on point 'add compute nodes to the aggregates of shared RP aggregates' regarding Shared NFS, so that will think on designing the same
12:22:56 sean-k-mooney the product id will be differente for the PF vs VF so if you have the correct product id and no VF are allocated i think it just works but again that is just going off memory i have not tried this in a year or two
12:25:28 shilpasd gibi: at driver level, how ww will get aggragtes associated to shared RP and will map those to compute node? And if compute node already has aggregates (not the same as of shared RP) are we really bother of them?
12:37:59 openstackgerrit Lee Yarwood proposed openstack/nova master: docs: Extract rescue from reboot https://review.opendev.org/694529
12:39:23 gibi shilpasd: there are two types of aggregates: there are placement aggregates https://docs.openstack.org/api-ref/placement/#resource-provider-aggregates and there are nova host aggregates https://docs.openstack.org/api-ref/compute/#host-aggregates-os-aggregates
12:39:50 gibi the disk sharing RP needs to be in the same placement aggregate as the compute RP it shares disk with
12:40:34 gibi we agreed that a new nova config option will tell the nova compute the uuid of the placement aggregate
12:40:49 gibi in which the admin added the sharing disk RP
12:40:58 openstackgerrit Lee Yarwood proposed openstack/nova master: docs: Extract rescue from reboot https://review.opendev.org/694529
12:41:26 gibi so when the compute start up it can change if its own RP (the compute PR) is already part of the placement aggregate configured in it's nova.conf
12:41:42 gibi and if not then call PUT
12:41:42 gibi /resource_providers/{uuid}/aggregatesPUT
12:41:47 gibi /resource_providers/{uuid}/aggregates
12:42:05 gibi to put it's compute PR into the that aggregate
12:43:15 gibi shilpasd: an PR can be in any number or placement aggregates
12:43:44 gibi so if the compute RP is already in other aggregates that does not really matter to us
12:45:48 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.opendev.org/693849
12:45:49 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: Boot from volume instance rescue https://review.opendev.org/694063
12:46:07 shilpasd gibi: new nova config option which holds uuid of the placement aggregate or uuid of the shared RP?
12:47:37 shilpasd and it will at compute level or libvirt level?
12:49:41 gibi shilpasd: uuid of the placement aggregate
12:49:51 gibi shilpasd: and I think the logic can be in the compute manager level
12:51:15 gibi shilpasd: have you seen this mailthread ? http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010624.html
12:52:38 shilpasd gibi: no, will go through, thnaks for sharing
12:52:55 gibi shilpasd: I think that summarized most of what we talked above
12:53:36 shilpasd gibi: yes, that's clear
12:53:56 shilpasd gibi: thank you
12:54:54 gibi shilpasd: you're welcome
12:56:14 openstackgerrit Merged openstack/nova master: Add functional recreate test for bug 1852610 https://review.opendev.org/694351
12:56:14 openstack bug 1852610 in OpenStack Compute (nova) "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Medium,In progress] https://launchpad.net/bugs/1852610 - Assigned to Matt Riedemann (mriedem)
13:45:59 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove duplicate ServerMovingTests._resize_and_check_allocations https://review.opendev.org/694538
14:18:11 openstackgerrit Matt Riedemann proposed openstack/nova stable/train: Add functional recreate test for bug 1852610 https://review.opendev.org/694544
14:18:11 openstack bug 1852610 in OpenStack Compute (nova) "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Medium,In progress] https://launchpad.net/bugs/1852610 - Assigned to Matt Riedemann (mriedem)
14:18:11 openstackgerrit Matt Riedemann proposed openstack/nova stable/train: Add functional recreate revert resize test for bug 1852610 https://review.opendev.org/694545

Earlier   Later