| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-12 | |||
| 12:45:27 | sean-k-mooney | well the first validation happens in the numa toplogy filter | |
| 12:46:07 | sean-k-mooney | but its not really validation we try to use the values to filter a host and that either work or it doesnt | |
| 12:46:34 | sean-k-mooney | so i think doing it in the prefilter which is even earlier in scheduling is fine | |
| 12:46:51 | alex_xu | wo do, let me find the link | |
| 12:47:28 | alex_xu | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L659 | |
| 12:48:37 | sean-k-mooney | oh ok i really never knew that was there | |
| 12:48:47 | sean-k-mooney | in that case sure you can add the same validate for pmem | |
| 12:48:49 | alex_xu | so we can resolve the conflict of hw:pmem and resource:* somewhere like that | |
| 12:49:00 | alex_xu | \o/ finally find something sean-k-mooney doesn't know | |
| 12:49:58 | alex_xu | so I can override resources:* when hw:pmem present. | |
| 12:49:59 | sean-k-mooney | hehe :) there are plenty of things i dont know like how to use a mac without breaking it or how to spell litrally everything | |
| 12:50:06 | aspiers | XD | |
| 12:50:12 | alex_xu | hah | |
| 12:50:46 | aspiers | sean-k-mooney: maybe not, but I bet you can help me with this problem | |
| 12:50:53 | alex_xu | the last question is what should I do for the user only specify the resources:*. so I can prase the resources:* and put them into instance.vpmems. | |
| 12:50:55 | aspiers | https://review.opendev.org/#/c/644565/5/nova/virt/libvirt/driver.py@1625 | |
| 12:51:26 | aspiers | I need to change _sev_required() so that it checks the resource allocations for the instance (the MEMORY_ENCRYPTION_CONTEXT class) | |
| 12:51:53 | aspiers | but the callers don't have access to the resource allocations | |
| 12:51:58 | sean-k-mooney | alex_xu: i think we shoudl require uses to use hw:pmem personally as we might want to use the resouce class for something else in the future | |
| 12:51:58 | aspiers | AFAICS | |
| 12:52:23 | aspiers | they have access to the request context and the instance, but not the allocations | |
| 12:52:25 | sean-k-mooney | alex_xu: im thinking of the case where we might want to back guest ram by PMEM | |
| 12:52:34 | sean-k-mooney | e.g. as an alternive to hugepages | |
| 12:52:39 | sean-k-mooney | or file backed memeory | |
| 12:53:07 | sean-k-mooney | aspiers: looking | |
| 12:53:19 | alex_xu | good idea | |
| 12:53:34 | aspiers | I found _instance_to_allocations_dict() in scheduler/client/report.py which maybe could extract the allocations from the instance, but I don't know if that's right | |
| 12:53:47 | alex_xu | do you mean reject pmem resource class in extra spec resource:*? | |
| 12:53:55 | aspiers | also, nothing uses that method any more, since Iec02942d38 removed the last callers of it | |
| 12:54:46 | sean-k-mooney | alex_xu: we could but i was thinking just not populating the instance.vpmems from it | |
| 12:55:34 | alex_xu | that will do an allocation in the placement, but in the end, the instance won't attach the pmem | |
| 12:55:39 | sean-k-mooney | aspiers: do you need to cehck the resouce allocation or jsut the flavor | |
| 12:55:46 | aspiers | sean-k-mooney: the allocation | |
| 12:55:54 | aspiers | that's what efried said yesterday | |
| 12:56:37 | sean-k-mooney | im not sure why checking the request_sepc or flavor/image would not be enough | |
| 12:56:52 | aspiers | sean-k-mooney: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-06-11.log.html#t2019-06-11T16:49:51 | |
| 13:00:08 | aspiers | sean-k-mooney: but maybe when efried wrote that he didn't know that _sev_required() needs to be called from multiple locations in the driver code which don't have access to allocations | |
| 13:00:18 | aspiers | I guess I could set another instance variable | |
| 13:00:22 | sean-k-mooney | the instance contains the request_sepc and the request spec contains the request_resouces | |
| 13:00:29 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L98 | |
| 13:01:04 | aspiers | but requested_resources is different from allocations, no? | |
| 13:01:12 | sean-k-mooney | it is | |
| 13:01:24 | sean-k-mooney | and it does not currenly hav the info you want see the comment above | |
| 13:01:43 | aspiers | right | |
| 13:01:48 | sean-k-mooney | im just wondering is that better to use in the future | |
| 13:01:51 | aspiers | in fact efried wants me to help fix that | |
| 13:02:29 | aspiers | I'm guessing that code which wants to know whether to add SEV bits to the libvirt XML config should be checking what got allocated, not just what got requested | |
| 13:02:50 | sean-k-mooney | they should be the same thing | |
| 13:03:10 | sean-k-mooney | if placemnet returned an allcoation candatae then eveything that was requested was allcoated | |
| 13:03:32 | aspiers | I guess in the MEM_ENCRYPTION_CONTEXT case where only 1 unit is requested they have to be the same | |
| 13:03:34 | sean-k-mooney | looking at the instance object i dont see the allcoation stored anywhere | |
| 13:03:43 | aspiers | yeah I couldn't either | |
| 13:05:15 | aspiers | I found https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L2434 | |
| 13:07:18 | aspiers | that passes allocations to spawn() via https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L2203 | |
| 13:08:01 | sean-k-mooney | ya we do pass them to the driver | |
| 13:08:03 | sean-k-mooney | https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L2207 | |
| 13:08:16 | aspiers | that's 4 lines below my link :) | |
| 13:08:19 | aspiers | same thing | |
| 13:08:43 | aspiers | but then allocs doesn't get passed around to all the smaller methods called by spawn() | |
| 13:09:06 | sean-k-mooney | right its only used to get teh mdevs | |
| 13:09:07 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3237 | |
| 13:09:49 | aspiers | so maybe I can set an _sev_required instance variable in spawn() | |
| 13:10:26 | aspiers | or somewhere underneath that | |
| 13:10:26 | sean-k-mooney | well you could jsut pass the allcoation to _get_guest_xml | |
| 13:10:45 | aspiers | that's a good idea | |
| 13:10:57 | aspiers | I guess I only need it for constructing XML | |
| 13:11:02 | aspiers | so that would scope it correctly | |
| 13:11:20 | aspiers | hopefully efried will come online before I disappear down this rathole :) | |
| 13:12:33 | sean-k-mooney | its still a little messy | |
| 13:12:43 | sean-k-mooney | i think really we want to add the allcotion to the instance | |
| 13:13:02 | sean-k-mooney | that way you dont need to pass a new param all over the place | |
| 13:13:07 | aspiers | yeah | |
| 13:13:16 | sean-k-mooney | and i think more things will need the allocation in the future anyway | |
| 13:13:54 | sean-k-mooney | we track things like pci device allcotation in teh instance already so i think that is the best path forward. | |
| 13:14:18 | aspiers | sounds reasonable but I can't really comment | |
| 13:15:17 | sean-k-mooney | efried: when you are online any objection to adding the allocations form placmenet to the instnace object? | |
| 13:21:46 | artom | sean-k-mooney, sorta like network_info in info_cache? | |
| 13:22:14 | sean-k-mooney | not quite although kind of | |
| 13:23:11 | sean-k-mooney | artom: we store the list of pci devices allocated to an instance in the instance | |
| 13:23:13 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L200 | |
| 13:23:33 | sean-k-mooney | so that we can use the same ones on reboot | |
| 13:23:58 | sean-k-mooney | it would be somewhat like storing the vifs in the info_cache | |
| 13:24:23 | sean-k-mooney | but keeping the allocation in the insantce so we dont have to look them up on swapn or reboot makes sense | |
| 13:24:59 | sean-k-mooney | *start or reboot | |
| 13:30:15 | sean-k-mooney | ... | |
| 13:30:31 | sean-k-mooney | i think our vgpu code is busted | |
| 13:31:21 | sean-k-mooney | bauzas: did you ever test stopping an instance with vgpus | |
| 13:31:29 | sean-k-mooney | then starting it again | |
| 13:32:48 | sean-k-mooney | power_on in the libvirt driver delegate to _hard_reboot which assumes the xml still exists to get the mdevs from | |
| 13:32:50 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2897-L2900 | |
| 13:33:01 | sean-k-mooney | power_off calls destroy | |
| 13:33:33 | sean-k-mooney | so if you shutdown an instance with vgpus and then start it again it wont have vgpus... | |
| 13:39:19 | kashyap | Speaking of assigned devices, somepeople might find this tool ("mdevctl") useful: https://github.com/awilliam/mdevctl | |
| 13:39:28 | kashyap | More info and background about it here: | |
| 13:39:40 | kashyap | https://www.redhat.com/archives/libvir-list/2019-May/msg00722.html | |
| 13:39:44 | kashyap | ("mdevctl: A shoestring mediated device management and persistence utility") | |
| 13:40:09 | kashyap | It's a shell tool. So, be gentle with your eyes :D | |
| 13:40:11 | sean-k-mooney | ya maybe | |
| 13:41:44 | sean-k-mooney | they are not that hard to manage i personlaly would praobly do it with systemd rather then udev personally | |
| 13:42:24 | kashyap | I don't know, if the VFIO maintainer suggests that, then I'd go the route they suggest. | |