Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-12
17:08:51 aspiers I'll do whatever you gurus think is best :)
17:08:57 efried let's see what mriedem thinks
17:09:00 aspiers OK
17:09:37 dansmith we have virtapi for compute drivers to ask compute manger for help. compute manager already has an in-memory cache of a bunch of this information, so letting the driver ask manager for allocation info for an instance when it needs it, which can be a readthrough cache operation would also be reasonable
17:10:05 efried ah, that could work. What's that virtapi called?
17:10:41 dansmith storing this on the instance is pointless to me because I think we always need to get it from placement to be sure it's right I think, and the instance record is already a massive thing we shoot across the stressed RPC bus with a ton of extra crap we don't need 90% of the time
17:11:02 efried ComputeDriver.virtapi...
17:11:06 dansmith but, as I said, it matters only what mriedem thinks, so ... no point in even discussing until he opines
17:11:28 aspiers RPC bloat sounds like a very valid concern
17:16:23 sean-k-mooney RPC bloat being the reason not to put it in the instance?
17:16:40 aspiers that's how I interpreted what dansmith said
17:16:47 efried This virtapi thing is pretty small right now, basically just has wait_for_instance_event
17:16:51 aspiers yeah
17:16:59 aspiers also, I don't see allocations cached in the manager
17:17:06 sean-k-mooney dansmith: the allcoation should not change over the lifetime of the instance without a server action to modify it
17:17:10 dansmith efried: it used to have a bunch of services the driver used, but has whittled down
17:17:29 sean-k-mooney e.g. it will only cahnge on move operattion/resize/rebulds/or attaches/detaches
17:17:57 dansmith sean-k-mooney: in some future where I can allocate non-compute resources for the instance, the instance's allocation can change not from nova right?
17:18:02 efried but if dansmith likes the idea of extending it with allocations_for_instance, which we can cache in the manager, I'm down with that idea.
17:18:15 dansmith aspiers: no, we cache inventory AFAIK, but not allocations at the moment
17:18:26 sean-k-mooney dansmith: im notsure about that
17:18:43 sean-k-mooney are you thinking baout things like changing the bandwith allcoation via neutron
17:18:51 sean-k-mooney e.g. by changing the qos policy
17:19:17 efried aspiers: The inventory (basically all the provider tree information for all the hosts managed by this compute) is cached in the SchedulerReportClient today. But yeah, not the allocations.
17:19:40 sean-k-mooney i guess that could happen in the future
17:19:55 dansmith sean-k-mooney: I'm thinking of allocating things like a new volume, or even ephemeral things like networked secure enclave keys or something
17:20:17 sean-k-mooney dansmith: well a new volume would be a volume attach right
17:20:19 efried as long as those ^ things happen in nova, we're good, but yeah, if they happen outside of nova, weirdness.
17:20:22 dansmith sean-k-mooney: it also means our rpc messages and database footprint scales with whatever else we add into the allocation in the future
17:20:48 efried I think we have other problems to solve ("heal allocations"??) if we start allowing outside entities to modify instance allocations
17:20:56 dansmith sean-k-mooney: not always in the case of manilla or similar higher level services
17:21:01 sean-k-mooney that is true altheough we have a 64k userdata blob in there already
17:21:11 efried I'm happy with the virtapi solution personally.
17:21:18 efried I don't even think we need to bother caching it, at least initially.
17:21:28 sean-k-mooney for mania the filesystem is not own by the instnace
17:21:30 efried Since it's on demand and should be a pretty rare ask.
17:21:53 sean-k-mooney the instace can have acess too it but its not tied to the lifetime of the instnace
17:22:30 sean-k-mooney efried: i do kindof dislike the idea that if placement goes offline you cant reboot an instnace
17:22:32 dansmith sean-k-mooney: okay, you understand I'm talking about high-level services that sit on top of infra that would have per-instance allocations right?
17:22:45 sean-k-mooney dansmith: yes
17:23:01 sean-k-mooney and i get that it could be extendted that way in the future
17:24:51 sean-k-mooney without storing the allcoation somewhere in nova then placment becmoes a singel point of failture for starting or stroping vms if they use sev or vgpus
17:25:33 dansmith I definitely don't like that we'd depend on placement to reboot an instance or do other things, but I'd rather us not depend on the placement *data* in that case,
17:25:38 dansmith and not just solve it by replicating everything everywhere
17:26:01 efried yeah, when I found out we rebuild the domxml every time we reboot I was like whaaa?
17:26:16 sean-k-mooney well that makes sense in some ways
17:26:26 efried it must
17:26:30 efried in some ways
17:26:31 sean-k-mooney that is not really the root cause of the dependcy in the vgpu case
17:26:32 dansmith is this all coming up because we've already translated a request into a "will have SEV" and the only way we can think to know that later is persist the whole allocation?
17:26:46 efried dansmith: more or less
17:26:46 sean-k-mooney no
17:26:49 efried heh
17:26:53 efried anti-jinx
17:26:58 sean-k-mooney we can look at the request spec
17:27:03 dansmith efried: well, that's a bad reason to do that, IMHO
17:27:05 efried dansmith: It's not "the only way we can think to know that later"
17:27:06 sean-k-mooney and use the request for sev
17:27:17 efried we could reinvoke the logic we used to decide the first time
17:27:32 dansmith efried: point being, let's store what we need about that decision and not just couple the guts of the virt driver into the placement data
17:27:35 efried by looking at the flavor & image - which we have access to via the Instance obj, right?
17:27:46 sean-k-mooney ya we can compute it form the info in the request spec which these fucntion already have acess too
17:28:09 dansmith because the other thing is, allocations are somewhat tied to a microversion in placement, and while it seems hard to imagine the core structure changing, we've changed a lot of the external data structures of placement since we started
17:29:45 sean-k-mooney so once the comment is address i personally would have check the requestd_resouces object in the request spec https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L93-L100
17:30:06 sean-k-mooney but we also have acess to the image and flavor via the instance yes
17:31:14 dansmith efried: re: [10:27:05] <efried>dansmith: It's not "the only way we can think to know that later"
17:31:36 dansmith efried: I know not the only way, but "the way", collapsed to "only" for dramatic effect
17:36:56 sean-k-mooney so the instance is already passed to all the places we need to check if the instace requested sev
17:37:47 sean-k-mooney so why dont we jsut hav ea single function that takes the instance object and determins if sev is needed and reuse that everywhere we care.
17:38:22 sean-k-mooney that could live in nova.virt.hardware.yp with the numa stuff?
17:38:58 openstackgerrit Artom Lifshitz proposed openstack/nova master: Revert resize: wait for events according to hybrid plug https://review.opendev.org/644881
17:38:58 openstackgerrit Artom Lifshitz proposed openstack/nova master: WIP: Include direct-physical in compute manager events check https://review.opendev.org/664431
17:38:59 openstackgerrit Artom Lifshitz proposed openstack/nova master: [DNM] testing bug/1813789 revert resize events https://review.opendev.org/664442
17:40:11 sean-k-mooney like is_realtime_enabled https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1436-L1438
17:40:34 openstackgerrit Eric Fried proposed openstack/nova master: WIP: ComputeVirtAPI.allocations_for_instance https://review.opendev.org/664986
17:40:39 efried aspiers: for grins, you can try this out ^
17:40:45 efried dansmith: this what you had in mind?
17:41:23 dansmith efried: mostly
17:41:43 efried I explained why no caching in the commit message
17:41:46 sean-k-mooney efried: but is there any reason to have to check the allotion the athoritive info is stored in the image/flavor not the allcoation
17:42:09 efried sean-k-mooney: just to not have to duplicate that logic.
17:42:20 efried but in this case, no, not really.
17:42:24 sean-k-mooney an api call is a lot more expensive
17:42:28 dansmith is a single call to that less expensive than calling the placement?
17:42:29 efried Accessing the flavor/image meta from Instance is cheap
17:42:54 efried dansmith: assuming we already have the flavor & image available in the Instance object, it's all local, yeah.
17:43:00 efried well
17:43:06 efried some of it involves pawing at sysfs
17:43:13 sean-k-mooney we do
17:43:14 dansmith it better not,
17:43:24 dansmith because the api nodes' sysfs wouldn't be the same as compute
17:43:58 sean-k-mooney we dont need to look at sysfs for schduling
17:44:04 efried right, the compute pawed at sysfs to expose the inventory, so by virtue of having the MEM_ENC_CTX inventory at all, we know we did the sysfs stuff already.
17:44:09 sean-k-mooney we may or may not for xml generation
17:44:21 dansmith efried: ack, I see
17:44:25 efried but how would I know that on the compute without looking at placement? :P
17:44:42 sean-k-mooney efried: know what?
17:45:00 efried know that I can SEV
17:45:11 sean-k-mooney because its asked for in teh flavor/image
17:45:17 efried no

Earlier   Later