Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-14
14:58:58 alex_xu yea, same question
14:59:03 dansmith ages ago we had *some* CI system (not vmware) that could do same-host cold migration because it would find bugs in that, but maybe it was xen or smokestack or something
14:59:31 dansmith just seems like a terribly deep assumption to make
14:59:43 mriedem idk, the xenapi driver doesn't support it either https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L67
14:59:53 mriedem only the vmware driver does
15:00:23 mriedem i also don't understand how init_host is connected to a move claim
15:00:33 dansmith alex_xu: explain again why you need to tie flavors to the pmem at all for accounting?
15:00:51 dansmith mriedem: this is for host restart, rebuilding state for what has been given to what
15:01:09 dansmith mriedem: but I don't understand why we need to correlate that with anything other than instance uuid, which is done in the xml, AFAIK
15:01:30 mriedem yeah....i thought the pmem "allocation" type data was stored in instance_extra or something
15:01:33 mriedem keyed by instance uuid
15:01:42 alex_xu for same host resize, we records the claim in memory. but when we confirm and revert the resize, we need unclaim for dst claim or src claim. to distinguish that, only the instance_uuid isn't enough. I use instance_uuid + flavor.id.
15:01:57 alex_xu dansmith: mriedem sean-k-mooney ^
15:02:25 dansmith alex_xu: isn't the whole point of this pmem stuff that the contents get moved with the instance if it crosses to another host?
15:02:42 dansmith (tbh I don't know what the point of this is, but...)
15:02:48 mriedem if you're adding a new move claim interface to the virt driver, why wouldn't you just pass the information down to the driver that the RT already knows - i.e. the old and new flavor and if it's a same-host resize?
15:03:40 mriedem if we have to piece information together from what the hypervisor says in the domain xml, we screwed up somewhere in the design :)
15:03:42 alex_xu when you confirm resize, you neeed unclaim the assigment for the source host. But for the same host, if only use instance_uuid, when you unclaim by instance_uuid, both the source and dest claim will be unclaim
15:03:58 alex_xu same problem for revert resize
15:05:01 dansmith alex_xu: so change the source allocation to be by migration uuid like we do for allocations?
15:05:59 alex_xu dansmith: the problem is when we revert the migration uuid back to instance uuid. for the resize to different host, those claim on two host's memory, we can't change the migration uuid back to instance uuid when the resize finish
15:06:20 dansmith alex_xu: why not?
15:06:32 dansmith if you can change it from instance to migration, why can't you change it back?
15:08:10 alex_xu dansmith: we claim the vpmem on the dest host(prep_resize), at that point, we need to change the source host claim's instance_uuid to migration_uuid.
15:08:27 dansmith ...right
15:09:31 alex_xu yea...there is no way to change source host claim
15:11:26 dansmith are you saying there's no way to rename a pmem device or whatever you're naming (instance, flavor) ?
15:12:13 artom alex_xu, I feel like we should coordinate on the claim thing
15:12:41 artom Because IIUC your series touches on the same points as NUMA LM
15:12:59 alex_xu dansmith: when nova-compute startup, we populate all the assignment of vpmem and vgpu into memory, store those devices claim in the memory. key by (instance, flavor)
15:13:15 alex_xu artom: yea
15:13:38 sean-k-mooney alex_xu: well why dont we just store them with the instance as teh key
15:13:51 dansmith alex_xu: are the pmem devices actually named something specific in linux? or are they just /dev/pmem/2 and we maintain the mapping between them and the instance?
15:13:51 alex_xu sean-k-mooney: back to the same host resize problem :)
15:14:10 artom alex_xu, what's your series? I'm having trouble searching gerrit for patches you own
15:14:11 mriedem (instance, flavor) is redundant because we have instance.flavor
15:14:28 alex_xu dansmith: they are just /dev/pmem, but I don't get what relationship with the vpmem name at here
15:14:51 sean-k-mooney alex_xu: then use instace+flavor_name which is in the xml or add the uuid to the xml
15:14:52 dansmith alex_xu: okay so I don't understand why you have a problem accounting things with migration uuid vs. instance uuid at any given point
15:15:01 mriedem i think dan is saying, you claim the new_flavor devices on the dest host with the instance, and claim the old_flavor devices on the source host with the migration
15:15:26 dansmith sean-k-mooney: it's not a matter of storing that data, and further, flavor is an object that exists and is keyed in another database, so I think using it as a key here is a bad idea as well
15:15:33 dansmith mriedem: right
15:15:34 mriedem and by "claim the old_flavor devices on the source host with the migration" i mean, swap the mapping for the old_flavor device claim on the source host with the migration for the instance
15:15:41 dansmith mriedem: same pattern as we have elsewhere
15:16:13 alex_xu dansmith: the resize call to the dest host to claim the resource first, then how can we change the in-memory claim info in the source host? by another rpc call, back to the source host?
15:16:18 mriedem when you revert on the source host, you revert that mapping as well
15:16:38 dansmith alex_xu: you get a finish call or a revert call, so you do it then and there right?
15:16:40 mriedem alex_xu: do it when prep_resize casts to resize_instance on the source host?
15:16:46 sean-k-mooney well we could flip it and claim with the migration uuid on the dest
15:16:52 dansmith sean-k-mooney: no
15:17:02 sean-k-mooney ok
15:17:02 dansmith sean-k-mooney: because then you have to change it again if they keep the instance, which is the natural path
15:17:32 sean-k-mooney ya that is true you would have to change it in confrim resize
15:17:34 alex_xu dansmith: we should change to migration uuid before we add new claim for the target
15:17:42 mriedem or like dan said just swap the source host claim during confirm_resize or finish_revert_resize (i'm not sure you'd even need to change the latter)
15:17:57 alex_xu otherwise how can we distinguish the src and dest for the same host resize...
15:18:14 mriedem for same host resize,
15:18:21 mriedem you have some mapping of devices claimed on that host, right?
15:18:25 dansmith alex_xu: are you familiar with how we do this for placement allocations?
15:18:37 mriedem the instance maps to the new flavor device allocations, the migration maps to the old flavor device allocations
15:18:44 dansmith yes ^
15:18:46 alex_xu yea, I know those thing for placement
15:19:11 mriedem so for same host resize move claim it seems you'd:
15:19:17 mriedem 1. map migration = old flavor
15:19:27 mriedem 2. change instance mapping to point from old to new flavor
15:19:43 mriedem on confirm, you remove the mapping for the migration,
15:19:47 mriedem on revert, you:
15:19:52 mriedem 1. map instance to old flavor,
15:19:54 mriedem 2. drop the migration mapping
15:19:56 dansmith don't say flavor here
15:19:58 dansmith but yes
15:20:07 mriedem flavor device allocation thing 2000
15:20:29 mriedem will flavour work?
15:20:31 dansmith no
15:21:13 dansmith I think building more things using the existing pattern of using migration uuid to reserve the old resources is a good idea
15:21:28 alex_xu emm...I try to remember which case we said no for this in the beginning
15:21:33 alex_xu luyao: ^ help me
15:22:44 luyao alex_xu: I'm always trying...
15:25:39 tssurya mriedem: do you also prefer to push all the instance state checking and updates into the manager like dansmith ? since we have a lock there
15:26:01 alex_xu mriedem: dansmith the first rpc call to the comptute node for the resize is send to the dest src. so you can do map 'migration = old flavor' first
15:26:54 dansmith alex_xu: you can't, but you don't have to do the migration mapping until you hit the source for the first time
15:27:08 dansmith alex_xu: isn't this the exact same set of steps for allocations in placement? so the ordering should work the same way
15:27:53 openstackgerrit YAMAMOTO Takashi proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/675021
15:27:54 openstackgerrit YAMAMOTO Takashi proposed openstack/nova master: Revert "Pass migration to finish_revert_migration()" https://review.opendev.org/676442
15:28:10 mriedem well, it's a little different with placement allocations since we can manage those at the top in conductor
15:28:15 alex_xu dansmith: we switch instance_uuid to migration_uuid in the conductor, right?
15:28:47 dansmith okay, that's fair
15:29:06 mriedem for example, if we claim the new devices on the dest in prep_resize, cast to resize_instance on the source, swap the old devices to the migration, and then something fails during the disk transfer or whatever, we won't go back to the dest to cleanup that old claim - but the RT should fix that up in a periodic
15:29:46 mriedem i don't know where these "claims" are stored in memory though - in the compute manager? RT? driver?
15:30:09 mriedem i'll just say, the RT logic is already super complex, and now it sounds like we're going to be duplicating parts of it elsewhere...
15:30:11 alex_xu mriedem: swap should be happened first for same host resize
15:30:32 alex_xu mriedem: claim store in driver
15:30:54 mriedem cleaning up from a failed same host resize is simpler since yo'ure on the same host, but not for different hosts
15:31:18 sean-k-mooney alex_xu: this is diferent then how we claim pci devices and cpu/hugepage right?
15:31:53 mriedem the driver doing resource tracking now.... :(
15:31:56 sean-k-mooney because for those i thought we stored the claims in the db via the RT rather then in memeroy
15:32:02 alex_xu sean-k-mooney: yes, at least the vpmem and vgpu is managed by virt driver. pci and cpu managed by resource tracker
15:32:02 mriedem the resource tracker doing resource tracking still ... :(
15:32:19 mriedem what a mess
15:32:31 mriedem some things in placement, some things in legacy nova tables and the RT, some things now in the driver

Earlier   Later