| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-14 | |||
| 15:13:38 | sean-k-mooney | alex_xu: well why dont we just store them with the instance as teh key | |
| 15:13:51 | alex_xu | sean-k-mooney: back to the same host resize problem :) | |
| 15:13:51 | dansmith | alex_xu: are the pmem devices actually named something specific in linux? or are they just /dev/pmem/2 and we maintain the mapping between them and the instance? | |
| 15:14:10 | artom | alex_xu, what's your series? I'm having trouble searching gerrit for patches you own | |
| 15:14:11 | mriedem | (instance, flavor) is redundant because we have instance.flavor | |
| 15:14:28 | alex_xu | dansmith: they are just /dev/pmem, but I don't get what relationship with the vpmem name at here | |
| 15:14:51 | sean-k-mooney | alex_xu: then use instace+flavor_name which is in the xml or add the uuid to the xml | |
| 15:14:52 | dansmith | alex_xu: okay so I don't understand why you have a problem accounting things with migration uuid vs. instance uuid at any given point | |
| 15:15:01 | mriedem | i think dan is saying, you claim the new_flavor devices on the dest host with the instance, and claim the old_flavor devices on the source host with the migration | |
| 15:15:26 | dansmith | sean-k-mooney: it's not a matter of storing that data, and further, flavor is an object that exists and is keyed in another database, so I think using it as a key here is a bad idea as well | |
| 15:15:33 | dansmith | mriedem: right | |
| 15:15:34 | mriedem | and by "claim the old_flavor devices on the source host with the migration" i mean, swap the mapping for the old_flavor device claim on the source host with the migration for the instance | |
| 15:15:41 | dansmith | mriedem: same pattern as we have elsewhere | |
| 15:16:13 | alex_xu | dansmith: the resize call to the dest host to claim the resource first, then how can we change the in-memory claim info in the source host? by another rpc call, back to the source host? | |
| 15:16:18 | mriedem | when you revert on the source host, you revert that mapping as well | |
| 15:16:38 | dansmith | alex_xu: you get a finish call or a revert call, so you do it then and there right? | |
| 15:16:40 | mriedem | alex_xu: do it when prep_resize casts to resize_instance on the source host? | |
| 15:16:46 | sean-k-mooney | well we could flip it and claim with the migration uuid on the dest | |
| 15:16:52 | dansmith | sean-k-mooney: no | |
| 15:17:02 | dansmith | sean-k-mooney: because then you have to change it again if they keep the instance, which is the natural path | |
| 15:17:02 | sean-k-mooney | ok | |
| 15:17:32 | sean-k-mooney | ya that is true you would have to change it in confrim resize | |
| 15:17:34 | alex_xu | dansmith: we should change to migration uuid before we add new claim for the target | |
| 15:17:42 | mriedem | or like dan said just swap the source host claim during confirm_resize or finish_revert_resize (i'm not sure you'd even need to change the latter) | |
| 15:17:57 | alex_xu | otherwise how can we distinguish the src and dest for the same host resize... | |
| 15:18:14 | mriedem | for same host resize, | |
| 15:18:21 | mriedem | you have some mapping of devices claimed on that host, right? | |
| 15:18:25 | dansmith | alex_xu: are you familiar with how we do this for placement allocations? | |
| 15:18:37 | mriedem | the instance maps to the new flavor device allocations, the migration maps to the old flavor device allocations | |
| 15:18:44 | dansmith | yes ^ | |
| 15:18:46 | alex_xu | yea, I know those thing for placement | |
| 15:19:11 | mriedem | so for same host resize move claim it seems you'd: | |
| 15:19:17 | mriedem | 1. map migration = old flavor | |
| 15:19:27 | mriedem | 2. change instance mapping to point from old to new flavor | |
| 15:19:43 | mriedem | on confirm, you remove the mapping for the migration, | |
| 15:19:47 | mriedem | on revert, you: | |
| 15:19:52 | mriedem | 1. map instance to old flavor, | |
| 15:19:54 | mriedem | 2. drop the migration mapping | |
| 15:19:56 | dansmith | don't say flavor here | |
| 15:19:58 | dansmith | but yes | |
| 15:20:07 | mriedem | flavor device allocation thing 2000 | |
| 15:20:29 | mriedem | will flavour work? | |
| 15:20:31 | dansmith | no | |
| 15:21:13 | dansmith | I think building more things using the existing pattern of using migration uuid to reserve the old resources is a good idea | |
| 15:21:28 | alex_xu | emm...I try to remember which case we said no for this in the beginning | |
| 15:21:33 | alex_xu | luyao: ^ help me | |
| 15:22:44 | luyao | alex_xu: I'm always trying... | |
| 15:25:39 | tssurya | mriedem: do you also prefer to push all the instance state checking and updates into the manager like dansmith ? since we have a lock there | |
| 15:26:01 | alex_xu | mriedem: dansmith the first rpc call to the comptute node for the resize is send to the dest src. so you can do map 'migration = old flavor' first | |
| 15:26:54 | dansmith | alex_xu: you can't, but you don't have to do the migration mapping until you hit the source for the first time | |
| 15:27:08 | dansmith | alex_xu: isn't this the exact same set of steps for allocations in placement? so the ordering should work the same way | |
| 15:27:53 | openstackgerrit | YAMAMOTO Takashi proposed openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/675021 | |
| 15:27:54 | openstackgerrit | YAMAMOTO Takashi proposed openstack/nova master: Revert "Pass migration to finish_revert_migration()" https://review.opendev.org/676442 | |
| 15:28:10 | mriedem | well, it's a little different with placement allocations since we can manage those at the top in conductor | |
| 15:28:15 | alex_xu | dansmith: we switch instance_uuid to migration_uuid in the conductor, right? | |
| 15:28:47 | dansmith | okay, that's fair | |
| 15:29:06 | mriedem | for example, if we claim the new devices on the dest in prep_resize, cast to resize_instance on the source, swap the old devices to the migration, and then something fails during the disk transfer or whatever, we won't go back to the dest to cleanup that old claim - but the RT should fix that up in a periodic | |
| 15:29:46 | mriedem | i don't know where these "claims" are stored in memory though - in the compute manager? RT? driver? | |
| 15:30:09 | mriedem | i'll just say, the RT logic is already super complex, and now it sounds like we're going to be duplicating parts of it elsewhere... | |
| 15:30:11 | alex_xu | mriedem: swap should be happened first for same host resize | |
| 15:30:32 | alex_xu | mriedem: claim store in driver | |
| 15:30:54 | mriedem | cleaning up from a failed same host resize is simpler since yo'ure on the same host, but not for different hosts | |
| 15:31:18 | sean-k-mooney | alex_xu: this is diferent then how we claim pci devices and cpu/hugepage right? | |
| 15:31:53 | mriedem | the driver doing resource tracking now.... :( | |
| 15:31:56 | sean-k-mooney | because for those i thought we stored the claims in the db via the RT rather then in memeroy | |
| 15:32:02 | mriedem | the resource tracker doing resource tracking still ... :( | |
| 15:32:02 | alex_xu | sean-k-mooney: yes, at least the vpmem and vgpu is managed by virt driver. pci and cpu managed by resource tracker | |
| 15:32:19 | mriedem | what a mess | |
| 15:32:31 | mriedem | some things in placement, some things in legacy nova tables and the RT, some things now in the driver | |
| 15:32:58 | sean-k-mooney | we should really be keepign all this in the RT untill its in placment | |
| 15:33:04 | sean-k-mooney | if its ever in placment | |
| 15:33:19 | alex_xu | sean-k-mooney: mriedem in the future, when pci and numa move to placement, then we needn't store in RT, right? then also managing in the virt driver? | |
| 15:33:19 | sean-k-mooney | put this in memory in the driver is worring | |
| 15:33:24 | sean-k-mooney | or at least complex | |
| 15:33:41 | sean-k-mooney | alex_xu: it will still be needed in some cases | |
| 15:33:42 | alex_xu | sean-k-mooney: hah, you say different with dansmith :) | |
| 15:34:03 | sean-k-mooney | we placement wont be tracking indeivusal device assignment | |
| 15:34:18 | sean-k-mooney | e.g which vf(pci addres) the vm is using | |
| 15:34:38 | sean-k-mooney | to do that we would need to create a RP per vf which we are not going to do | |
| 15:34:42 | alex_xu | sean-k-mooney: yea, that is what happen for vgpu and vpmem | |
| 15:35:15 | dansmith | I'm not sure what sean-k-mooney said that is different than me | |
| 15:35:24 | sean-k-mooney | so the "assignment" infomation will always need to live in nova | |
| 15:35:47 | sean-k-mooney | the tally count of how many are avaliable wil be in placment | |
| 15:36:02 | dansmith | sean-k-mooney: when we previously discussed this, I wanted to avoid nova storing a mapping between the actual pmem device and the instance in our database, for a specific reason | |
| 15:36:08 | alex_xu | I think dansmith said we use libvirt to persistent the assigment of devices, not DB. sean-k-mooney is talk about we still need the DB | |
| 15:37:04 | dansmith | right, so we have the mapping between instances and pmem devices stored in the libvirt xml | |
| 15:37:04 | sean-k-mooney | well currently we regenerate teh xmls on lots of operations so storign the mapping in the xml will be invasive | |
| 15:37:28 | dansmith | whatever, I give up, do whatever ya | |
| 15:37:31 | dansmith | 'll want | |
| 15:37:35 | sean-k-mooney | it will be there implictly i guess | |
| 15:37:55 | alex_xu | dansmith: no... | |
| 15:38:06 | sean-k-mooney | dansmith: we dont need to store it in the db provide we will never use it in a filter | |
| 15:38:26 | sean-k-mooney | we only need to store the pci info in the db to use it with the numa/pci passthough filters | |
| 15:38:41 | sean-k-mooney | same for the numa toplogy blob | |
| 15:39:05 | sean-k-mooney | we could caluate them locally on the host and keep it in meory otherwise | |
| 15:39:40 | mriedem | what happens when i need a weigher to pick hosts with more or less allocated pmems?! | |
| 15:39:47 | sean-k-mooney | so if the only schduling for vpmem is done via placmeent then the assignment could be track via the xml | |
| 15:40:13 | mriedem | or pmem affinity | |
| 15:40:18 | sean-k-mooney | mriedem: we would either need to call placement for the data or we cant | |
| 15:40:55 | mriedem | btw, are there a fair number of rhosp users using vgpus now that you're on queens? | |