Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-08-05
11:38:44 sean-k-mooney oh its after this
11:38:57 sean-k-mooney you fix things up later?
11:39:05 gibi https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1557
11:39:35 sean-k-mooney ah https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1609-L1610
11:39:43 gibi first heal_allocation was added my mriedem then I added port allocation healing to it when the QoS was first done
11:40:02 gibi now I need to fix up the QoS support due to the format change in the resource request as part of the pps feature
11:40:27 sean-k-mooney i was actully wonderign if we use resources_from_flavor for any other operation in nova
11:40:34 gibi fortunetly not
11:40:38 gibi that would be a huge bug
11:40:44 sean-k-mooney yep
11:41:05 gibi we should move that util function to the manage.py to show it is not used by any other code
11:41:11 sean-k-mooney ah i see nov amanage trhen unit and func tests
11:42:10 sean-k-mooney ya i guess we could
11:43:33 sean-k-mooney ok so this would affect vGPU, Cyborg and vPMEM
11:43:48 gibi vPMEM is also not on the root provider?
11:44:13 sean-k-mooney i belive we have multipel namespaces fo differnet size so i think its nested
11:44:53 sean-k-mooney it may also impact inventories created via provider.yaml
11:45:29 gibi sean-k-mooney: vpmem is on the root according to this doc https://docs.openstack.org/nova/latest/admin/virtual-persistent-memory.html
11:45:58 gibi does provider.yaml today support creating child RPs?
11:46:08 gibi or inventories on child rps?
11:46:55 sean-k-mooney ah right we are using different RC for the different namespaces so that does not need to be nested
11:47:33 gibi we don't have tests but I assume that anytign that is on the root healed correctly
11:47:54 gibi I mean we have tests for basic mem, disk, vcpu and port QoS but not to the others like vpmem
11:48:57 sean-k-mooney gibi: so i tdoes not look like we can crete nested rps with the file but we can specifcy the uuid
11:49:09 sean-k-mooney so if you precreate the RPs the it can manage them
11:49:38 gibi ok, so provider yaml can add inventories to existing childs like cyborg, vgpu or QoS childs. So that is also affected
11:49:42 sean-k-mooney so it wont use nested RP by default but there is nothing to prevent you doing that
11:49:55 sean-k-mooney yes exactly
11:50:09 gibi but there is no way I can detect that from the manage CLI. I can detect vgpu and cyborg dev profile in the flavor and blow up
11:50:52 sean-k-mooney well im wondering why we cant try and retrive the structure form placment initally
11:51:41 gibi sean-k-mooney: so you mean if the instance has nested allocation then we dont try to heal it
11:51:44 gibi sean-k-mooney: that can be done
11:51:58 gibi sean-k-mooney: but we cannot detect that an instance would need a nested allocation if that allocation is missig
11:51:59 sean-k-mooney well that could be a first step
11:52:24 gibi and the whole reason of heal allocation is to heal missing allocations :)
11:52:37 sean-k-mooney but i was wondering if we could derive where the allocation should come from using the tree structure of plamcnet and its current allocation if they existis
11:53:07 sean-k-mooney gibi: well for the we kind of can
11:53:26 sean-k-mooney if we se it uses a CUSTOM_CYBORG_THING
11:54:05 sean-k-mooney and we look at the RP tree for the host and see that is not on the root RP we know it need to be healed usign the nested RP inventory
11:54:32 sean-k-mooney if and only if that RC exist on only one nested RP we can heal it
11:54:41 gibi cyborg is requested via device profile name in the flavor so that can be detected easier, for any CUSTOM_FOO your idea is viable
11:55:11 sean-k-mooney but if the same RC exits on multiple RPS really only the virt dirver would be able to figure out what rp is correct
11:55:48 sean-k-mooney e.g. it would have to use inform form the libvirt domain or similar to try and determin which pGPU the vGPU was allocated form
11:56:23 gibi sean-k-mooney: yes, the ambiguity cannot be resolved in the CLI, we have that thing already implemented (rejected) for the QoS healing
11:56:46 gibi there the VF - PF - port - PF RP relationship cannot be disambiguated in the CLI
11:57:30 sean-k-mooney part of me thinks we should be storing some addtion info in our db to allow this
11:57:40 sean-k-mooney but im not sure what that should be
12:01:26 sean-k-mooney what annoys me about this problem is even if we save a copy of the inital allocation summeries and the placment query that we used to generate it im not conviced that is sufficent to reconstuct the allocations remotely
12:01:46 gibi I stop here now. I will add some blocking and documentation to the CLI about vgpu and cyborg dev profile to prevent damage. then I will continue adding QoS support. other can take up removing the vgpu and cyborg block by implementing support
12:02:09 sean-k-mooney gibi: it kind of feels like the only way to do this would be to have nova-manage call the virt dirver over rpc to have it fix it
12:02:31 sean-k-mooney gibi: ack
12:02:44 gibi yes, the full support most probably would need that
12:03:00 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Mark live migration jobs as non-voting due to bug #1912310 https://review.opendev.org/c/openstack/nova/+/803585
12:03:34 songwenping sean-k-mooney: we have a problem on cyborg that you may know, please give some tips if you have free time.
12:03:47 songwenping https://review.opendev.org/c/openstack/cyborg/+/797403 this patch backport to victoria, but it requires oslo.db==10.0.0 for cybort, because this patch https://review.opendev.org/c/openstack/oslo.db/+/792124 resolved duplicate key error for mysql. how can we fix the cyborg tempest?
12:04:08 songwenping https://review.opendev.org/c/openstack/cyborg/+/797403 this patch backport to victoria, but it requires oslo.db==10.0.0 for cyborg tempest, because this patch https://review.opendev.org/c/openstack/oslo.db/+/792124 resolved duplicate key error for mysql. how can we fix the cyborg tempest?
12:06:56 sean-k-mooney ok so you are gettign duplicate DeviceProfile uuids in this case 977806ca-4e8e-40c2-aa3a-09cef2903336
12:08:07 sean-k-mooney and this is currently how you create your device profiles https://github.com/openstack/cyborg/blob/1052efe93b5e7aa351b1f50cfe80f504dcf48b72/cyborg/db/sqlalchemy/api.py#L499-L517
12:10:51 sean-k-mooney so this is the contraing that is filing
12:10:53 sean-k-mooney pymysql.err.IntegrityError: (1062, "Duplicate entry 'fpga_same_test' for key 'device_profiles.uniq_device_profiles0name'")
12:14:48 sean-k-mooney ok i see
12:14:52 sean-k-mooney https://review.opendev.org/c/openstack/oslo.db/+/792124/6/oslo_db/sqlalchemy/exc_filters.py
12:15:11 sean-k-mooney so thet way that code is ment to work is it parses device_profiles.uniq_device_profiles0name
12:16:00 sean-k-mooney and it should extract the unique constrait by firsts discarding eveything before uniq_ leaving device_profiles0name
12:16:34 sean-k-mooney the it splits that on the 0 to get the table device_profiles and columns in this case name
12:17:10 sean-k-mooney so the unique constrati you were expecting was device_profiles name not the uuid
12:19:12 sean-k-mooney without that fix it was using all colums as unique constriats? i guess at least that the oslo db level
12:19:42 sean-k-mooney oh i see
12:20:11 sean-k-mooney In mysql 8.0.19 , Duplicate key error information is extended to
12:20:13 sean-k-mooney include the table name of the key.Previously, duplicate key error
12:20:15 sean-k-mooney information included only the key value and key name.
12:21:46 bauzas gibi: sorry was out for lunch
12:22:05 sean-k-mooney songwenping: im surpised that affect https://github.com/openstack/cyborg/blob/1052efe93b5e7aa351b1f50cfe80f504dcf48b72/cyborg/db/sqlalchemy/api.py#L511-L516
12:22:15 bauzas gibi: about the VGPU healed allocations, well, we already verify the VGPU RC for the audit command but we don't do this for the heal_allocations
12:22:55 bauzas gibi: also, given we would now use other custom RCs, maybe we could also modify both the audit and heal_allocs commands to have a new attribute for telling which RCs to look at
12:25:15 songwenping sean-k-mooney: yes, the mysql version is update.
12:25:18 sean-k-mooney songwenping: without the oslo.db patch teh colum will be named "device_profiles.uniq_device_profiles0name" with it it will be "name"
12:25:20 sean-k-mooney oh i see
12:25:43 sean-k-mooney ok i get what is happening now so e.columns is a dict
12:26:01 sean-k-mooney if 'name' in e.columns: is checkking if there is a key that exactly match
12:26:36 sean-k-mooney the keey has chaned to now have the table name prefixed
12:26:54 sean-k-mooney so the fix you can do in cyborg is to add an elif
12:27:47 songwenping right, the e.columns isnot ['name'] any more.
12:29:34 sean-k-mooney ya so we just need to make the comparisone a little more robost
12:30:44 songwenping if i make the name and id all conflict, the e.columns is ['id'] and isnot ['id','name']
12:31:32 sean-k-mooney https://paste.opendev.org/show/807904/
12:31:47 sean-k-mooney i think this will work ^
12:32:33 songwenping this is good for now.
12:32:59 sean-k-mooney https://paste.opendev.org/show/807905/
12:33:03 sean-k-mooney or maybe that
12:33:38 sean-k-mooney add an else just in case we have a conflicat that is not on name or uuid although it would be treated as a uuid conflict today
12:34:36 sean-k-mooney songwenping: but ya the other way to do this is to preporcess the columns dict and stip the table prefix
12:35:57 songwenping does other projects have the same problems?
12:36:32 songwenping i see only cyborg distingush the uuid and name conflict.
12:36:41 sean-k-mooney i think we might define unique constraints differently then cyborg does
12:39:20 sean-k-mooney songwenping: so this is the other way to fix it https://paste.opendev.org/show/807908/
12:39:57 sean-k-mooney all that has changed here is instead of using e.colums directly i have generated a new columns dict and then the exsitng if else just uses that
12:39:59 songwenping this is same as the oslo.db does.
12:40:07 sean-k-mooney yep more or less
12:40:32 songwenping but whether it depends on mysql version

Earlier   Later