Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-08-05
10:16:39 lyarwood stephenfin: so is this behind a microversion?
10:16:53 stephenfin absolutely
10:17:01 lyarwood right cool then burn away
10:59:28 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Skip volume and block live migrations due to bug #1912310 https://review.opendev.org/c/openstack/nova/+/803585
11:00:02 lyarwood ^ gibi / stephenfin ; okay I've had enough of https://bugs.launchpad.net/nova/+bug/1912310 borking things in the check and gate queues, lets skip things and ask the QEMU folks to look at this
11:02:53 stephenfin agree
11:04:09 gibi lyarwood: do we have left any live migration coverage?
11:04:56 lyarwood ah, not in the nova-live-migration job because it isn't using shared storage
11:04:59 lyarwood let me respin this
11:08:54 gibi bauzas: I'm just wondering did we ever updated nova-manage placement heal_allocation to support healing vGPUs?
11:13:49 gibi bauzas: I think if today we have an instance with vGPU request and the heal_allocation CLI is used it will try to allocate VGPU resource from the root RP that will fail
11:14:22 gibi I guess the same is true for flavor based cyborg devices
11:15:40 gibi the code simply assumes that everything coming from the flavor need to be allocated from the root RP https://github.com/openstack/nova/blob/2ffd9738602531e93495a1feca76bbb687c3e72c/nova/cmd/manage.py#L1700-L1705
11:17:56 gibi https://github.com/openstack/nova/blob/2ffd9738602531e93495a1feca76bbb687c3e72c/nova/scheduler/utils.py#L607-L612
11:18:08 gibi this is a scary TODO^^
11:20:09 gibi amending the cyborg part. If the flavor has cyborg device profile extra_spec then that is simply ignored by the heal_allocation code
11:20:33 gibi with a --force used it can mean that a healthy cyborg allocation is dropped from an instance
11:20:53 gibi /o\
11:36:41 sean-k-mooney ya... that is not ideal
11:37:11 sean-k-mooney gibi: that also affects QOS
11:37:22 gibi sean-k-mooney: not, there is explicit QoS support there
11:37:34 gibi I'm just about to extend that with the packet rate handling
11:37:44 gibi when I detected the missing pieces
11:38:44 sean-k-mooney oh its after this
11:38:57 sean-k-mooney you fix things up later?
11:39:05 gibi https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1557
11:39:35 sean-k-mooney ah https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1609-L1610
11:39:43 gibi first heal_allocation was added my mriedem then I added port allocation healing to it when the QoS was first done
11:40:02 gibi now I need to fix up the QoS support due to the format change in the resource request as part of the pps feature
11:40:27 sean-k-mooney i was actully wonderign if we use resources_from_flavor for any other operation in nova
11:40:34 gibi fortunetly not
11:40:38 gibi that would be a huge bug
11:40:44 sean-k-mooney yep
11:41:05 gibi we should move that util function to the manage.py to show it is not used by any other code
11:41:11 sean-k-mooney ah i see nov amanage trhen unit and func tests
11:42:10 sean-k-mooney ya i guess we could
11:43:33 sean-k-mooney ok so this would affect vGPU, Cyborg and vPMEM
11:43:48 gibi vPMEM is also not on the root provider?
11:44:13 sean-k-mooney i belive we have multipel namespaces fo differnet size so i think its nested
11:44:53 sean-k-mooney it may also impact inventories created via provider.yaml
11:45:29 gibi sean-k-mooney: vpmem is on the root according to this doc https://docs.openstack.org/nova/latest/admin/virtual-persistent-memory.html
11:45:58 gibi does provider.yaml today support creating child RPs?
11:46:08 gibi or inventories on child rps?
11:46:55 sean-k-mooney ah right we are using different RC for the different namespaces so that does not need to be nested
11:47:33 gibi we don't have tests but I assume that anytign that is on the root healed correctly
11:47:54 gibi I mean we have tests for basic mem, disk, vcpu and port QoS but not to the others like vpmem
11:48:57 sean-k-mooney gibi: so i tdoes not look like we can crete nested rps with the file but we can specifcy the uuid
11:49:09 sean-k-mooney so if you precreate the RPs the it can manage them
11:49:38 gibi ok, so provider yaml can add inventories to existing childs like cyborg, vgpu or QoS childs. So that is also affected
11:49:42 sean-k-mooney so it wont use nested RP by default but there is nothing to prevent you doing that
11:49:55 sean-k-mooney yes exactly
11:50:09 gibi but there is no way I can detect that from the manage CLI. I can detect vgpu and cyborg dev profile in the flavor and blow up
11:50:52 sean-k-mooney well im wondering why we cant try and retrive the structure form placment initally
11:51:41 gibi sean-k-mooney: so you mean if the instance has nested allocation then we dont try to heal it
11:51:44 gibi sean-k-mooney: that can be done
11:51:58 gibi sean-k-mooney: but we cannot detect that an instance would need a nested allocation if that allocation is missig
11:51:59 sean-k-mooney well that could be a first step
11:52:24 gibi and the whole reason of heal allocation is to heal missing allocations :)
11:52:37 sean-k-mooney but i was wondering if we could derive where the allocation should come from using the tree structure of plamcnet and its current allocation if they existis
11:53:07 sean-k-mooney gibi: well for the we kind of can
11:53:26 sean-k-mooney if we se it uses a CUSTOM_CYBORG_THING
11:54:05 sean-k-mooney and we look at the RP tree for the host and see that is not on the root RP we know it need to be healed usign the nested RP inventory
11:54:32 sean-k-mooney if and only if that RC exist on only one nested RP we can heal it
11:54:41 gibi cyborg is requested via device profile name in the flavor so that can be detected easier, for any CUSTOM_FOO your idea is viable
11:55:11 sean-k-mooney but if the same RC exits on multiple RPS really only the virt dirver would be able to figure out what rp is correct
11:55:48 sean-k-mooney e.g. it would have to use inform form the libvirt domain or similar to try and determin which pGPU the vGPU was allocated form
11:56:23 gibi sean-k-mooney: yes, the ambiguity cannot be resolved in the CLI, we have that thing already implemented (rejected) for the QoS healing
11:56:46 gibi there the VF - PF - port - PF RP relationship cannot be disambiguated in the CLI
11:57:30 sean-k-mooney part of me thinks we should be storing some addtion info in our db to allow this
11:57:40 sean-k-mooney but im not sure what that should be
12:01:26 sean-k-mooney what annoys me about this problem is even if we save a copy of the inital allocation summeries and the placment query that we used to generate it im not conviced that is sufficent to reconstuct the allocations remotely
12:01:46 gibi I stop here now. I will add some blocking and documentation to the CLI about vgpu and cyborg dev profile to prevent damage. then I will continue adding QoS support. other can take up removing the vgpu and cyborg block by implementing support
12:02:09 sean-k-mooney gibi: it kind of feels like the only way to do this would be to have nova-manage call the virt dirver over rpc to have it fix it
12:02:31 sean-k-mooney gibi: ack
12:02:44 gibi yes, the full support most probably would need that
12:03:00 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Mark live migration jobs as non-voting due to bug #1912310 https://review.opendev.org/c/openstack/nova/+/803585
12:03:34 songwenping sean-k-mooney: we have a problem on cyborg that you may know, please give some tips if you have free time.
12:03:47 songwenping https://review.opendev.org/c/openstack/cyborg/+/797403 this patch backport to victoria, but it requires oslo.db==10.0.0 for cybort, because this patch https://review.opendev.org/c/openstack/oslo.db/+/792124 resolved duplicate key error for mysql. how can we fix the cyborg tempest?
12:04:08 songwenping https://review.opendev.org/c/openstack/cyborg/+/797403 this patch backport to victoria, but it requires oslo.db==10.0.0 for cyborg tempest, because this patch https://review.opendev.org/c/openstack/oslo.db/+/792124 resolved duplicate key error for mysql. how can we fix the cyborg tempest?
12:06:56 sean-k-mooney ok so you are gettign duplicate DeviceProfile uuids in this case 977806ca-4e8e-40c2-aa3a-09cef2903336
12:08:07 sean-k-mooney and this is currently how you create your device profiles https://github.com/openstack/cyborg/blob/1052efe93b5e7aa351b1f50cfe80f504dcf48b72/cyborg/db/sqlalchemy/api.py#L499-L517
12:10:51 sean-k-mooney so this is the contraing that is filing
12:10:53 sean-k-mooney pymysql.err.IntegrityError: (1062, "Duplicate entry 'fpga_same_test' for key 'device_profiles.uniq_device_profiles0name'")
12:14:48 sean-k-mooney ok i see
12:14:52 sean-k-mooney https://review.opendev.org/c/openstack/oslo.db/+/792124/6/oslo_db/sqlalchemy/exc_filters.py
12:15:11 sean-k-mooney so thet way that code is ment to work is it parses device_profiles.uniq_device_profiles0name
12:16:00 sean-k-mooney and it should extract the unique constrait by firsts discarding eveything before uniq_ leaving device_profiles0name
12:16:34 sean-k-mooney the it splits that on the 0 to get the table device_profiles and columns in this case name
12:17:10 sean-k-mooney so the unique constrati you were expecting was device_profiles name not the uuid
12:19:12 sean-k-mooney without that fix it was using all colums as unique constriats? i guess at least that the oslo db level
12:19:42 sean-k-mooney oh i see
12:20:11 sean-k-mooney In mysql 8.0.19 , Duplicate key error information is extended to
12:20:13 sean-k-mooney include the table name of the key.Previously, duplicate key error
12:20:15 sean-k-mooney information included only the key value and key name.
12:21:46 bauzas gibi: sorry was out for lunch
12:22:05 sean-k-mooney songwenping: im surpised that affect https://github.com/openstack/cyborg/blob/1052efe93b5e7aa351b1f50cfe80f504dcf48b72/cyborg/db/sqlalchemy/api.py#L511-L516
12:22:15 bauzas gibi: about the VGPU healed allocations, well, we already verify the VGPU RC for the audit command but we don't do this for the heal_allocations
12:22:55 bauzas gibi: also, given we would now use other custom RCs, maybe we could also modify both the audit and heal_allocs commands to have a new attribute for telling which RCs to look at
12:25:15 songwenping sean-k-mooney: yes, the mysql version is update.
12:25:18 sean-k-mooney songwenping: without the oslo.db patch teh colum will be named "device_profiles.uniq_device_profiles0name" with it it will be "name"

Earlier   Later