| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-05 | |||
| 09:27:39 | bauzas | gibi: my last 2 days are dedicated to Gerrit looking fwiw | |
| 09:28:53 | gibi | bauzas: I'm affraid to ask you to look at the packet rate as you will dissappear for 3 weeks so I will be blocked waiting you to come back and re-review just before FF. but if you have time... | |
| 09:29:10 | bauzas | that's understandable | |
| 09:29:34 | bauzas | one day, feature freezes won't happen on end of Augusts | |
| 09:30:22 | bauzas | fun fact, my wife is always grumbling about the fact our release cadence is always hitting our usual resting periods :) | |
| 09:31:24 | gibi | this end of summer FF is not idea, the W FF happened at a better time for me at least | |
| 09:31:50 | opendevreview | Sylvain Bauza proposed openstack/nova master: Change the admin-visible logs for mdev support https://review.opendev.org/c/openstack/nova/+/803378 | |
| 09:31:50 | opendevreview | Sylvain Bauza proposed openstack/nova master: Provide the mdev class for every PCI device https://review.opendev.org/c/openstack/nova/+/802918 | |
| 09:31:51 | opendevreview | Sylvain Bauza proposed openstack/nova master: Provide and use other RCs for mdevs if needed https://review.opendev.org/c/openstack/nova/+/803233 | |
| 09:31:51 | opendevreview | Sylvain Bauza proposed openstack/nova master: Expose the mdev class https://review.opendev.org/c/openstack/nova/+/801743 | |
| 09:31:52 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP: Cleanup GPU vs. mdev wording https://review.opendev.org/c/openstack/nova/+/803379 | |
| 09:32:20 | bauzas | (just a rebase for other changes but https://review.opendev.org/c/openstack/nova/+/803378 which fixes two nits) | |
| 09:32:58 | bauzas | gibi: right, but honestly, only europeans feel this being problematic so I'm OK to live with it | |
| 09:35:38 | gibi | ^^ looks good to me | |
| 09:35:50 | songwenping | sean-k-mooney: https://review.opendev.org/c/openstack/cyborg/+/797403 this patch backport to victoria, but it requires oslo.db==10.0.0, because this patch https://review.opendev.org/c/openstack/oslo.db/+/792124 resolved duplicate key error for mysql. how can we fix the cyborg tempest? | |
| 09:38:09 | opendevreview | Lee Yarwood proposed openstack/nova master: Add regression test for bug 1938326 https://review.opendev.org/c/openstack/nova/+/802801 | |
| 09:38:09 | opendevreview | Lee Yarwood proposed openstack/nova master: compute: Query the service group API within check_instance_host https://review.opendev.org/c/openstack/nova/+/802317 | |
| 09:59:19 | lyarwood | gibi: https://bugs.launchpad.net/nova/+bug/1912310/comments/7 - TIL about logsearch, is that a local tool or something I could use? | |
| 10:00:01 | gibi | lyarwood: I would say highly experimental https://github.com/gibizer/zuul-log-search | |
| 10:00:17 | lyarwood | awesome | |
| 10:01:04 | gibi | I'm plannig for the time when logstash will be gone | |
| 10:01:44 | gibi | this logsearch needs decent net bandwidth and a plenty of local space but it does not need a always running service | |
| 10:03:44 | gibi | right now I'm trying to add capability for matching build queries against stored search patterns to help figuring out why a job failed | |
| 10:04:33 | kashyap | lyarwood: Once again ... thanks for the 'nova-mange libvirt update_machine_type'! :) | |
| 10:07:21 | gibi | lyarwood: here are my config dir too with some bug patterns https://github.com/gibizer/zuul-log-search-config | |
| 10:08:19 | gibi | lyarwood: and as always, pull requests are welcome :) | |
| 10:13:08 | stephenfin | lyarwood, bauzas, gibi: Reworking this hostname API change. I was asked to drop the 'OS-EXT-SRV-ATTR:' prefix for the 'hostname' field from responses to make it align with the request. Do we want to drop all of these prefixes? | |
| 10:13:46 | stephenfin | so 'OS-EXT-SRV-ATTR:hostname' becomes 'hostname', 'OS-SRV-USG:launched_at' becomes 'launched_at', ... | |
| 10:14:09 | stephenfin | I ask because when I'm doing one and already spinning up the microversion machinery, it makes sense for me to do them all | |
| 10:14:10 | gibi | doesn't make much sense to keep them, they are there only for historical reasons as far as I remember | |
| 10:14:23 | stephenfin | yes, due to the old-school extension framework | |
| 10:14:56 | gibi | kill them with fire! | |
| 10:15:06 | stephenfin | OS-EXT-SRV => os-extended-server-attributes extension; OS-SRV-USG => os-server-usage extension etc. | |
| 10:15:18 | stephenfin | (I only learned that today from https://wiki.openstack.org/wiki/NovaAPIv2tov3) | |
| 10:16:05 | stephenfin | 🔥🔥🔥 | |
| 10:16:39 | lyarwood | stephenfin: so is this behind a microversion? | |
| 10:16:53 | stephenfin | absolutely | |
| 10:17:01 | lyarwood | right cool then burn away | |
| 10:59:28 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip volume and block live migrations due to bug #1912310 https://review.opendev.org/c/openstack/nova/+/803585 | |
| 11:00:02 | lyarwood | ^ gibi / stephenfin ; okay I've had enough of https://bugs.launchpad.net/nova/+bug/1912310 borking things in the check and gate queues, lets skip things and ask the QEMU folks to look at this | |
| 11:02:53 | stephenfin | agree | |
| 11:04:09 | gibi | lyarwood: do we have left any live migration coverage? | |
| 11:04:56 | lyarwood | ah, not in the nova-live-migration job because it isn't using shared storage | |
| 11:04:59 | lyarwood | let me respin this | |
| 11:08:54 | gibi | bauzas: I'm just wondering did we ever updated nova-manage placement heal_allocation to support healing vGPUs? | |
| 11:13:49 | gibi | bauzas: I think if today we have an instance with vGPU request and the heal_allocation CLI is used it will try to allocate VGPU resource from the root RP that will fail | |
| 11:14:22 | gibi | I guess the same is true for flavor based cyborg devices | |
| 11:15:40 | gibi | the code simply assumes that everything coming from the flavor need to be allocated from the root RP https://github.com/openstack/nova/blob/2ffd9738602531e93495a1feca76bbb687c3e72c/nova/cmd/manage.py#L1700-L1705 | |
| 11:17:56 | gibi | https://github.com/openstack/nova/blob/2ffd9738602531e93495a1feca76bbb687c3e72c/nova/scheduler/utils.py#L607-L612 | |
| 11:18:08 | gibi | this is a scary TODO^^ | |
| 11:20:09 | gibi | amending the cyborg part. If the flavor has cyborg device profile extra_spec then that is simply ignored by the heal_allocation code | |
| 11:20:33 | gibi | with a --force used it can mean that a healthy cyborg allocation is dropped from an instance | |
| 11:20:53 | gibi | /o\ | |
| 11:36:41 | sean-k-mooney | ya... that is not ideal | |
| 11:37:11 | sean-k-mooney | gibi: that also affects QOS | |
| 11:37:22 | gibi | sean-k-mooney: not, there is explicit QoS support there | |
| 11:37:34 | gibi | I'm just about to extend that with the packet rate handling | |
| 11:37:44 | gibi | when I detected the missing pieces | |
| 11:38:44 | sean-k-mooney | oh its after this | |
| 11:38:57 | sean-k-mooney | you fix things up later? | |
| 11:39:05 | gibi | https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1557 | |
| 11:39:35 | sean-k-mooney | ah https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1609-L1610 | |
| 11:39:43 | gibi | first heal_allocation was added my mriedem then I added port allocation healing to it when the QoS was first done | |
| 11:40:02 | gibi | now I need to fix up the QoS support due to the format change in the resource request as part of the pps feature | |
| 11:40:27 | sean-k-mooney | i was actully wonderign if we use resources_from_flavor for any other operation in nova | |
| 11:40:34 | gibi | fortunetly not | |
| 11:40:38 | gibi | that would be a huge bug | |
| 11:40:44 | sean-k-mooney | yep | |
| 11:41:05 | gibi | we should move that util function to the manage.py to show it is not used by any other code | |
| 11:41:11 | sean-k-mooney | ah i see nov amanage trhen unit and func tests | |
| 11:42:10 | sean-k-mooney | ya i guess we could | |
| 11:43:33 | sean-k-mooney | ok so this would affect vGPU, Cyborg and vPMEM | |
| 11:43:48 | gibi | vPMEM is also not on the root provider? | |
| 11:44:13 | sean-k-mooney | i belive we have multipel namespaces fo differnet size so i think its nested | |
| 11:44:53 | sean-k-mooney | it may also impact inventories created via provider.yaml | |
| 11:45:29 | gibi | sean-k-mooney: vpmem is on the root according to this doc https://docs.openstack.org/nova/latest/admin/virtual-persistent-memory.html | |
| 11:45:58 | gibi | does provider.yaml today support creating child RPs? | |
| 11:46:08 | gibi | or inventories on child rps? | |
| 11:46:55 | sean-k-mooney | ah right we are using different RC for the different namespaces so that does not need to be nested | |
| 11:47:33 | gibi | we don't have tests but I assume that anytign that is on the root healed correctly | |
| 11:47:54 | gibi | I mean we have tests for basic mem, disk, vcpu and port QoS but not to the others like vpmem | |
| 11:48:57 | sean-k-mooney | gibi: so i tdoes not look like we can crete nested rps with the file but we can specifcy the uuid | |
| 11:49:09 | sean-k-mooney | so if you precreate the RPs the it can manage them | |
| 11:49:38 | gibi | ok, so provider yaml can add inventories to existing childs like cyborg, vgpu or QoS childs. So that is also affected | |
| 11:49:42 | sean-k-mooney | so it wont use nested RP by default but there is nothing to prevent you doing that | |
| 11:49:55 | sean-k-mooney | yes exactly | |
| 11:50:09 | gibi | but there is no way I can detect that from the manage CLI. I can detect vgpu and cyborg dev profile in the flavor and blow up | |
| 11:50:52 | sean-k-mooney | well im wondering why we cant try and retrive the structure form placment initally | |
| 11:51:41 | gibi | sean-k-mooney: so you mean if the instance has nested allocation then we dont try to heal it | |
| 11:51:44 | gibi | sean-k-mooney: that can be done | |
| 11:51:58 | gibi | sean-k-mooney: but we cannot detect that an instance would need a nested allocation if that allocation is missig | |
| 11:51:59 | sean-k-mooney | well that could be a first step | |
| 11:52:24 | gibi | and the whole reason of heal allocation is to heal missing allocations :) | |
| 11:52:37 | sean-k-mooney | but i was wondering if we could derive where the allocation should come from using the tree structure of plamcnet and its current allocation if they existis | |
| 11:53:07 | sean-k-mooney | gibi: well for the we kind of can | |
| 11:53:26 | sean-k-mooney | if we se it uses a CUSTOM_CYBORG_THING | |
| 11:54:05 | sean-k-mooney | and we look at the RP tree for the host and see that is not on the root RP we know it need to be healed usign the nested RP inventory | |
| 11:54:32 | sean-k-mooney | if and only if that RC exist on only one nested RP we can heal it | |
| 11:54:41 | gibi | cyborg is requested via device profile name in the flavor so that can be detected easier, for any CUSTOM_FOO your idea is viable | |
| 11:55:11 | sean-k-mooney | but if the same RC exits on multiple RPS really only the virt dirver would be able to figure out what rp is correct | |