| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-31 | |||
| 07:21:06 | gibi | the duplicate project entry (due to a race) makes the new single transaction session invalid and later operations are failed in the same transaction hence the sqlalchemy.exc.InvalidRequestError: This session is in 'inactive' state, due to the SQL transaction being rolled back; no further SQL can be emitted within this transaction. | |
| 07:21:14 | gibi | that is my running assumption now | |
| 07:22:45 | bauzas | yup, I agree | |
| 07:23:01 | gibi | I have to go offline for ~2 hours | |
| 07:23:16 | bauzas | I need to filter 100+ emails but I guess you found something | |
| 07:23:16 | gibi | gthiemonge: could you please file a bug for placement about it? | |
| 07:23:38 | gthiemonge | gibi: yes, thanks! | |
| 07:25:25 | gibi | bauzas: https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/handlers/util.py#L56 this is the first exception | |
| 07:26:15 | gibi | and that is the second Aug 30 13:07:04.208104 nested-virt-ubuntu-focal-ovh-bhs1-0026183953 devstack@placement-api.service[89049]: ERROR placement.fault_wrap proj = project_obj.Project.get_by_external_id(ctx, project_id) | |
| 07:26:22 | gibi | sorry wrong buffer | |
| 07:26:28 | gibi | so the second https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/handlers/util.py#L58 | |
| 07:26:44 | gibi | this is strange I don't think consumer types changed this exact logic | |
| 07:28:05 | gibi | https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/util.py#b65 yepp the logic is the same in the from state https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/util.py#b65 | |
| 07:29:56 | gthiemonge | gibi: https://storyboard.openstack.org/#!/story/2009159 | |
| 07:30:18 | gibi | but yes, the Project.create() and the Project.get_by_external_id was two separate transaction before the consumer_types | |
| 07:30:36 | gibi | this is the start of the new and only transaction https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/allocation.py#424 | |
| 07:30:45 | gibi | gthiemonge: thanks | |
| 07:35:25 | gibi | right now I don't know how to solve this. as we need one big transaction but we should not stop due to a project creation race | |
| 07:35:53 | gibi | maybe we need to re-try the whole transaction from the start if the race is caugth | |
| 07:36:29 | gibi | and now I go offline, back around 11:30 CEST | |
| 07:42:55 | opendevreview | Federico Ressi proposed openstack/nova master: Write requests failure details to log https://review.opendev.org/c/openstack/nova/+/806683 | |
| 08:03:46 | opendevreview | Federico Ressi proposed openstack/nova master: [WIP] Investigate on Nova quota limits problem https://review.opendev.org/c/openstack/nova/+/806683 | |
| 08:37:49 | opendevreview | Merged openstack/nova master: docs: admin/networking rename neutron_tunneled to neutron_tunnel https://review.opendev.org/c/openstack/nova/+/806197 | |
| 09:46:55 | gibi | bauzas, melwitt: I will try to add a reproduction test for https://storyboard.openstack.org/#!/story/2009159 | |
| 09:47:28 | bauzas | ack | |
| 09:49:43 | gibi | stephenfin, lyarwood: a quick re-review on the pps series would be much appreciated :) (starting here https://review.opendev.org/c/openstack/nova/+/800086 ) | |
| 09:54:32 | lyarwood | ack done | |
| 09:58:57 | kevinz | kashyap: sbauza: Hi, bother again :-) could you help to review this live migration patch?live migration on arm64 patch, https://review.opendev.org/c/openstack/nova/+/763928, the comments has been addressed. really appreciated! | |
| 10:01:57 | kevinz | bauzas: ^^, thanks for reviewing | |
| 10:02:45 | gibi | lyarwood: thanks | |
| 10:13:48 | kashyap | kevinz: Hi | |
| 10:14:04 | kashyap | kevinz: Just catching up after PTO; will look today | |
| 10:23:43 | bauzas | kevinz: ditto as kashyap, I'm just back from a long period of inactivity | |
| 10:24:38 | bauzas | this reminds me actually... core reviews would be appreciated for https://review.opendev.org/c/openstack/nova/+/802918 | |
| 10:24:42 | bauzas | and the series | |
| 10:30:20 | kevinz | kashyap: bauzas: Thanks! Really appreciated :-) | |
| 10:54:50 | opendevreview | Merged openstack/nova master: nova-manage: Introduce volume show, refresh, get_connector commands https://review.opendev.org/c/openstack/nova/+/800634 | |
| 10:54:54 | opendevreview | Merged openstack/placement master: Refactor consumer type methods for readability https://review.opendev.org/c/openstack/placement/+/806035 | |
| 10:55:06 | opendevreview | Merged openstack/nova stable/wallaby: Reduce mocking in test_reject_open_redirect for compat https://review.opendev.org/c/openstack/nova/+/803092 | |
| 11:21:02 | gibi | /12 | |
| 11:46:50 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add reproducer for Project creation race bug https://review.opendev.org/c/openstack/placement/+/806730 | |
| 11:47:47 | gibi | melwitt, bauzas: ^^ it is the reproducer to the issue octavia folks reported today. I could not create a real race in the func test env so I cheated a but | |
| 11:47:50 | gibi | bit | |
| 12:22:51 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Address nits for configurable-instance-hostnames series https://review.opendev.org/c/openstack/nova/+/806735 | |
| 12:24:02 | gibi | sean-k-mooney: what is the expected behavior of a sriov interface attach in an env where the PCI devices are part of numa0 but the instance cpus are part of numa1? | |
| 12:25:11 | sean-k-mooney | it depend on if the vm has a numa toplogy and the numa policy | |
| 12:25:25 | sean-k-mooney | if the vm has no numa toplogy its fine it should boot | |
| 12:25:44 | sean-k-mooney | if it it has cpu pinning it should not boot unless you use pci_numa_toplogy=prefer | |
| 12:26:02 | sean-k-mooney | which will prefer a host where affinity can be acive but not enforce it on the host | |
| 12:26:31 | gibi | sean-k-mooney: what about pci_numa_toplogy=legacy? | |
| 12:27:21 | sean-k-mooney | legacy means if the device report numa info its enforced but if it does not it allowed regardless of the cpu selected | |
| 12:27:53 | sean-k-mooney | so legacy which is the default rquired numa affinity if we have numa info for the device and the cpus/memory | |
| 12:28:20 | gibi | OK, so In my internal case the VM has topology due to cpu pinning and they not specify policy so they get the legacy one | |
| 12:28:33 | sean-k-mooney | yep | |
| 12:29:00 | gibi | and the pci device has numa info so the basically get strict affinity | |
| 12:29:38 | gibi | with hw:pci_numa_affinity_policy=preferred they would get best effort affinity | |
| 12:29:39 | sean-k-mooney | yep | |
| 12:29:52 | sean-k-mooney | we have test for this in whitebox by the way | |
| 12:29:54 | sean-k-mooney | https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute/test_sriov.py#L221-L432 | |
| 12:30:17 | sean-k-mooney | we also have func tests for this i think in nova | |
| 12:30:38 | sean-k-mooney | preffer is basically currenlty implemtned as a weigher | |
| 12:30:51 | sean-k-mooney | i have a very long running todo to make prefer prefer numa nodes on the comptue too | |
| 12:31:00 | sean-k-mooney | its a simple chagne just never got around to do it | |
| 12:32:17 | sean-k-mooney | to implement prefer on the compute node we litrally just need an else on https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2201-L2205 that sorts in teh reverse order | |
| 12:33:24 | sean-k-mooney | gibi: for backwards compatiablity reasons we made legacy the default but prefer is arguable a more user friendly default | |
| 12:33:34 | gibi | ack thanks | |
| 12:34:04 | gibi | sean-k-mooney: do we have the per port numa affinity supported by nova? | |
| 12:34:19 | gibi | i.e. the nova side of https://bugs.launchpad.net/neutron/+bug/1886798 | |
| 12:34:26 | sean-k-mooney | yes | |
| 12:34:37 | sean-k-mooney | i need to check which release but we do in wallaby | |
| 12:34:41 | sean-k-mooney | maybe victoria | |
| 12:35:00 | gibi | so for interface attach the user could set the policy on the port if the instace was booted without the flavor extra sepc | |
| 12:35:03 | gibi | spec | |
| 12:35:07 | sean-k-mooney | https://github.com/openstack/nova-specs/blob/master/specs/wallaby/implemented/port-scoped-sriov-numa-affinity.rst | |
| 12:35:15 | gibi | cool, thanks | |
| 12:35:59 | sean-k-mooney | ya i think we covered that usecase when you added sriov attach support too | |
| 12:37:02 | sean-k-mooney | both feature were added in wallaby and im 99% sure that we pass the posicy when trying to claim the pci device on attach and will reject it if the numa affintiy cant be achived | |
| 12:37:14 | sean-k-mooney | but you can relax the policy with the neutron port | |
| 12:37:29 | sean-k-mooney | so the vm can default to legacy or strict and then you can add a port with prefer | |
| 12:39:57 | gibi | cool. I do remember passing policy around in sriov attach impl | |
| 12:40:31 | sean-k-mooney | yep we do https://github.com/openstack/nova/blob/649b2452da07c25e1251c6185552278f91b7c517/nova/compute/manager.py#L7788-L7794 | |
| 12:40:53 | sean-k-mooney | hehe ya was just finding that in the code | |
| 12:43:06 | sean-k-mooney | that gets the flavor/image policy then we override that with the port policy if set https://github.com/openstack/nova/blob/649b2452da07c25e1251c6185552278f91b7c517/nova/network/neutron.py#L2023-L2026 | |
| 12:43:33 | sean-k-mooney | https://github.com/openstack/nova/blob/649b2452da07c25e1251c6185552278f91b7c517/nova/network/neutron.py#L2084 | |
| 12:43:50 | sean-k-mooney | https://github.com/openstack/nova/blob/649b2452da07c25e1251c6185552278f91b7c517/nova/network/neutron.py#L2162-L2164 | |
| 12:44:13 | sean-k-mooney | gibi: i rememebr discussing the precednece of these two during the spec review | |
| 12:47:15 | opendevreview | sean mooney proposed openstack/nova master: Add 'hw:vif_multiqueue_enabled' flavor extra spec https://review.opendev.org/c/openstack/nova/+/792356 | |
| 12:47:16 | opendevreview | sean mooney proposed openstack/nova master: docs: Document virtio-net multiqueue https://review.opendev.org/c/openstack/nova/+/792362 | |
| 12:47:16 | opendevreview | sean mooney proposed openstack/nova master: Move 'hw:pmu', 'hw_pmu' parsing to nova.virt.hardware https://review.opendev.org/c/openstack/nova/+/792364 | |
| 12:48:06 | sean-k-mooney | stephenfin: i have adressed my nits regarding the acidental cahnge form 403 to 500 and resolved the merge conflict otherwise this is as you left it | |
| 12:48:42 | sean-k-mooney | i think those should be good to merge now if people have time to review | |
| 12:59:44 | kashyap | kevinz: Looked at it; I see you've also done real tests with it. I don't see anything glaring in there. Also the conditional is AArch64 specific. Thx for reworking. | |
| 13:02:10 | bauzas | sean-k-mooney: correct me if I'm wrong, but I see a last revision of generic mdevs that is only due for fixing pep8 mypy issues because we were not explicitely returning None ? | |
| 13:02:36 | bauzas | plus the fact you moved the documentation bit out of the last change into a specific WIP change, right? | |
| 13:02:37 | sean-k-mooney | bauzas: yes | |
| 13:03:22 | sean-k-mooney | stephenfin: was ok with adressing the doc update in a seperate change so suggested we split it out to not block the feature | |
| 13:03:33 | sean-k-mooney | since the doc update can happen after FF | |
| 13:04:23 | sean-k-mooney | bauzas: effectivly what stephenfin would like is if we have a stuf file that explains vgpu are now implemented in terms of generic mdevs and then links to mdev doc | |
| 13:04:45 | sean-k-mooney | bauzas: if we want to have specific vgpu docs we can keep them int vgpu doc | |
| 13:05:26 | sean-k-mooney | bauzas: but ya i did not actully change any of your codes functionality form where you left it just the mypy update | |