| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-31 | |||
| 01:35:41 | opendevreview | Merged openstack/nova master: [func test] refactor assertPortMatchesAllocation https://review.opendev.org/c/openstack/nova/+/792458 | |
| 01:35:49 | opendevreview | Merged openstack/nova master: [func test] refactor asserts in qos tests https://review.opendev.org/c/openstack/nova/+/798930 | |
| 01:35:57 | opendevreview | Merged openstack/nova master: [func test] ports with both bw and pps resources https://review.opendev.org/c/openstack/nova/+/792394 | |
| 05:34:44 | opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | |
| 05:34:44 | opendevreview | melanie witt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/c/openstack/nova/+/712139 | |
| 05:34:45 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 05:34:45 | opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | |
| 05:34:46 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 05:34:46 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 05:34:47 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 05:34:47 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 05:34:48 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 05:34:49 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 07:03:02 | bauzas | good morning Nova | |
| 07:04:18 | gibi | bauzas: welcome back | |
| 07:04:31 | gibi | bauzas: have you managed to recharge your batteries? | |
| 07:05:39 | gthiemonge | Hi Folks, do you know some recent issues with placement? the Octavia gates have been failing since the end of last week, with some exceptions in the placement-api logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-placement-api.txt | |
| 07:05:41 | bauzas | gibi: let me see | |
| 07:05:44 | bauzas | gibi: 80% :p | |
| 07:05:55 | gibi | bauzas: that looks good :) | |
| 07:06:56 | gthiemonge | logstash shows that it started on 2021-08-25 in many projects: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22placement.exception.ProjectNotFound%3A%20No%20such%20project%5C%22 | |
| 07:06:58 | bauzas | gibi: you as well ? | |
| 07:07:41 | bauzas | gthiemonge: looking | |
| 07:07:48 | gibi | gthiemonge: looking | |
| 07:08:07 | gibi | bauzas: sure, I had a nice weekend with a good book | |
| 07:09:16 | gibi | gthiemonge: 2/3rd of the hits are actually successful job runs | |
| 07:09:44 | gibi | gthiemonge: are you sure this is the root cause of the failed ones? | |
| 07:10:18 | gthiemonge | gibi: yeah, for Octavia at least | |
| 07:10:36 | gibi | gthiemonge: could you link an octavia failre please? | |
| 07:12:13 | gthiemonge | gibi: yes sure | |
| 07:12:39 | gthiemonge | octavia worker logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-o-cw.txt | |
| 07:12:49 | bauzas | gibi: do you know why we persist the project UUIDs ? for the unified-limits or for the consumer types ? | |
| 07:13:07 | gthiemonge | error is at Aug 30 13:07:06.582519 with a "{'code': 500, 'created': '2021-08-30T13:07:05Z', 'message': 'No valid host was found. There are not enough hosts available.'," | |
| 07:13:14 | gibi | bauzas: we have projecti and user id for a log time in placement | |
| 07:13:21 | gthiemonge | nova scheduler: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-n-sch.txt | |
| 07:13:43 | gthiemonge | at Aug 30 13:07:04.212245 there's a Internal Server Error | |
| 07:13:57 | gibi | bauzas: but I don't think placement uses it directly | |
| 07:14:10 | gibi | bauzas: so possible yes, for unified limits so you can group usages per project | |
| 07:14:16 | gibi | gthiemonge: thanks, looking | |
| 07:15:33 | gthiemonge | gibi: if that helps: it fails on Octavia when we request 2 server creations at the same time, it doesn't fail when creating one server | |
| 07:15:59 | gibi | gthiemonge: thanks, this will be a race | |
| 07:16:53 | gibi | we changed transactional behavior of allocation update in https://review.opendev.org/q/topic:bp/support-consumer-types | |
| 07:17:49 | bauzas | gibi: surely for unified limits, but l looked at the series and nothing merged yet | |
| 07:18:36 | bauzas | gibi: yeah, hence my confusion : we merged stuff for consumer types but none of them touched the projects table | |
| 07:18:55 | gibi | bauzas: here you can query usages per project https://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages | |
| 07:19:11 | bauzas | oh now I remember | |
| 07:19:25 | bauzas | gibi: gotcha, thanks | |
| 07:19:26 | gibi | bauzas: but consumer_types changes how the allocation transaction handling is done | |
| 07:19:54 | gibi | bauzas: before consumer_type we had actually two transactions per allocation update, after consumer_types we have one | |
| 07:21:06 | gibi | the duplicate project entry (due to a race) makes the new single transaction session invalid and later operations are failed in the same transaction hence the sqlalchemy.exc.InvalidRequestError: This session is in 'inactive' state, due to the SQL transaction being rolled back; no further SQL can be emitted within this transaction. | |
| 07:21:14 | gibi | that is my running assumption now | |
| 07:22:45 | bauzas | yup, I agree | |
| 07:23:01 | gibi | I have to go offline for ~2 hours | |
| 07:23:16 | gibi | gthiemonge: could you please file a bug for placement about it? | |
| 07:23:16 | bauzas | I need to filter 100+ emails but I guess you found something | |
| 07:23:38 | gthiemonge | gibi: yes, thanks! | |
| 07:25:25 | gibi | bauzas: https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/handlers/util.py#L56 this is the first exception | |
| 07:26:15 | gibi | and that is the second Aug 30 13:07:04.208104 nested-virt-ubuntu-focal-ovh-bhs1-0026183953 devstack@placement-api.service[89049]: ERROR placement.fault_wrap proj = project_obj.Project.get_by_external_id(ctx, project_id) | |
| 07:26:22 | gibi | sorry wrong buffer | |
| 07:26:28 | gibi | so the second https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/handlers/util.py#L58 | |
| 07:26:44 | gibi | this is strange I don't think consumer types changed this exact logic | |
| 07:28:05 | gibi | https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/util.py#b65 yepp the logic is the same in the from state https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/util.py#b65 | |
| 07:29:56 | gthiemonge | gibi: https://storyboard.openstack.org/#!/story/2009159 | |
| 07:30:18 | gibi | but yes, the Project.create() and the Project.get_by_external_id was two separate transaction before the consumer_types | |
| 07:30:36 | gibi | this is the start of the new and only transaction https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/allocation.py#424 | |
| 07:30:45 | gibi | gthiemonge: thanks | |
| 07:35:25 | gibi | right now I don't know how to solve this. as we need one big transaction but we should not stop due to a project creation race | |
| 07:35:53 | gibi | maybe we need to re-try the whole transaction from the start if the race is caugth | |
| 07:36:29 | gibi | and now I go offline, back around 11:30 CEST | |
| 07:42:55 | opendevreview | Federico Ressi proposed openstack/nova master: Write requests failure details to log https://review.opendev.org/c/openstack/nova/+/806683 | |
| 08:03:46 | opendevreview | Federico Ressi proposed openstack/nova master: [WIP] Investigate on Nova quota limits problem https://review.opendev.org/c/openstack/nova/+/806683 | |
| 08:37:49 | opendevreview | Merged openstack/nova master: docs: admin/networking rename neutron_tunneled to neutron_tunnel https://review.opendev.org/c/openstack/nova/+/806197 | |
| 09:46:55 | gibi | bauzas, melwitt: I will try to add a reproduction test for https://storyboard.openstack.org/#!/story/2009159 | |
| 09:47:28 | bauzas | ack | |
| 09:49:43 | gibi | stephenfin, lyarwood: a quick re-review on the pps series would be much appreciated :) (starting here https://review.opendev.org/c/openstack/nova/+/800086 ) | |
| 09:54:32 | lyarwood | ack done | |
| 09:58:57 | kevinz | kashyap: sbauza: Hi, bother again :-) could you help to review this live migration patch?live migration on arm64 patch, https://review.opendev.org/c/openstack/nova/+/763928, the comments has been addressed. really appreciated! | |
| 10:01:57 | kevinz | bauzas: ^^, thanks for reviewing | |
| 10:02:45 | gibi | lyarwood: thanks | |
| 10:13:48 | kashyap | kevinz: Hi | |
| 10:14:04 | kashyap | kevinz: Just catching up after PTO; will look today | |
| 10:23:43 | bauzas | kevinz: ditto as kashyap, I'm just back from a long period of inactivity | |
| 10:24:38 | bauzas | this reminds me actually... core reviews would be appreciated for https://review.opendev.org/c/openstack/nova/+/802918 | |
| 10:24:42 | bauzas | and the series | |
| 10:30:20 | kevinz | kashyap: bauzas: Thanks! Really appreciated :-) | |
| 10:54:50 | opendevreview | Merged openstack/nova master: nova-manage: Introduce volume show, refresh, get_connector commands https://review.opendev.org/c/openstack/nova/+/800634 | |
| 10:54:54 | opendevreview | Merged openstack/placement master: Refactor consumer type methods for readability https://review.opendev.org/c/openstack/placement/+/806035 | |
| 10:55:06 | opendevreview | Merged openstack/nova stable/wallaby: Reduce mocking in test_reject_open_redirect for compat https://review.opendev.org/c/openstack/nova/+/803092 | |
| 11:21:02 | gibi | /12 | |
| 11:46:50 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add reproducer for Project creation race bug https://review.opendev.org/c/openstack/placement/+/806730 | |
| 11:47:47 | gibi | melwitt, bauzas: ^^ it is the reproducer to the issue octavia folks reported today. I could not create a real race in the func test env so I cheated a but | |
| 11:47:50 | gibi | bit | |
| 12:22:51 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Address nits for configurable-instance-hostnames series https://review.opendev.org/c/openstack/nova/+/806735 | |
| 12:24:02 | gibi | sean-k-mooney: what is the expected behavior of a sriov interface attach in an env where the PCI devices are part of numa0 but the instance cpus are part of numa1? | |
| 12:25:11 | sean-k-mooney | it depend on if the vm has a numa toplogy and the numa policy | |
| 12:25:25 | sean-k-mooney | if the vm has no numa toplogy its fine it should boot | |
| 12:25:44 | sean-k-mooney | if it it has cpu pinning it should not boot unless you use pci_numa_toplogy=prefer | |
| 12:26:02 | sean-k-mooney | which will prefer a host where affinity can be acive but not enforce it on the host | |
| 12:26:31 | gibi | sean-k-mooney: what about pci_numa_toplogy=legacy? | |
| 12:27:21 | sean-k-mooney | legacy means if the device report numa info its enforced but if it does not it allowed regardless of the cpu selected | |