Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-08-30
14:26:57 sean-k-mooney we proably can just use devstack-tempest ill see if there is a better job we can use in the future. that is a simple fix
15:20:12 ganso elodilles, lyarwood: Hi! If you have a spare minute could you please take a quick look at the backport now for victoria? it is clean and same as the one for wallaby from last week. Thanks in advance! https://review.opendev.org/c/openstack/nova/+/806004
15:22:44 opendevreview Ghanshyam proposed openstack/nova master: Convert features not supported error to HTTPBadRequest https://review.opendev.org/c/openstack/nova/+/806294
15:46:03 elodilles ganso: +2'd. Thanks for the backport! (fyi, lyarwood is on holiday today)
15:47:09 ganso elodilles: thanks! I will ping him tomorrow =)
15:50:11 elodilles no problem :)
16:06:20 opendevreview Merged openstack/nova master: tests: Validate AZ values https://review.opendev.org/c/openstack/nova/+/801523
17:05:36 opendevreview Merged openstack/nova master: Add force kwarg to delete_allocation_for_instance https://review.opendev.org/c/openstack/nova/+/688802
17:05:45 sean-k-mooney :)
17:15:39 opendevreview Merged openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/694802
17:15:55 opendevreview Merged openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/695189
17:16:17 opendevreview Merged openstack/nova master: Reduce mocking in test_reject_open_redirect for compat https://review.opendev.org/c/openstack/nova/+/803091
17:16:25 opendevreview Merged openstack/nova master: extend_volume of libvirt/volume/iscsi should not use device_path https://review.opendev.org/c/openstack/nova/+/801003
17:38:03 opendevreview sean mooney proposed openstack/nova stable/victoria: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806626
17:39:28 sean-k-mooney gibi: elodilles by the way are we goign to backport https://review.opendev.org/c/openstack/nova/+/688802
17:56:40 opendevreview sean mooney proposed openstack/nova stable/ussuri: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806628
18:03:58 opendevreview sean mooney proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629
19:51:45 gibi sean-k-mooney: I don't konw. The consumer types feature is only on master so we have a smaller issue on stable. And that smaller issue there since stein if I remember correclty.
21:56:27 gmann gibi: added releasenotes in this https://review.opendev.org/c/openstack/nova/+/806294
#openstack-nova - 2021-08-31
00:52:21 opendevreview Merged openstack/nova master: [func test] neutron fixture for extended resource request https://review.opendev.org/c/openstack/nova/+/794306
00:52:29 opendevreview Merged openstack/nova master: Detect port-resource-request-groups neutron API extension https://review.opendev.org/c/openstack/nova/+/793618
00:52:44 opendevreview Merged openstack/nova master: Reject server create with extended resource req https://review.opendev.org/c/openstack/nova/+/793619
00:53:02 opendevreview Merged openstack/nova master: Reject server operations with extended resource req https://review.opendev.org/c/openstack/nova/+/793620
01:35:16 opendevreview Merged openstack/nova master: Add same_subtree field to RequestLevelParams https://review.opendev.org/c/openstack/nova/+/791503
01:35:25 opendevreview Merged openstack/nova master: Bump min placement microversion to 1.36 https://review.opendev.org/c/openstack/nova/+/791504
01:35:33 opendevreview Merged openstack/nova master: Support same_subtree in allocation_canadidate query https://review.opendev.org/c/openstack/nova/+/791505
01:35:41 opendevreview Merged openstack/nova master: [func test] refactor assertPortMatchesAllocation https://review.opendev.org/c/openstack/nova/+/792458
01:35:49 opendevreview Merged openstack/nova master: [func test] refactor asserts in qos tests https://review.opendev.org/c/openstack/nova/+/798930
01:35:57 opendevreview Merged openstack/nova master: [func test] ports with both bw and pps resources https://review.opendev.org/c/openstack/nova/+/792394
05:34:44 opendevreview melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142
05:34:44 opendevreview melanie witt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/c/openstack/nova/+/712139
05:34:45 opendevreview melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707
05:34:45 opendevreview melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143
05:34:46 opendevreview melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301
05:34:46 opendevreview melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749
05:34:47 opendevreview melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498
05:34:47 opendevreview melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180
05:34:48 opendevreview melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499
05:34:49 opendevreview melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271
07:03:02 bauzas good morning Nova
07:04:18 gibi bauzas: welcome back
07:04:31 gibi bauzas: have you managed to recharge your batteries?
07:05:39 gthiemonge Hi Folks, do you know some recent issues with placement? the Octavia gates have been failing since the end of last week, with some exceptions in the placement-api logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-placement-api.txt
07:05:41 bauzas gibi: let me see
07:05:44 bauzas gibi: 80% :p
07:05:55 gibi bauzas: that looks good :)
07:06:56 gthiemonge logstash shows that it started on 2021-08-25 in many projects: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22placement.exception.ProjectNotFound%3A%20No%20such%20project%5C%22
07:06:58 bauzas gibi: you as well ?
07:07:41 bauzas gthiemonge: looking
07:07:48 gibi gthiemonge: looking
07:08:07 gibi bauzas: sure, I had a nice weekend with a good book
07:09:16 gibi gthiemonge: 2/3rd of the hits are actually successful job runs
07:09:44 gibi gthiemonge: are you sure this is the root cause of the failed ones?
07:10:18 gthiemonge gibi: yeah, for Octavia at least
07:10:36 gibi gthiemonge: could you link an octavia failre please?
07:12:13 gthiemonge gibi: yes sure
07:12:39 gthiemonge octavia worker logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-o-cw.txt
07:12:49 bauzas gibi: do you know why we persist the project UUIDs ? for the unified-limits or for the consumer types ?
07:13:07 gthiemonge error is at Aug 30 13:07:06.582519 with a "{'code': 500, 'created': '2021-08-30T13:07:05Z', 'message': 'No valid host was found. There are not enough hosts available.',"
07:13:14 gibi bauzas: we have projecti and user id for a log time in placement
07:13:21 gthiemonge nova scheduler: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-n-sch.txt
07:13:43 gthiemonge at Aug 30 13:07:04.212245 there's a Internal Server Error
07:13:57 gibi bauzas: but I don't think placement uses it directly
07:14:10 gibi bauzas: so possible yes, for unified limits so you can group usages per project
07:14:16 gibi gthiemonge: thanks, looking
07:15:33 gthiemonge gibi: if that helps: it fails on Octavia when we request 2 server creations at the same time, it doesn't fail when creating one server
07:15:59 gibi gthiemonge: thanks, this will be a race
07:16:53 gibi we changed transactional behavior of allocation update in https://review.opendev.org/q/topic:bp/support-consumer-types
07:17:49 bauzas gibi: surely for unified limits, but l looked at the series and nothing merged yet
07:18:36 bauzas gibi: yeah, hence my confusion : we merged stuff for consumer types but none of them touched the projects table
07:18:55 gibi bauzas: here you can query usages per project https://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages
07:19:11 bauzas oh now I remember
07:19:25 bauzas gibi: gotcha, thanks
07:19:26 gibi bauzas: but consumer_types changes how the allocation transaction handling is done
07:19:54 gibi bauzas: before consumer_type we had actually two transactions per allocation update, after consumer_types we have one
07:21:06 gibi the duplicate project entry (due to a race) makes the new single transaction session invalid and later operations are failed in the same transaction hence the sqlalchemy.exc.InvalidRequestError: This session is in 'inactive' state, due to the SQL transaction being rolled back; no further SQL can be emitted within this transaction.
07:21:14 gibi that is my running assumption now
07:22:45 bauzas yup, I agree
07:23:01 gibi I have to go offline for ~2 hours
07:23:16 gibi gthiemonge: could you please file a bug for placement about it?
07:23:16 bauzas I need to filter 100+ emails but I guess you found something
07:23:38 gthiemonge gibi: yes, thanks!
07:25:25 gibi bauzas: https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/handlers/util.py#L56 this is the first exception
07:26:15 gibi and that is the second Aug 30 13:07:04.208104 nested-virt-ubuntu-focal-ovh-bhs1-0026183953 devstack@placement-api.service[89049]: ERROR placement.fault_wrap proj = project_obj.Project.get_by_external_id(ctx, project_id)
07:26:22 gibi sorry wrong buffer
07:26:28 gibi so the second https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/handlers/util.py#L58
07:26:44 gibi this is strange I don't think consumer types changed this exact logic
07:28:05 gibi https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/util.py#b65 yepp the logic is the same in the from state https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/util.py#b65
07:29:56 gthiemonge gibi: https://storyboard.openstack.org/#!/story/2009159
07:30:18 gibi but yes, the Project.create() and the Project.get_by_external_id was two separate transaction before the consumer_types
07:30:36 gibi this is the start of the new and only transaction https://review.opendev.org/c/openstack/placement/+/679441/25/placement/handlers/allocation.py#424
07:30:45 gibi gthiemonge: thanks
07:35:25 gibi right now I don't know how to solve this. as we need one big transaction but we should not stop due to a project creation race
07:35:53 gibi maybe we need to re-try the whole transaction from the start if the race is caugth
07:36:29 gibi and now I go offline, back around 11:30 CEST
07:42:55 opendevreview Federico Ressi proposed openstack/nova master: Write requests failure details to log https://review.opendev.org/c/openstack/nova/+/806683
08:03:46 opendevreview Federico Ressi proposed openstack/nova master: [WIP] Investigate on Nova quota limits problem https://review.opendev.org/c/openstack/nova/+/806683
08:37:49 opendevreview Merged openstack/nova master: docs: admin/networking rename neutron_tunneled to neutron_tunnel https://review.opendev.org/c/openstack/nova/+/806197
09:46:55 gibi bauzas, melwitt: I will try to add a reproduction test for https://storyboard.openstack.org/#!/story/2009159
09:47:28 bauzas ack

Earlier   Later