| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-30 | |||
| 13:56:20 | gibi | I don't know about any full blocker | |
| 13:56:49 | sean-k-mooney | ok i was seeing the nova-ceph-multistore job fail in several patch but have not dug in to see if its the same issue | |
| 13:57:29 | sean-k-mooney | oh "'Failed to delete allocations for consumer 2064788c-9fa0-474e-a66c-72cf97b45922. .." | |
| 13:57:34 | sean-k-mooney | ya so its just that | |
| 13:58:55 | gibi | yes | |
| 13:59:10 | sean-k-mooney | any idea why it would hit the multistore job more often | |
| 13:59:36 | sean-k-mooney | it looks like that is mostly the failure so ill hold off until the force patch lands | |
| 14:00:04 | sean-k-mooney | they dont have +w anyway so they can wait | |
| 14:00:46 | elodilles | well, i don't exactly followed which job you were talking about but I am less concerned enabling new CI jobs on stable than disabling one o:) | |
| 14:00:47 | gibi | Ive no ide about the increased frequency of multistore failure | |
| 14:01:07 | gibi | elodilles: it would be a job running https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute | |
| 14:01:35 | sean-k-mooney | elodilles: which is defiedn here https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/.zuul.yaml#L31-L81 | |
| 14:02:13 | sean-k-mooney | gibi: artom is currntly actully spliting it into two jobs one that use the old cpu pinning config and the main one will only use the new way | |
| 14:02:24 | gibi | ack | |
| 14:02:35 | sean-k-mooney | gibi: right now if the reshap happens at the wong time the job breaks | |
| 14:02:45 | sean-k-mooney | so we are just going to split it to aovid that | |
| 14:02:57 | gibi | sure make sense | |
| 14:03:17 | elodilles | gibi sean-k-mooney : I guess these would land on master and then backported on the most recent stable branch, am I right? | |
| 14:05:06 | elodilles | (hmmm, it looks quite heavy, according to its parent: tempest-multinode-full-py3) | |
| 14:07:10 | gibi | personally I would take it on master first | |
| 14:16:28 | sean-k-mooney | elodilles: well it need 2 nodes but it does not run run all the tempest test becasue we use the regex to limit it | |
| 14:16:59 | sean-k-mooney | tox_envlist: all | |
| 14:17:01 | sean-k-mooney | tempest_concurrency: 1 | |
| 14:17:03 | sean-k-mooney | tempest_test_regex: ^whitebox_tempest_plugin\. | |
| 14:17:30 | sean-k-mooney | so i just ues that job to set up 2 node devstack with tempest then we just run the test from the plugin | |
| 14:20:58 | elodilles | oh, i see, i missed that | |
| 14:21:22 | sean-k-mooney | we proably could inherit form something better to make that more obvious | |
| 14:26:12 | elodilles | well, when someone reviews it thoroughly i think it'll be obvious o:) but that's true that at first glance the tempest-multinode-*full*-py3 suggests some time and resource heavy test job o:) | |
| 14:26:57 | sean-k-mooney | we proably can just use devstack-tempest ill see if there is a better job we can use in the future. that is a simple fix | |
| 15:20:12 | ganso | elodilles, lyarwood: Hi! If you have a spare minute could you please take a quick look at the backport now for victoria? it is clean and same as the one for wallaby from last week. Thanks in advance! https://review.opendev.org/c/openstack/nova/+/806004 | |
| 15:22:44 | opendevreview | Ghanshyam proposed openstack/nova master: Convert features not supported error to HTTPBadRequest https://review.opendev.org/c/openstack/nova/+/806294 | |
| 15:46:03 | elodilles | ganso: +2'd. Thanks for the backport! (fyi, lyarwood is on holiday today) | |
| 15:47:09 | ganso | elodilles: thanks! I will ping him tomorrow =) | |
| 15:50:11 | elodilles | no problem :) | |
| 16:06:20 | opendevreview | Merged openstack/nova master: tests: Validate AZ values https://review.opendev.org/c/openstack/nova/+/801523 | |
| 17:05:36 | opendevreview | Merged openstack/nova master: Add force kwarg to delete_allocation_for_instance https://review.opendev.org/c/openstack/nova/+/688802 | |
| 17:05:45 | sean-k-mooney | :) | |
| 17:15:39 | opendevreview | Merged openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/694802 | |
| 17:15:55 | opendevreview | Merged openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/695189 | |
| 17:16:17 | opendevreview | Merged openstack/nova master: Reduce mocking in test_reject_open_redirect for compat https://review.opendev.org/c/openstack/nova/+/803091 | |
| 17:16:25 | opendevreview | Merged openstack/nova master: extend_volume of libvirt/volume/iscsi should not use device_path https://review.opendev.org/c/openstack/nova/+/801003 | |
| 17:38:03 | opendevreview | sean mooney proposed openstack/nova stable/victoria: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806626 | |
| 17:39:28 | sean-k-mooney | gibi: elodilles by the way are we goign to backport https://review.opendev.org/c/openstack/nova/+/688802 | |
| 17:56:40 | opendevreview | sean mooney proposed openstack/nova stable/ussuri: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806628 | |
| 18:03:58 | opendevreview | sean mooney proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629 | |
| 19:51:45 | gibi | sean-k-mooney: I don't konw. The consumer types feature is only on master so we have a smaller issue on stable. And that smaller issue there since stein if I remember correclty. | |
| 21:56:27 | gmann | gibi: added releasenotes in this https://review.opendev.org/c/openstack/nova/+/806294 | |
| #openstack-nova - 2021-08-31 | |||
| 00:52:21 | opendevreview | Merged openstack/nova master: [func test] neutron fixture for extended resource request https://review.opendev.org/c/openstack/nova/+/794306 | |
| 00:52:29 | opendevreview | Merged openstack/nova master: Detect port-resource-request-groups neutron API extension https://review.opendev.org/c/openstack/nova/+/793618 | |
| 00:52:44 | opendevreview | Merged openstack/nova master: Reject server create with extended resource req https://review.opendev.org/c/openstack/nova/+/793619 | |
| 00:53:02 | opendevreview | Merged openstack/nova master: Reject server operations with extended resource req https://review.opendev.org/c/openstack/nova/+/793620 | |
| 01:35:16 | opendevreview | Merged openstack/nova master: Add same_subtree field to RequestLevelParams https://review.opendev.org/c/openstack/nova/+/791503 | |
| 01:35:25 | opendevreview | Merged openstack/nova master: Bump min placement microversion to 1.36 https://review.opendev.org/c/openstack/nova/+/791504 | |
| 01:35:33 | opendevreview | Merged openstack/nova master: Support same_subtree in allocation_canadidate query https://review.opendev.org/c/openstack/nova/+/791505 | |
| 01:35:41 | opendevreview | Merged openstack/nova master: [func test] refactor assertPortMatchesAllocation https://review.opendev.org/c/openstack/nova/+/792458 | |
| 01:35:49 | opendevreview | Merged openstack/nova master: [func test] refactor asserts in qos tests https://review.opendev.org/c/openstack/nova/+/798930 | |
| 01:35:57 | opendevreview | Merged openstack/nova master: [func test] ports with both bw and pps resources https://review.opendev.org/c/openstack/nova/+/792394 | |
| 05:34:44 | opendevreview | melanie witt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/c/openstack/nova/+/712139 | |
| 05:34:44 | opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | |
| 05:34:45 | opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | |
| 05:34:45 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 05:34:46 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 05:34:46 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 05:34:47 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 05:34:47 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 05:34:48 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 05:34:49 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 07:03:02 | bauzas | good morning Nova | |
| 07:04:18 | gibi | bauzas: welcome back | |
| 07:04:31 | gibi | bauzas: have you managed to recharge your batteries? | |
| 07:05:39 | gthiemonge | Hi Folks, do you know some recent issues with placement? the Octavia gates have been failing since the end of last week, with some exceptions in the placement-api logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-placement-api.txt | |
| 07:05:41 | bauzas | gibi: let me see | |
| 07:05:44 | bauzas | gibi: 80% :p | |
| 07:05:55 | gibi | bauzas: that looks good :) | |
| 07:06:56 | gthiemonge | logstash shows that it started on 2021-08-25 in many projects: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22placement.exception.ProjectNotFound%3A%20No%20such%20project%5C%22 | |
| 07:06:58 | bauzas | gibi: you as well ? | |
| 07:07:41 | bauzas | gthiemonge: looking | |
| 07:07:48 | gibi | gthiemonge: looking | |
| 07:08:07 | gibi | bauzas: sure, I had a nice weekend with a good book | |
| 07:09:16 | gibi | gthiemonge: 2/3rd of the hits are actually successful job runs | |
| 07:09:44 | gibi | gthiemonge: are you sure this is the root cause of the failed ones? | |
| 07:10:18 | gthiemonge | gibi: yeah, for Octavia at least | |
| 07:10:36 | gibi | gthiemonge: could you link an octavia failre please? | |
| 07:12:13 | gthiemonge | gibi: yes sure | |
| 07:12:39 | gthiemonge | octavia worker logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-o-cw.txt | |
| 07:12:49 | bauzas | gibi: do you know why we persist the project UUIDs ? for the unified-limits or for the consumer types ? | |
| 07:13:07 | gthiemonge | error is at Aug 30 13:07:06.582519 with a "{'code': 500, 'created': '2021-08-30T13:07:05Z', 'message': 'No valid host was found. There are not enough hosts available.'," | |
| 07:13:14 | gibi | bauzas: we have projecti and user id for a log time in placement | |
| 07:13:21 | gthiemonge | nova scheduler: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d26/761195/19/check/octavia-v2-act-stdby-dsvm-scenario/d26bf59/controller/logs/screen-n-sch.txt | |
| 07:13:43 | gthiemonge | at Aug 30 13:07:04.212245 there's a Internal Server Error | |
| 07:13:57 | gibi | bauzas: but I don't think placement uses it directly | |
| 07:14:10 | gibi | bauzas: so possible yes, for unified limits so you can group usages per project | |
| 07:14:16 | gibi | gthiemonge: thanks, looking | |
| 07:15:33 | gthiemonge | gibi: if that helps: it fails on Octavia when we request 2 server creations at the same time, it doesn't fail when creating one server | |
| 07:15:59 | gibi | gthiemonge: thanks, this will be a race | |
| 07:16:53 | gibi | we changed transactional behavior of allocation update in https://review.opendev.org/q/topic:bp/support-consumer-types | |
| 07:17:49 | bauzas | gibi: surely for unified limits, but l looked at the series and nothing merged yet | |
| 07:18:36 | bauzas | gibi: yeah, hence my confusion : we merged stuff for consumer types but none of them touched the projects table | |
| 07:18:55 | gibi | bauzas: here you can query usages per project https://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages | |
| 07:19:11 | bauzas | oh now I remember | |