| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-27 | |||
| 15:45:47 | gibi | there is a correlation with the consumer_type series landed and the uptick in the conflict failures in the gate | |
| 15:46:20 | gibi | I know correlation is not causation but still it feel relevant | |
| 15:46:34 | gibi | and the consumer_type series had this transaction logic change | |
| 15:46:36 | dansmith | hrm | |
| 15:46:50 | gibi | i have no better explanation at the moment | |
| 15:47:21 | dansmith | ack | |
| 15:53:21 | sean-k-mooney | dansmith: sorry was on internal call | |
| 15:54:17 | sean-k-mooney | ah am no i know there is a collation but not why | |
| 16:11:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support boot with extended resource request https://review.opendev.org/c/openstack/nova/+/800086 | |
| 16:12:39 | opendevreview | Ghanshyam proposed openstack/nova master: Convert features not supported error to HTTPBadRequest https://review.opendev.org/c/openstack/nova/+/806294 | |
| 16:13:01 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support move ops with extended resource request https://review.opendev.org/c/openstack/nova/+/800087 | |
| 16:14:27 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test] refactor interface attach with qos https://review.opendev.org/c/openstack/nova/+/800088 | |
| 16:15:49 | opendevreview | Balazs Gibizer proposed openstack/nova master: Support interface attach / detach with new resource request format https://review.opendev.org/c/openstack/nova/+/800089 | |
| 16:17:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place https://review.opendev.org/c/openstack/nova/+/793621 | |
| 16:18:32 | opendevreview | Balazs Gibizer proposed openstack/nova master: [nova-manage]support extended resource request https://review.opendev.org/c/openstack/nova/+/802060 | |
| 16:19:26 | opendevreview | sean mooney proposed openstack/nova master: Remove module level caching https://review.opendev.org/c/openstack/nova/+/806394 | |
| 16:19:27 | opendevreview | sean mooney proposed openstack/nova master: db: Handle parameters in DB strings https://review.opendev.org/c/openstack/nova/+/805663 | |
| 16:19:54 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reno for qos-minimum-guaranteed-packet-rate https://review.opendev.org/c/openstack/nova/+/805046 | |
| 16:30:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add force kwarg to delete_allocation_for_instance https://review.opendev.org/c/openstack/nova/+/688802 | |
| 16:30:39 | gibi | dansmith, sean-k-mooney, melwitt, lyarwood: based on the above disucssion I inverted the force default value in ^^ | |
| 16:31:09 | gibi | lyarwood, stephenfin: I fixed up the nits in the pps series. thanks for the valuable feedback | |
| 16:33:39 | gibi | and with that I end my week. o/ | |
| 16:42:15 | sean-k-mooney | gibi: https://review.opendev.org/c/openstack/project-config/+/787523 has now merged so we shoudl have the review priority lable avaiable just an fyi | |
| 16:45:37 | gmann | finally :) | |
| 16:46:14 | sean-k-mooney | i think https://review.opendev.org/q/project:openstack/nova+Review-Priority:1 is how to use it | |
| 16:46:30 | sean-k-mooney | but we dont have any patches with it set | |
| 16:46:36 | sean-k-mooney | https://review.opendev.org/q/project:openstack/nova+Review-Priority:0 get hits however | |
| 16:56:07 | gmann | or event this, https://review.opendev.org/q/project:openstack/nova+Review-Priority:1+label:Verified%253D1++NOT+label:Workflow%253C%253D-1 | |
| 16:56:08 | gmann | sean-k-mooney: ^^ | |
| 16:56:29 | gmann | *even | |
| 16:57:15 | sean-k-mooney | perhaps we will need at lest one patch marked as a prioroty to test it | |
| 16:57:22 | gmann | it can be appended with NOT+owner:self | |
| 16:57:56 | gmann | sean-k-mooney: you can test with 0 priority to test query | |
| 16:58:52 | melwitt | gibi: ack | |
| 17:22:16 | melwitt | dansmith: I didn't think changing to a single transaction would make that much of a timing difference either but the consumer types patch series hit that bug far more often than anything else did. I didn't see anything else in those patches that could possibly be related, so I guessed about the db transaction change. my guess might be wrong | |
| 17:22:38 | dansmith | I'm not saying you're wrong, I'm just surprised | |
| 17:23:05 | dansmith | and surprised lots of real deployments aren't having trouble with much busier systems before the consumer types patch made the transaction longer | |
| 17:23:19 | melwitt | I was surprised too and thought maybe it was a coincidence but once it merged everything was hitting the bug | |
| 17:23:26 | dansmith | and if that's really it, I sure hope we're not in for crazy pain if that made the transaction like waaaay longer or something | |
| 17:24:17 | melwitt | yeah, same. I haven't seen something like this before | |
| 17:37:31 | sean-k-mooney | i know that i have seen conflict in ci jobs form update avaiable resouces before the consome types change | |
| 17:38:01 | sean-k-mooney | so there was already some conficts happening | |
| 17:38:18 | sean-k-mooney | i dont think it caused test failures but i have seen it in the logs | |
| 17:43:15 | dansmith | oh definitely | |
| 17:57:57 | opendevreview | Ghanshyam proposed openstack/nova master: Convert features not supported error to HTTPBadRequest https://review.opendev.org/c/openstack/nova/+/806294 | |
| 18:28:37 | melwitt | yeah, I know it was occurring prior to consumer types, just saying that on the consumer types series it hit the bug often and after it merged it's happening more often | |
| 18:34:49 | melwitt | and re: "yeah, same. I haven't seen something like this before" I haven't seen grouping writes into a single db transaction cause a significant timing difference before | |
| 18:53:26 | dansmith | melwitt: it failed zuul, but I don't see any conflcit messages in there, although it is one of these "hung until timeout" sorts it seems | |
| 18:53:44 | dansmith | maybe that multicell failure is known and some other pattern? | |
| 18:56:18 | melwitt | dansmith: you're talking about the failure on the force kwarg patch right? I would expect fails on that to not be the conflict bug and at a glance it looks like it is indeed something different | |
| 18:58:36 | melwitt | unrelated to that, this is a new error in the controller compute log on that run: Remote error: DBReferenceError (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` | |
| 18:58:46 | melwitt | https://zuul.opendev.org/t/openstack/build/575784eb62da4365ab8a1942da0353bc/log/controller/logs/screen-n-cpu.txt#30972 | |
| 18:59:04 | dansmith | I saw that too | |
| 18:59:35 | dansmith | maybe that dropped an update and that's why the test waited until timeout? | |
| 19:00:20 | melwitt | and on the instance from the failed tempest test: [instance: 99c8640e-cb21-4fa9-bb90-d3361201ce7c] Failed to allocate network(s): nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed | |
| 19:00:20 | melwitt | Aug 27 17:45:54.463587 ubuntu-focal-rax-ord-0026150414 nova-compute[111496]: ERROR nova.compute.manager [instance: 99c8640e-cb21-4fa9-bb90-d3361201ce7c] Traceback (most recent call last): | |
| 19:00:31 | melwitt | https://zuul.opendev.org/t/openstack/build/575784eb62da4365ab8a1942da0353bc/log/controller/logs/screen-n-cpu.txt#33831 | |
| 19:02:17 | melwitt | server failed to spawn. the info cache error was on a different instance | |
| 19:04:48 | melwitt | [instance: 217a81e4-b9cf-4e9e-97a9-edcb0fab8349] Can not refresh info_cache because instance was not found | |
| 19:05:06 | melwitt | trying to refresh info cache on an instance that's gone.. that's odd | |
| 19:25:24 | melwitt | oh, it looks like a network event went to the compute/cell that the instance was not in, so when it tried to refresh the info cache, it was not found | |
| 21:20:53 | opendevreview | Merged openstack/nova stable/wallaby: Avoid modifying the Mock class in test https://review.opendev.org/c/openstack/nova/+/805759 | |
| 21:21:04 | opendevreview | Merged openstack/nova master: [func test] move port creation to the NeutronFixture https://review.opendev.org/c/openstack/nova/+/787206 | |
| #openstack-nova - 2021-08-28 | |||
| 09:56:06 | opendevreview | Merged openstack/nova stable/wallaby: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/805818 | |
| 11:20:56 | opendevreview | Merged openstack/nova stable/stein: Avoid lazy-loading instance.flavor in cold migration https://review.opendev.org/c/openstack/nova/+/774796 | |
| 11:26:43 | opendevreview | Merged openstack/nova stable/victoria: Avoid modifying the Mock class in test https://review.opendev.org/c/openstack/nova/+/805823 | |
| #openstack-nova - 2021-08-29 | |||
| 03:18:25 | opendevreview | melanie witt proposed openstack/nova master: Add stub unified limits driver https://review.opendev.org/c/openstack/nova/+/712137 | |
| 03:18:25 | opendevreview | melanie witt proposed openstack/nova master: Assert quota related API behavior when noop https://review.opendev.org/c/openstack/nova/+/712140 | |
| 03:18:26 | opendevreview | melanie witt proposed openstack/nova master: Make unified limits APIs return reserved of 0 https://review.opendev.org/c/openstack/nova/+/712141 | |
| 03:18:26 | opendevreview | melanie witt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/c/openstack/nova/+/712139 | |
| 03:18:27 | opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | |
| 03:18:27 | opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | |
| 03:18:28 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 03:18:28 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 03:18:29 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 03:18:30 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 03:18:30 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 03:18:32 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 03:18:32 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 03:38:33 | melwitt | gibi, johnthetubaguy[m], dansmith: unified limits set has been updated to do all limits in keystone (no config option quota limits when the UnifiedLimitsDriver is being used) ^ apologies for the delay. I have summarized what I did in https://review.opendev.org/c/openstack/nova/+/712139/13#message-b41a9e4a8a96136ca7e4bff126b575effa2fe839 | |
| 04:28:17 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 04:28:18 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 04:28:18 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 04:28:19 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 04:28:19 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 04:28:20 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 04:28:20 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 12:12:55 | maxba | hi guys, I have problem about multicast and broadcast in public network. I have much broadcast traffic in the network from stupid application, eg. winbox, windows smb, .... how can I drop these traffic just for my public network? | |
| 12:13:57 | maxba | I use openvswitch for l2 networking | |
| 12:15:37 | maxba | i tested iptables, ebtables, in layer of compute and network namespace of public network in the compute, they didn't work | |
| 16:00:46 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 16:00:47 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 16:00:47 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 16:00:48 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 16:00:48 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| #openstack-nova - 2021-08-30 | |||
| 06:11:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add force kwarg to delete_allocation_for_instance https://review.opendev.org/c/openstack/nova/+/688802 | |
| 06:11:56 | gibi | melwitt: ^^ removed the co-authored line as you requested | |
| 06:34:31 | gibi | lyarwood, stephenfin, bauzas: we are still pretty much blocking the openstack gate without ^^ | |
| 06:45:41 | lyarwood | I'm out today, last public holiday of the year in the UK but I'll review from my phone now. | |
| 06:49:43 | lyarwood | Okay done, LGTM. | |