| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-27 | |||
| 17:23:05 | dansmith | and surprised lots of real deployments aren't having trouble with much busier systems before the consumer types patch made the transaction longer | |
| 17:23:19 | melwitt | I was surprised too and thought maybe it was a coincidence but once it merged everything was hitting the bug | |
| 17:23:26 | dansmith | and if that's really it, I sure hope we're not in for crazy pain if that made the transaction like waaaay longer or something | |
| 17:24:17 | melwitt | yeah, same. I haven't seen something like this before | |
| 17:37:31 | sean-k-mooney | i know that i have seen conflict in ci jobs form update avaiable resouces before the consome types change | |
| 17:38:01 | sean-k-mooney | so there was already some conficts happening | |
| 17:38:18 | sean-k-mooney | i dont think it caused test failures but i have seen it in the logs | |
| 17:43:15 | dansmith | oh definitely | |
| 17:57:57 | opendevreview | Ghanshyam proposed openstack/nova master: Convert features not supported error to HTTPBadRequest https://review.opendev.org/c/openstack/nova/+/806294 | |
| 18:28:37 | melwitt | yeah, I know it was occurring prior to consumer types, just saying that on the consumer types series it hit the bug often and after it merged it's happening more often | |
| 18:34:49 | melwitt | and re: "yeah, same. I haven't seen something like this before" I haven't seen grouping writes into a single db transaction cause a significant timing difference before | |
| 18:53:26 | dansmith | melwitt: it failed zuul, but I don't see any conflcit messages in there, although it is one of these "hung until timeout" sorts it seems | |
| 18:53:44 | dansmith | maybe that multicell failure is known and some other pattern? | |
| 18:56:18 | melwitt | dansmith: you're talking about the failure on the force kwarg patch right? I would expect fails on that to not be the conflict bug and at a glance it looks like it is indeed something different | |
| 18:58:36 | melwitt | unrelated to that, this is a new error in the controller compute log on that run: Remote error: DBReferenceError (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`nova_cell1`.`instance_info_caches`, CONSTRAINT `instance_info_caches_instance_uuid_fkey` | |
| 18:58:46 | melwitt | https://zuul.opendev.org/t/openstack/build/575784eb62da4365ab8a1942da0353bc/log/controller/logs/screen-n-cpu.txt#30972 | |
| 18:59:04 | dansmith | I saw that too | |
| 18:59:35 | dansmith | maybe that dropped an update and that's why the test waited until timeout? | |
| 19:00:20 | melwitt | Aug 27 17:45:54.463587 ubuntu-focal-rax-ord-0026150414 nova-compute[111496]: ERROR nova.compute.manager [instance: 99c8640e-cb21-4fa9-bb90-d3361201ce7c] Traceback (most recent call last): | |
| 19:00:20 | melwitt | and on the instance from the failed tempest test: [instance: 99c8640e-cb21-4fa9-bb90-d3361201ce7c] Failed to allocate network(s): nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed | |
| 19:00:31 | melwitt | https://zuul.opendev.org/t/openstack/build/575784eb62da4365ab8a1942da0353bc/log/controller/logs/screen-n-cpu.txt#33831 | |
| 19:02:17 | melwitt | server failed to spawn. the info cache error was on a different instance | |
| 19:04:48 | melwitt | [instance: 217a81e4-b9cf-4e9e-97a9-edcb0fab8349] Can not refresh info_cache because instance was not found | |
| 19:05:06 | melwitt | trying to refresh info cache on an instance that's gone.. that's odd | |
| 19:25:24 | melwitt | oh, it looks like a network event went to the compute/cell that the instance was not in, so when it tried to refresh the info cache, it was not found | |
| 21:20:53 | opendevreview | Merged openstack/nova stable/wallaby: Avoid modifying the Mock class in test https://review.opendev.org/c/openstack/nova/+/805759 | |
| 21:21:04 | opendevreview | Merged openstack/nova master: [func test] move port creation to the NeutronFixture https://review.opendev.org/c/openstack/nova/+/787206 | |
| #openstack-nova - 2021-08-28 | |||
| 09:56:06 | opendevreview | Merged openstack/nova stable/wallaby: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/805818 | |
| 11:20:56 | opendevreview | Merged openstack/nova stable/stein: Avoid lazy-loading instance.flavor in cold migration https://review.opendev.org/c/openstack/nova/+/774796 | |
| 11:26:43 | opendevreview | Merged openstack/nova stable/victoria: Avoid modifying the Mock class in test https://review.opendev.org/c/openstack/nova/+/805823 | |
| #openstack-nova - 2021-08-29 | |||
| 03:18:25 | opendevreview | melanie witt proposed openstack/nova master: Assert quota related API behavior when noop https://review.opendev.org/c/openstack/nova/+/712140 | |
| 03:18:25 | opendevreview | melanie witt proposed openstack/nova master: Add stub unified limits driver https://review.opendev.org/c/openstack/nova/+/712137 | |
| 03:18:26 | opendevreview | melanie witt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/c/openstack/nova/+/712139 | |
| 03:18:26 | opendevreview | melanie witt proposed openstack/nova master: Make unified limits APIs return reserved of 0 https://review.opendev.org/c/openstack/nova/+/712141 | |
| 03:18:27 | opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | |
| 03:18:27 | opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | |
| 03:18:28 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 03:18:28 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 03:18:29 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 03:18:30 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 03:18:30 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 03:18:32 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 03:18:32 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 03:38:33 | melwitt | gibi, johnthetubaguy[m], dansmith: unified limits set has been updated to do all limits in keystone (no config option quota limits when the UnifiedLimitsDriver is being used) ^ apologies for the delay. I have summarized what I did in https://review.opendev.org/c/openstack/nova/+/712139/13#message-b41a9e4a8a96136ca7e4bff126b575effa2fe839 | |
| 04:28:17 | opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | |
| 04:28:18 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 04:28:18 | opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | |
| 04:28:19 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 04:28:19 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 04:28:20 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 04:28:20 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| 12:12:55 | maxba | hi guys, I have problem about multicast and broadcast in public network. I have much broadcast traffic in the network from stupid application, eg. winbox, windows smb, .... how can I drop these traffic just for my public network? | |
| 12:13:57 | maxba | I use openvswitch for l2 networking | |
| 12:15:37 | maxba | i tested iptables, ebtables, in layer of compute and network namespace of public network in the compute, they didn't work | |
| 16:00:46 | opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | |
| 16:00:47 | opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to unified limits https://review.opendev.org/c/openstack/nova/+/713498 | |
| 16:00:47 | opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | |
| 16:00:48 | opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | |
| 16:00:48 | opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | |
| #openstack-nova - 2021-08-30 | |||
| 06:11:31 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add force kwarg to delete_allocation_for_instance https://review.opendev.org/c/openstack/nova/+/688802 | |
| 06:11:56 | gibi | melwitt: ^^ removed the co-authored line as you requested | |
| 06:34:31 | gibi | lyarwood, stephenfin, bauzas: we are still pretty much blocking the openstack gate without ^^ | |
| 06:45:41 | lyarwood | I'm out today, last public holiday of the year in the UK but I'll review from my phone now. | |
| 06:49:43 | lyarwood | Okay done, LGTM. | |
| 06:57:38 | gibi | lyarwood: thanks, enjoy your day off | |
| 08:39:45 | abhishekk | gibi, py38 post failure for https://review.opendev.org/c/openstack/nova/+/688802, could you please dd recheck ? | |
| 10:01:54 | elodilles | gibi: could you please have a quick look at this placement release patch for stable/ussuri? (it's a generated patch to avoid release rush around EM transition): https://review.opendev.org/c/openstack/releases/+/802110 | |
| 10:07:12 | gibi | elodilles: ack I will check | |
| 10:09:25 | gibi | abhishekk feel free to recheck next time | |
| 10:11:15 | gibi | elodilles: done and thanks | |
| 10:12:18 | elodilles | gibi: thanks \o/ | |
| 12:21:37 | gibi | the force kwargs patch https://review.opendev.org/c/openstack/nova/+/688802 bounced from the gate due to bug 1912310, I've requeued it | |
| 12:24:06 | sean-k-mooney | what causes https://bugs.launchpad.net/nova/+bug/1912310 | |
| 12:25:16 | gibi | I saw libvirt internal errors like | |
| 12:25:17 | gibi | 2021-07-30 08:56:25.528+0000: 57632: error : virProcessRunInFork:1159 : internal error: child reported (status=125): unable to open /dev/sda: No such device or address | |
| 12:26:10 | sean-k-mooney | ok so it looks like its actully libvirt that is having issue not nova connecting to it | |
| 12:26:16 | gibi | yepp | |
| 12:26:21 | gibi | as far as I understand | |
| 12:27:04 | gibi | there are also occasions with | |
| 12:27:05 | gibi | virKeepAliveTimerInternal:137 : internal error: connection closed due to keepalive timeout | |
| 12:27:14 | sean-k-mooney | we are not seeing any OOM events or anythying else strange on the node at the time are we | |
| 12:27:53 | gibi | I just like two occurence and found no such thing | |
| 12:27:58 | gibi | I just checked like | |
| 12:28:20 | gibi | the nova-live-migration job set to non-voting due to this | |
| 12:28:29 | gibi | but it seems we can hit the same in nova-next too | |
| 12:28:34 | gibi | but a lot less frequently | |
| 12:29:14 | sean-k-mooney | yep if failind in nova-net in this case | |
| 12:29:16 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/f888b58ca23f49fc8f9046e9c2ad18a0/log/controller/logs/screen-dstat.txt | |
| 12:29:37 | gibi | yes | |
| 12:29:48 | gibi | that is basically a first time I see it in nova-next | |
| 12:29:51 | sean-k-mooney | we got donw to 120MB a few times but i dont see any really evidence of memory issues so likely not the kernel randomly killing things | |
| 12:32:07 | gibi | around the time of the failure we were floating around 300MB free | |
| 12:33:52 | sean-k-mooney | ya its unlikely to be the cause but we have seen OOM issue break libvirt and other process in weird ways before. | |
| 12:34:20 | gibi | true, oom can cause weird thing | |
| 12:34:22 | gibi | s | |
| 12:35:09 | sean-k-mooney | ill quickly check the cloud archive | |
| 12:35:19 | sean-k-mooney | perhaps there is a newer libvirt avaiable we coudl use instead | |
| 12:36:07 | gibi | didn't we use the max available? | |
| 12:37:00 | sean-k-mooney | well im not sure we are using the xena cloud archive currently | |
| 12:37:14 | sean-k-mooney | but looking at it they are not shiping libvirt/qemu in the cloud archive currently | |