| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-19 | |||
| 19:13:21 | dansmith | heh okay | |
| 19:34:21 | openstackgerrit | Merged openstack/nova stable/rocky: Fix os-keypairs pagination links https://review.opendev.org/711898 | |
| 21:06:20 | artom | dansmith, hrmm, so in the end maybe talking about https://review.opendev.org/#/c/705545/4/nova/tests/functional/integrated_helpers.py@370 could help | |
| 21:07:50 | artom | So, places that redefine _setup_compute_service just don't inherit from the ServicesMixing | |
| 21:08:14 | artom | I think I brought about some confusion when I called it the ServicesHelper mixin | |
| 21:08:22 | artom | It's really the PeriodicHelperMixin | |
| 21:08:30 | artom | Ie - you only get run_periodics in it | |
| 21:08:51 | artom | But because run_periodics depends on self.computes, you get _start_compute as well, which populates self.compute | |
| 21:09:18 | openstackgerrit | Merged openstack/nova stable/train: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713739 | |
| 21:09:24 | openstackgerrit | Merged openstack/nova stable/queens: Avoid circular reference during serialization https://review.opendev.org/713132 | |
| 21:09:31 | openstackgerrit | Merged openstack/nova stable/ocata: Add functional regression test for bug 1849409 https://review.opendev.org/690746 | |
| 21:09:31 | openstack | bug 1849409 in OpenStack Compute (nova) ocata "openstack server list --deleted --limit -1 hangs" [High,In progress] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem) | |
| 21:09:36 | openstackgerrit | Merged openstack/nova stable/ocata: Fix listing deleted servers with a marker https://review.opendev.org/690747 | |
| 21:09:41 | openstackgerrit | Merged openstack/nova stable/ocata: Fix incompatible version handling in BuildRequest https://review.opendev.org/660456 | |
| 21:09:48 | openstackgerrit | Merged openstack/nova stable/ocata: Hook resource_tracker to remove stale node information https://review.opendev.org/676467 | |
| 21:14:53 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713982 | |
| 21:17:09 | openstackgerrit | Merged openstack/nova master: Switching new default roles in os-volumes-attachments policies https://review.opendev.org/710190 | |
| 21:48:53 | melwitt | zzzeek: I've been looking at some debug logs in gate runs and I notice a lot of ROLLBACK logged, would you think it's concerning? https://zuul.opendev.org/t/openstack/build/b4f6a11e4b0940e0a9513f7113bfb95c/log/logs/screen-n-sch.txt#1691 | |
| 21:50:12 | zzzeek | melwitt: when a connection is done being used, it is returned to the pool where a ROLLBACK is emitted. if you have lots of short single-statement transactions then you'd see a lot of these | |
| 21:50:33 | melwitt | zzzeek: oh, I see. thanks | |
| 21:50:49 | zzzeek | melwitt: these can be a problem performance wise | |
| 21:51:03 | zzzeek | but only if yo're observing that | |
| 21:53:01 | melwitt | zzzeek: yeah ... not yet sure what we're seeing, but this is the file where we have the gate bug where we wait 60 seconds without getting rows back from the database. could be a bug on our side. but I'm studying the logs looking for anything that seems like it could cause trouble. what would be a way of avoiding the rollbacks? for the sake of experiment | |
| 22:02:14 | zzzeek | melwitt: there's a parameter "rollback_on_return" to craete_engine | |
| 22:02:35 | zzzeek | oops, that's not it, one sec | |
| 22:03:01 | melwitt | rollback_reader_sessions ? | |
| 22:03:11 | zzzeek | reset_on_return | |
| 22:03:19 | zzzeek | https://docs.sqlalchemy.org/en/13/core/engines.html?highlight=reset_on_return#sqlalchemy.create_engine.params.pool_reset_on_return | |
| 22:03:19 | melwitt | oh k thanks | |
| 22:03:46 | zzzeek | but...it's dangerous to use this | |
| 22:04:20 | zzzeek | in fact it can't actually work correctly, because there is always a transaction with MySQL InnoDB | |
| 22:04:29 | zzzeek | so you will have very poor effects from using htis | |
| 22:04:41 | zzzeek | melwitt: your only option is to try the NullPool and turn off pooling entirely | |
| 22:04:59 | zzzeek | melwitt: unfortunately oslo.db has no option for that so if you wanted to test it you'd need to monkeypatch oslo.db | |
| 22:05:44 | melwitt | oh. good to know. I wouldn't want to use it then. I was just curious if there was any good/recommended way to address the performance problem with rollbacks if there were one | |
| 22:08:10 | melwitt | it's weird. in the scheduler, there's this constant flow of service select/update/commit happening for the service heartbeats, and when things fail, all of the other selects (for services, compute_nodes) just disappear entirely | |
| 22:08:44 | melwitt | and all I see are the steady select/update/commit for the heartbeats | |
| 22:10:58 | openstackgerrit | Merged openstack/nova master: libvirt: Fix unit test error block info on non x86 architecture https://review.opendev.org/713905 | |
| 22:15:19 | melwitt | ohhh dang I actually see now that the debug logging I added in my DNM patch is _not_ showing up, meaning we're not even trying to query for compute_nodes. that's bad. /me keeps digging | |
| 22:31:06 | zzzeek | melwitt: sorry I dont know that code very well, keep me in the loop :) | |
| 22:31:16 | zzzeek | (dont know it at all really ... :) ) | |
| 22:37:06 | melwitt | oh, derp, false alarm I think. I only put the debug statements in the master branch and this is a grenade job so if it failed on the old side, that would be stable/train which doesn't have the debug logging on it. I need to cherry pick this DNM to train again and run with Depends-On | |
| 22:42:19 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 22:42:19 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 22:42:29 | openstackgerrit | melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116 | |
| 22:43:13 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| #openstack-nova - 2020-03-20 | |||
| 00:08:09 | openstackgerrit | Merged openstack/nova stable/train: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713836 | |
| 01:55:57 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 01:55:57 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 05:00:33 | openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 06:49:52 | ygk_12345 | can someone look into this please https://bugs.launchpad.net/nova/+bug/1868203 | |
| 06:49:52 | openstack | Launchpad bug 1868203 in OpenStack Compute (nova) "nova-compute error" [Undecided,New] | |
| 06:58:05 | openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 08:01:39 | openstackgerrit | Brin Zhang proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 08:07:15 | openstackgerrit | Brin Zhang proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 08:14:52 | brinzhang_ | lyarwood:https://review.opendev.org/#/c/682772/ fixed your comment, and add the test case you asked, pls review again if you are free ^^ | |
| 09:06:56 | openstackgerrit | Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494 | |
| 09:20:15 | nightmare_unreal | hey guys, how can I run tests for python-openstackclient | |
| 09:20:25 | nightmare_unreal | i wanted to test it locally before submitting a patch | |
| 09:24:01 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Only stop n-cpu and q-agt during evacuation testing https://review.opendev.org/714057 | |
| 09:24:14 | lyarwood | nightmare_unreal: tox -e py36 ; tox -e pep8 | |
| 09:24:33 | lyarwood | sean-k-mooney: ^ btw, I missed that we are running c-vol and g-api on the subnodes for some reason | |
| 09:24:54 | lyarwood | I was sure we removed these recently | |
| 09:29:19 | openstackgerrit | Merged openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759 | |
| 09:30:13 | nightmare_unreal | thanks lyarwood but in which directory to run ? inside devstack or inside openstackclient | |
| 09:31:02 | ygk_12345 | hi all | |
| 09:31:36 | ygk_12345 | can someone look into this once ? https://bugs.launchpad.net/nova/+bug/1868203 | |
| 09:31:36 | openstack | ygk_12345: Error: Could not gather data from Launchpad for bug #1868203 (https://launchpad.net/bugs/1868203). The error has been logged | |
| 09:35:10 | lyarwood | nightmare_unreal: openstackclient sorry | |
| 09:35:40 | nightmare_unreal | thanks | |
| 09:41:43 | nightmare_unreal | thanks lyarwood | |
| 09:57:53 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 09:57:53 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 10:02:40 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 10:02:40 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 10:06:02 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 10:06:02 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 10:06:42 | ygk_12345 | can someone look into this https://bugs.launchpad.net/nova/+bug/1868203 | |
| 10:06:42 | openstack | Launchpad bug 1868203 in OpenStack Compute (nova) "nova-compute error" [Undecided,New] | |
| 10:07:09 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing flavor_access policies https://review.opendev.org/713556 | |
| 10:08:54 | nightmare_unreal | how to add my changes to doc or how to build doc for openstackclient ? | |
| 10:09:12 | lyarwood | ygk_12345: can you attach the n-cpu log to the bug? Looks like you're trying to launch qemu-system-ppc64 instances and failing? | |
| 10:09:46 | lyarwood | nightmare_unreal: tox -e docs | |
| 10:10:00 | ygk_12345 | lyarwood it is openstack ansible rocky. where can I find the n-cpu log ? | |
| 10:10:20 | lyarwood | nightmare_unreal: review tox.ini and https://docs.openstack.org/python-openstackclient/latest/contributor/developing.html | |
| 10:10:27 | nightmare_unreal | thanks lyarwood , but how to add my doc code. for eg. i made a new flag | |
| 10:10:57 | lyarwood | nightmare_unreal: look for changes under docs (using tig and/or tig blame) for examples. | |
| 10:11:09 | jkulik | Can someone close https://bugs.launchpad.net/nova/+bug/1855776 it should have been fixed by https://review.opendev.org/#/c/710214/ as it seems to be a duplicate of https://bugs.launchpad.net/nova/+bug/1865040 | |
| 10:11:09 | openstack | Launchpad bug 1855776 in OpenStack Compute (nova) "Aggregate ID validation" [Undecided,In progress] - Assigned to Mykola Yakovliev (vegasq) | |
| 10:11:10 | openstack | Launchpad bug 1865040 in OpenStack Compute (nova) "Able to show update and delete aggregate with invalid id" [Undecided,Fix released] - Assigned to Johannes Kulik (jkulik) | |
| 10:11:11 | nightmare_unreal | thanks | |
| 10:11:28 | openstackgerrit | Luyao Zhong proposed openstack/nova stable/train: bug-fix: Reject live migration with vpmem https://review.opendev.org/714064 | |
| 10:11:28 | lyarwood | ygk_12345: sorry no idea | |
| 10:14:38 | lyarwood | jkulik: if you can still reproduce against master feel free to reuse https://bugs.launchpad.net/nova/+bug/1855776 and submit a fix. | |
| 10:14:38 | openstack | Launchpad bug 1855776 in OpenStack Compute (nova) "Aggregate ID validation" [Undecided,In progress] - Assigned to Mykola Yakovliev (vegasq) | |
| 10:15:40 | jkulik | lyarwood, it looks exactly like the problem in the second bug, which got closed by merging. I just want that bug closed, too :) | |
| 10:17:46 | gibi | ygk_12345: regarding bug #1868203 I feels like a QEMU problem on your compute. If QEMU fails to start then nova will not know the available resources on that compute | |
| 10:17:46 | openstack | bug 1868203 in OpenStack Compute (nova) "nova-compute error" [Undecided,New] https://launchpad.net/bugs/1868203 | |
| 10:18:04 | gibi | ygk_12345: you should reach out the QEMU devs | |
| 10:20:33 | gibi | jkulik: marked the bug as duplicate of 1865040 | |