| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-19 | |||
| 17:21:16 | sean-k-mooney | os-vif also launches its own privsep deamon and we dont have a rootwrap entry | |
| 17:40:01 | sean-k-mooney | is there a reason that os-brick needs to have rootwap to start privsep? | |
| 17:43:09 | sean-k-mooney | ok so it still uses rootwap directly https://github.com/openstack/os-brick/blob/master/os_brick/privileged/rootwrap.py | |
| 17:46:27 | sean-k-mooney | stephenfin: so ya i think nova nolonger need to have that rootwap filter. it could be move to os-brick but basically if that is the only usage i satand by my comment that nova does not use rootwap os-brick does | |
| 17:47:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834 | |
| 17:47:41 | sean-k-mooney | it looks like os-brick only has 4 usages of rootwap remining too | |
| 17:49:11 | sean-k-mooney | actully 3 | |
| 17:50:34 | sean-k-mooney | actully no its 7 i didnt scroll im going to check if those need to be elevated or not | |
| 17:51:56 | sean-k-mooney | ya so all of them pass run_as_root=true which means they actully use privsep to run them | |
| 17:52:03 | sean-k-mooney | because we hit here https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f28b7e6ff/os_brick/privileged/rootwrap.py#L169 | |
| 17:52:23 | sean-k-mooney | and execute_root is a privsep function https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f28b7e6ff/os_brick/privileged/rootwrap.py#L190-L194 | |
| 17:59:00 | sean-k-mooney | it looks like oslo rootwrap was removed as a os-bricks depency in liberty too https://github.com/openstack/os-brick/commit/7d341f154397ef38c480bdd9cf6d2ccad6502f13 but https://github.com/openstack/os-brick/commit/dbf77fba1061cb4e93b3db5f8117d6ccc689f702#diff-0d141267b46cdfd7a9dfe6100d79fe33 removed it as an actul depency in newton | |
| 18:01:32 | sean-k-mooney | the way os-bricks is using privsep is not secure but i dont think its usign the filters at all | |
| 18:42:49 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713961 | |
| 18:43:39 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713962 | |
| 19:03:44 | artom | dansmith, quick chat about https://review.opendev.org/#/c/705545/4/nova/tests/functional/integrated_helpers.py@370 ? | |
| 19:10:53 | dansmith | artom: just back to my desk, have to do something first and then yeah | |
| 19:12:48 | artom | dansmith, so, I looked around some more, thought some more, and I think I've come around to your way of thinking | |
| 19:12:57 | artom | I gotta go walk the kiddos | |
| 19:13:21 | dansmith | heh okay | |
| 19:34:21 | openstackgerrit | Merged openstack/nova stable/rocky: Fix os-keypairs pagination links https://review.opendev.org/711898 | |
| 21:06:20 | artom | dansmith, hrmm, so in the end maybe talking about https://review.opendev.org/#/c/705545/4/nova/tests/functional/integrated_helpers.py@370 could help | |
| 21:07:50 | artom | So, places that redefine _setup_compute_service just don't inherit from the ServicesMixing | |
| 21:08:14 | artom | I think I brought about some confusion when I called it the ServicesHelper mixin | |
| 21:08:22 | artom | It's really the PeriodicHelperMixin | |
| 21:08:30 | artom | Ie - you only get run_periodics in it | |
| 21:08:51 | artom | But because run_periodics depends on self.computes, you get _start_compute as well, which populates self.compute | |
| 21:09:18 | openstackgerrit | Merged openstack/nova stable/train: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713739 | |
| 21:09:24 | openstackgerrit | Merged openstack/nova stable/queens: Avoid circular reference during serialization https://review.opendev.org/713132 | |
| 21:09:31 | openstackgerrit | Merged openstack/nova stable/ocata: Add functional regression test for bug 1849409 https://review.opendev.org/690746 | |
| 21:09:31 | openstack | bug 1849409 in OpenStack Compute (nova) ocata "openstack server list --deleted --limit -1 hangs" [High,In progress] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem) | |
| 21:09:36 | openstackgerrit | Merged openstack/nova stable/ocata: Fix listing deleted servers with a marker https://review.opendev.org/690747 | |
| 21:09:41 | openstackgerrit | Merged openstack/nova stable/ocata: Fix incompatible version handling in BuildRequest https://review.opendev.org/660456 | |
| 21:09:48 | openstackgerrit | Merged openstack/nova stable/ocata: Hook resource_tracker to remove stale node information https://review.opendev.org/676467 | |
| 21:14:53 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713982 | |
| 21:17:09 | openstackgerrit | Merged openstack/nova master: Switching new default roles in os-volumes-attachments policies https://review.opendev.org/710190 | |
| 21:48:53 | melwitt | zzzeek: I've been looking at some debug logs in gate runs and I notice a lot of ROLLBACK logged, would you think it's concerning? https://zuul.opendev.org/t/openstack/build/b4f6a11e4b0940e0a9513f7113bfb95c/log/logs/screen-n-sch.txt#1691 | |
| 21:50:12 | zzzeek | melwitt: when a connection is done being used, it is returned to the pool where a ROLLBACK is emitted. if you have lots of short single-statement transactions then you'd see a lot of these | |
| 21:50:33 | melwitt | zzzeek: oh, I see. thanks | |
| 21:50:49 | zzzeek | melwitt: these can be a problem performance wise | |
| 21:51:03 | zzzeek | but only if yo're observing that | |
| 21:53:01 | melwitt | zzzeek: yeah ... not yet sure what we're seeing, but this is the file where we have the gate bug where we wait 60 seconds without getting rows back from the database. could be a bug on our side. but I'm studying the logs looking for anything that seems like it could cause trouble. what would be a way of avoiding the rollbacks? for the sake of experiment | |
| 22:02:14 | zzzeek | melwitt: there's a parameter "rollback_on_return" to craete_engine | |
| 22:02:35 | zzzeek | oops, that's not it, one sec | |
| 22:03:01 | melwitt | rollback_reader_sessions ? | |
| 22:03:11 | zzzeek | reset_on_return | |
| 22:03:19 | zzzeek | https://docs.sqlalchemy.org/en/13/core/engines.html?highlight=reset_on_return#sqlalchemy.create_engine.params.pool_reset_on_return | |
| 22:03:19 | melwitt | oh k thanks | |
| 22:03:46 | zzzeek | but...it's dangerous to use this | |
| 22:04:20 | zzzeek | in fact it can't actually work correctly, because there is always a transaction with MySQL InnoDB | |
| 22:04:29 | zzzeek | so you will have very poor effects from using htis | |
| 22:04:41 | zzzeek | melwitt: your only option is to try the NullPool and turn off pooling entirely | |
| 22:04:59 | zzzeek | melwitt: unfortunately oslo.db has no option for that so if you wanted to test it you'd need to monkeypatch oslo.db | |
| 22:05:44 | melwitt | oh. good to know. I wouldn't want to use it then. I was just curious if there was any good/recommended way to address the performance problem with rollbacks if there were one | |
| 22:08:10 | melwitt | it's weird. in the scheduler, there's this constant flow of service select/update/commit happening for the service heartbeats, and when things fail, all of the other selects (for services, compute_nodes) just disappear entirely | |
| 22:08:44 | melwitt | and all I see are the steady select/update/commit for the heartbeats | |
| 22:10:58 | openstackgerrit | Merged openstack/nova master: libvirt: Fix unit test error block info on non x86 architecture https://review.opendev.org/713905 | |
| 22:15:19 | melwitt | ohhh dang I actually see now that the debug logging I added in my DNM patch is _not_ showing up, meaning we're not even trying to query for compute_nodes. that's bad. /me keeps digging | |
| 22:31:06 | zzzeek | melwitt: sorry I dont know that code very well, keep me in the loop :) | |
| 22:31:16 | zzzeek | (dont know it at all really ... :) ) | |
| 22:37:06 | melwitt | oh, derp, false alarm I think. I only put the debug statements in the master branch and this is a grenade job so if it failed on the old side, that would be stable/train which doesn't have the debug logging on it. I need to cherry pick this DNM to train again and run with Depends-On | |
| 22:42:19 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 22:42:19 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 22:42:29 | openstackgerrit | melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116 | |
| 22:43:13 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| #openstack-nova - 2020-03-20 | |||
| 00:08:09 | openstackgerrit | Merged openstack/nova stable/train: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713836 | |
| 01:55:57 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 01:55:57 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 05:00:33 | openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 06:49:52 | ygk_12345 | can someone look into this please https://bugs.launchpad.net/nova/+bug/1868203 | |
| 06:49:52 | openstack | Launchpad bug 1868203 in OpenStack Compute (nova) "nova-compute error" [Undecided,New] | |
| 06:58:05 | openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 08:01:39 | openstackgerrit | Brin Zhang proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 08:07:15 | openstackgerrit | Brin Zhang proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | |
| 08:14:52 | brinzhang_ | lyarwood:https://review.opendev.org/#/c/682772/ fixed your comment, and add the test case you asked, pls review again if you are free ^^ | |
| 09:06:56 | openstackgerrit | Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494 | |
| 09:20:15 | nightmare_unreal | hey guys, how can I run tests for python-openstackclient | |
| 09:20:25 | nightmare_unreal | i wanted to test it locally before submitting a patch | |
| 09:24:01 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Only stop n-cpu and q-agt during evacuation testing https://review.opendev.org/714057 | |
| 09:24:14 | lyarwood | nightmare_unreal: tox -e py36 ; tox -e pep8 | |
| 09:24:33 | lyarwood | sean-k-mooney: ^ btw, I missed that we are running c-vol and g-api on the subnodes for some reason | |
| 09:24:54 | lyarwood | I was sure we removed these recently | |
| 09:29:19 | openstackgerrit | Merged openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759 | |
| 09:30:13 | nightmare_unreal | thanks lyarwood but in which directory to run ? inside devstack or inside openstackclient | |
| 09:31:02 | ygk_12345 | hi all | |
| 09:31:36 | ygk_12345 | can someone look into this once ? https://bugs.launchpad.net/nova/+bug/1868203 | |
| 09:31:36 | openstack | ygk_12345: Error: Could not gather data from Launchpad for bug #1868203 (https://launchpad.net/bugs/1868203). The error has been logged | |
| 09:35:10 | lyarwood | nightmare_unreal: openstackclient sorry | |
| 09:35:40 | nightmare_unreal | thanks | |
| 09:41:43 | nightmare_unreal | thanks lyarwood | |
| 09:57:53 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 09:57:53 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 10:02:40 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 10:02:40 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 10:06:02 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 10:06:02 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 10:06:42 | ygk_12345 | can someone look into this https://bugs.launchpad.net/nova/+bug/1868203 | |
| 10:06:42 | openstack | Launchpad bug 1868203 in OpenStack Compute (nova) "nova-compute error" [Undecided,New] | |
| 10:07:09 | openstackgerrit | Merged openstack/nova master: Add test coverage of existing flavor_access policies https://review.opendev.org/713556 | |
| 10:08:54 | nightmare_unreal | how to add my changes to doc or how to build doc for openstackclient ? | |