| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-19 | |||
| 16:46:26 | stephenfin | CI stability patch here looking for another core https://review.opendev.org/#/c/713611/ | |
| 16:48:33 | openstackgerrit | Merged openstack/nova master: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | |
| 16:48:40 | openstackgerrit | Merged openstack/nova master: virt: Pass request context to extend_volume https://review.opendev.org/706899 | |
| 16:50:16 | lyarwood | stephenfin: nice! | |
| 16:50:25 | lyarwood | stephenfin: did you end up fixing that upstream? | |
| 16:50:41 | stephenfin | Yup, thankfully | |
| 16:50:59 | sean-k-mooney | so they accpeted your change related to async stream handeling? | |
| 16:51:02 | stephenfin | My original fix was correct. It just took a while to figure out why | |
| 16:51:29 | lyarwood | awesome | |
| 16:52:03 | stephenfin | It wasn't anything to do with async. Turns out 'read(N)' on anything inheriting from 'RawIOBase' doesn't guarantee N bytes | |
| 16:52:13 | stephenfin | unlike 'BufferedIOBase' | |
| 16:54:35 | stephenfin | lyarwood: is this a real issue? https://zuul.opendev.org/t/openstack/build/9dc93d24e9634d7e95ee6f2763b38613/log/logs/screen-n-cpu.txt#5506 | |
| 16:54:49 | stephenfin | trying to figure out why https://review.opendev.org/#/c/704759/10 failed | |
| 16:57:27 | stephenfin | actually, nvm, it's not that. It's a simple 404 https://zuul.opendev.org/t/openstack/build/9dc93d24e9634d7e95ee6f2763b38613/log/logs/grenade.sh.txt#46542 | |
| 16:57:49 | lyarwood | stephenfin: no that's fine | |
| 16:58:03 | lyarwood | stephenfin: yeah I spoke about it earlier, apt-get install failing right | |
| 16:58:09 | stephenfin | yuup | |
| 16:58:24 | stephenfin | from http://mirror.ord.rax.opendev.org/ | |
| 16:58:58 | sean-k-mooney | stephenfin: it returns upto N bytes correct | |
| 16:59:09 | stephenfin | sean-k-mooney: correct | |
| 16:59:23 | sean-k-mooney | stephenfin: i think that is the same behavior as the read syscall | |
| 16:59:36 | sean-k-mooney | e.g. if you were to invoke it form c | |
| 16:59:58 | sean-k-mooney | so you have to check how many bytes you actully got | |
| 17:00:04 | stephenfin | also correct | |
| 17:00:27 | sean-k-mooney | im guessign that was causing the header or soemthign to be tuncated | |
| 17:00:37 | sean-k-mooney | or other strange hard to dignose error | |
| 17:01:11 | sean-k-mooney | in anycase im glad you managed to fix it | |
| 17:03:34 | efried_gone | sean-k-mooney: re rootwrap, I think there's some vestigial code that we needed to keep because of a weird os-brick thing. I don't remember the details exactly. But we never figured out whether we could safely remove it, or how. | |
| 17:10:29 | sean-k-mooney | efried_gone: did we not remove all usage in nova itself | |
| 17:10:40 | sean-k-mooney | you might be very well right about os-brick | |
| 17:11:03 | sean-k-mooney | but we merged all the patches in the remove root wrap series | |
| 17:12:02 | stephenfin | sean-k-mooney: https://github.com/openstack/nova/tree/master/etc/nova/rootwrap.d/compute.filters | |
| 17:17:26 | efried_gone | yeah | |
| 17:20:33 | sean-k-mooney | right so that is so it can launch its own privsep deamon right | |
| 17:21:16 | sean-k-mooney | os-vif also launches its own privsep deamon and we dont have a rootwrap entry | |
| 17:40:01 | sean-k-mooney | is there a reason that os-brick needs to have rootwap to start privsep? | |
| 17:43:09 | sean-k-mooney | ok so it still uses rootwap directly https://github.com/openstack/os-brick/blob/master/os_brick/privileged/rootwrap.py | |
| 17:46:27 | sean-k-mooney | stephenfin: so ya i think nova nolonger need to have that rootwap filter. it could be move to os-brick but basically if that is the only usage i satand by my comment that nova does not use rootwap os-brick does | |
| 17:47:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834 | |
| 17:47:41 | sean-k-mooney | it looks like os-brick only has 4 usages of rootwap remining too | |
| 17:49:11 | sean-k-mooney | actully 3 | |
| 17:50:34 | sean-k-mooney | actully no its 7 i didnt scroll im going to check if those need to be elevated or not | |
| 17:51:56 | sean-k-mooney | ya so all of them pass run_as_root=true which means they actully use privsep to run them | |
| 17:52:03 | sean-k-mooney | because we hit here https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f28b7e6ff/os_brick/privileged/rootwrap.py#L169 | |
| 17:52:23 | sean-k-mooney | and execute_root is a privsep function https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f28b7e6ff/os_brick/privileged/rootwrap.py#L190-L194 | |
| 17:59:00 | sean-k-mooney | it looks like oslo rootwrap was removed as a os-bricks depency in liberty too https://github.com/openstack/os-brick/commit/7d341f154397ef38c480bdd9cf6d2ccad6502f13 but https://github.com/openstack/os-brick/commit/dbf77fba1061cb4e93b3db5f8117d6ccc689f702#diff-0d141267b46cdfd7a9dfe6100d79fe33 removed it as an actul depency in newton | |
| 18:01:32 | sean-k-mooney | the way os-bricks is using privsep is not secure but i dont think its usign the filters at all | |
| 18:42:49 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713961 | |
| 18:43:39 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713962 | |
| 19:03:44 | artom | dansmith, quick chat about https://review.opendev.org/#/c/705545/4/nova/tests/functional/integrated_helpers.py@370 ? | |
| 19:10:53 | dansmith | artom: just back to my desk, have to do something first and then yeah | |
| 19:12:48 | artom | dansmith, so, I looked around some more, thought some more, and I think I've come around to your way of thinking | |
| 19:12:57 | artom | I gotta go walk the kiddos | |
| 19:13:21 | dansmith | heh okay | |
| 19:34:21 | openstackgerrit | Merged openstack/nova stable/rocky: Fix os-keypairs pagination links https://review.opendev.org/711898 | |
| 21:06:20 | artom | dansmith, hrmm, so in the end maybe talking about https://review.opendev.org/#/c/705545/4/nova/tests/functional/integrated_helpers.py@370 could help | |
| 21:07:50 | artom | So, places that redefine _setup_compute_service just don't inherit from the ServicesMixing | |
| 21:08:14 | artom | I think I brought about some confusion when I called it the ServicesHelper mixin | |
| 21:08:22 | artom | It's really the PeriodicHelperMixin | |
| 21:08:30 | artom | Ie - you only get run_periodics in it | |
| 21:08:51 | artom | But because run_periodics depends on self.computes, you get _start_compute as well, which populates self.compute | |
| 21:09:18 | openstackgerrit | Merged openstack/nova stable/train: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713739 | |
| 21:09:24 | openstackgerrit | Merged openstack/nova stable/queens: Avoid circular reference during serialization https://review.opendev.org/713132 | |
| 21:09:31 | openstack | bug 1849409 in OpenStack Compute (nova) ocata "openstack server list --deleted --limit -1 hangs" [High,In progress] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem) | |
| 21:09:31 | openstackgerrit | Merged openstack/nova stable/ocata: Add functional regression test for bug 1849409 https://review.opendev.org/690746 | |
| 21:09:36 | openstackgerrit | Merged openstack/nova stable/ocata: Fix listing deleted servers with a marker https://review.opendev.org/690747 | |
| 21:09:41 | openstackgerrit | Merged openstack/nova stable/ocata: Fix incompatible version handling in BuildRequest https://review.opendev.org/660456 | |
| 21:09:48 | openstackgerrit | Merged openstack/nova stable/ocata: Hook resource_tracker to remove stale node information https://review.opendev.org/676467 | |
| 21:14:53 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713982 | |
| 21:17:09 | openstackgerrit | Merged openstack/nova master: Switching new default roles in os-volumes-attachments policies https://review.opendev.org/710190 | |
| 21:48:53 | melwitt | zzzeek: I've been looking at some debug logs in gate runs and I notice a lot of ROLLBACK logged, would you think it's concerning? https://zuul.opendev.org/t/openstack/build/b4f6a11e4b0940e0a9513f7113bfb95c/log/logs/screen-n-sch.txt#1691 | |
| 21:50:12 | zzzeek | melwitt: when a connection is done being used, it is returned to the pool where a ROLLBACK is emitted. if you have lots of short single-statement transactions then you'd see a lot of these | |
| 21:50:33 | melwitt | zzzeek: oh, I see. thanks | |
| 21:50:49 | zzzeek | melwitt: these can be a problem performance wise | |
| 21:51:03 | zzzeek | but only if yo're observing that | |
| 21:53:01 | melwitt | zzzeek: yeah ... not yet sure what we're seeing, but this is the file where we have the gate bug where we wait 60 seconds without getting rows back from the database. could be a bug on our side. but I'm studying the logs looking for anything that seems like it could cause trouble. what would be a way of avoiding the rollbacks? for the sake of experiment | |
| 22:02:14 | zzzeek | melwitt: there's a parameter "rollback_on_return" to craete_engine | |
| 22:02:35 | zzzeek | oops, that's not it, one sec | |
| 22:03:01 | melwitt | rollback_reader_sessions ? | |
| 22:03:11 | zzzeek | reset_on_return | |
| 22:03:19 | melwitt | oh k thanks | |
| 22:03:19 | zzzeek | https://docs.sqlalchemy.org/en/13/core/engines.html?highlight=reset_on_return#sqlalchemy.create_engine.params.pool_reset_on_return | |
| 22:03:46 | zzzeek | but...it's dangerous to use this | |
| 22:04:20 | zzzeek | in fact it can't actually work correctly, because there is always a transaction with MySQL InnoDB | |
| 22:04:29 | zzzeek | so you will have very poor effects from using htis | |
| 22:04:41 | zzzeek | melwitt: your only option is to try the NullPool and turn off pooling entirely | |
| 22:04:59 | zzzeek | melwitt: unfortunately oslo.db has no option for that so if you wanted to test it you'd need to monkeypatch oslo.db | |
| 22:05:44 | melwitt | oh. good to know. I wouldn't want to use it then. I was just curious if there was any good/recommended way to address the performance problem with rollbacks if there were one | |
| 22:08:10 | melwitt | it's weird. in the scheduler, there's this constant flow of service select/update/commit happening for the service heartbeats, and when things fail, all of the other selects (for services, compute_nodes) just disappear entirely | |
| 22:08:44 | melwitt | and all I see are the steady select/update/commit for the heartbeats | |
| 22:10:58 | openstackgerrit | Merged openstack/nova master: libvirt: Fix unit test error block info on non x86 architecture https://review.opendev.org/713905 | |
| 22:15:19 | melwitt | ohhh dang I actually see now that the debug logging I added in my DNM patch is _not_ showing up, meaning we're not even trying to query for compute_nodes. that's bad. /me keeps digging | |
| 22:31:06 | zzzeek | melwitt: sorry I dont know that code very well, keep me in the loop :) | |
| 22:31:16 | zzzeek | (dont know it at all really ... :) ) | |
| 22:37:06 | melwitt | oh, derp, false alarm I think. I only put the debug statements in the master branch and this is a grenade job so if it failed on the old side, that would be stable/train which doesn't have the debug logging on it. I need to cherry pick this DNM to train again and run with Depends-On | |
| 22:42:19 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 22:42:19 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 22:42:29 | openstackgerrit | melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116 | |
| 22:43:13 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| #openstack-nova - 2020-03-20 | |||
| 00:08:09 | openstackgerrit | Merged openstack/nova stable/train: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713836 | |