| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-19 | |||
| 15:40:29 | lyarwood | I was just checking what zuul was going to do in this case | |
| 15:41:27 | gibi | lyarwood: yeah that is mine, failing on some neutron tempest test case :/ | |
| 15:41:34 | sean-k-mooney | lyarwood: is this related to your evacuate change | |
| 15:44:16 | lyarwood | sean-k-mooney: no, my change is below it | |
| 15:44:23 | lyarwood | actually that doesn't matter does it | |
| 15:44:40 | lyarwood | gibi: was that the subnet creation bug I linked in my email? | |
| 15:45:24 | gibi | lyarwood: there was some timeout but did not look deeper | |
| 15:46:08 | gibi | lyarwood: nvm, I looked at a different job | |
| 15:47:05 | lyarwood | https://4cc1d8431535199f530e-fcfc85786424b7f5a3fcadff9da35f94.ssl.cf2.rackcdn.com/704759/10/gate/grenade-py3/9dc93d2/logs/grenade.sh.txt - looks like grenade failed to update libvirt/qemu | |
| 15:47:51 | gibi | yep | |
| 15:50:26 | gibi | nova team meeting in 10 minutes on #openstack-meeting-3 | |
| 15:52:10 | stephenfin | Can someone != gibi take a look at this, the final remove-nova-network patch? https://review.opendev.org/#/c/703974/ | |
| 15:52:47 | stephenfin | I don't want your kind round here :P | |
| 16:01:34 | gibi | nova meeting is running on #openstack-meeting-3 | |
| 16:24:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fix unit test error block info on non x86 architecture https://review.opendev.org/713905 | |
| 16:46:26 | stephenfin | CI stability patch here looking for another core https://review.opendev.org/#/c/713611/ | |
| 16:48:33 | openstackgerrit | Merged openstack/nova master: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | |
| 16:48:40 | openstackgerrit | Merged openstack/nova master: virt: Pass request context to extend_volume https://review.opendev.org/706899 | |
| 16:50:16 | lyarwood | stephenfin: nice! | |
| 16:50:25 | lyarwood | stephenfin: did you end up fixing that upstream? | |
| 16:50:41 | stephenfin | Yup, thankfully | |
| 16:50:59 | sean-k-mooney | so they accpeted your change related to async stream handeling? | |
| 16:51:02 | stephenfin | My original fix was correct. It just took a while to figure out why | |
| 16:51:29 | lyarwood | awesome | |
| 16:52:03 | stephenfin | It wasn't anything to do with async. Turns out 'read(N)' on anything inheriting from 'RawIOBase' doesn't guarantee N bytes | |
| 16:52:13 | stephenfin | unlike 'BufferedIOBase' | |
| 16:54:35 | stephenfin | lyarwood: is this a real issue? https://zuul.opendev.org/t/openstack/build/9dc93d24e9634d7e95ee6f2763b38613/log/logs/screen-n-cpu.txt#5506 | |
| 16:54:49 | stephenfin | trying to figure out why https://review.opendev.org/#/c/704759/10 failed | |
| 16:57:27 | stephenfin | actually, nvm, it's not that. It's a simple 404 https://zuul.opendev.org/t/openstack/build/9dc93d24e9634d7e95ee6f2763b38613/log/logs/grenade.sh.txt#46542 | |
| 16:57:49 | lyarwood | stephenfin: no that's fine | |
| 16:58:03 | lyarwood | stephenfin: yeah I spoke about it earlier, apt-get install failing right | |
| 16:58:09 | stephenfin | yuup | |
| 16:58:24 | stephenfin | from http://mirror.ord.rax.opendev.org/ | |
| 16:58:58 | sean-k-mooney | stephenfin: it returns upto N bytes correct | |
| 16:59:09 | stephenfin | sean-k-mooney: correct | |
| 16:59:23 | sean-k-mooney | stephenfin: i think that is the same behavior as the read syscall | |
| 16:59:36 | sean-k-mooney | e.g. if you were to invoke it form c | |
| 16:59:58 | sean-k-mooney | so you have to check how many bytes you actully got | |
| 17:00:04 | stephenfin | also correct | |
| 17:00:27 | sean-k-mooney | im guessign that was causing the header or soemthign to be tuncated | |
| 17:00:37 | sean-k-mooney | or other strange hard to dignose error | |
| 17:01:11 | sean-k-mooney | in anycase im glad you managed to fix it | |
| 17:03:34 | efried_gone | sean-k-mooney: re rootwrap, I think there's some vestigial code that we needed to keep because of a weird os-brick thing. I don't remember the details exactly. But we never figured out whether we could safely remove it, or how. | |
| 17:10:29 | sean-k-mooney | efried_gone: did we not remove all usage in nova itself | |
| 17:10:40 | sean-k-mooney | you might be very well right about os-brick | |
| 17:11:03 | sean-k-mooney | but we merged all the patches in the remove root wrap series | |
| 17:12:02 | stephenfin | sean-k-mooney: https://github.com/openstack/nova/tree/master/etc/nova/rootwrap.d/compute.filters | |
| 17:17:26 | efried_gone | yeah | |
| 17:20:33 | sean-k-mooney | right so that is so it can launch its own privsep deamon right | |
| 17:21:16 | sean-k-mooney | os-vif also launches its own privsep deamon and we dont have a rootwrap entry | |
| 17:40:01 | sean-k-mooney | is there a reason that os-brick needs to have rootwap to start privsep? | |
| 17:43:09 | sean-k-mooney | ok so it still uses rootwap directly https://github.com/openstack/os-brick/blob/master/os_brick/privileged/rootwrap.py | |
| 17:46:27 | sean-k-mooney | stephenfin: so ya i think nova nolonger need to have that rootwap filter. it could be move to os-brick but basically if that is the only usage i satand by my comment that nova does not use rootwap os-brick does | |
| 17:47:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834 | |
| 17:47:41 | sean-k-mooney | it looks like os-brick only has 4 usages of rootwap remining too | |
| 17:49:11 | sean-k-mooney | actully 3 | |
| 17:50:34 | sean-k-mooney | actully no its 7 i didnt scroll im going to check if those need to be elevated or not | |
| 17:51:56 | sean-k-mooney | ya so all of them pass run_as_root=true which means they actully use privsep to run them | |
| 17:52:03 | sean-k-mooney | because we hit here https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f28b7e6ff/os_brick/privileged/rootwrap.py#L169 | |
| 17:52:23 | sean-k-mooney | and execute_root is a privsep function https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f28b7e6ff/os_brick/privileged/rootwrap.py#L190-L194 | |
| 17:59:00 | sean-k-mooney | it looks like oslo rootwrap was removed as a os-bricks depency in liberty too https://github.com/openstack/os-brick/commit/7d341f154397ef38c480bdd9cf6d2ccad6502f13 but https://github.com/openstack/os-brick/commit/dbf77fba1061cb4e93b3db5f8117d6ccc689f702#diff-0d141267b46cdfd7a9dfe6100d79fe33 removed it as an actul depency in newton | |
| 18:01:32 | sean-k-mooney | the way os-bricks is using privsep is not secure but i dont think its usign the filters at all | |
| 18:42:49 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713961 | |
| 18:43:39 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713962 | |
| 19:03:44 | artom | dansmith, quick chat about https://review.opendev.org/#/c/705545/4/nova/tests/functional/integrated_helpers.py@370 ? | |
| 19:10:53 | dansmith | artom: just back to my desk, have to do something first and then yeah | |
| 19:12:48 | artom | dansmith, so, I looked around some more, thought some more, and I think I've come around to your way of thinking | |
| 19:12:57 | artom | I gotta go walk the kiddos | |
| 19:13:21 | dansmith | heh okay | |
| 19:34:21 | openstackgerrit | Merged openstack/nova stable/rocky: Fix os-keypairs pagination links https://review.opendev.org/711898 | |
| 21:06:20 | artom | dansmith, hrmm, so in the end maybe talking about https://review.opendev.org/#/c/705545/4/nova/tests/functional/integrated_helpers.py@370 could help | |
| 21:07:50 | artom | So, places that redefine _setup_compute_service just don't inherit from the ServicesMixing | |
| 21:08:14 | artom | I think I brought about some confusion when I called it the ServicesHelper mixin | |
| 21:08:22 | artom | It's really the PeriodicHelperMixin | |
| 21:08:30 | artom | Ie - you only get run_periodics in it | |
| 21:08:51 | artom | But because run_periodics depends on self.computes, you get _start_compute as well, which populates self.compute | |
| 21:09:18 | openstackgerrit | Merged openstack/nova stable/train: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713739 | |
| 21:09:24 | openstackgerrit | Merged openstack/nova stable/queens: Avoid circular reference during serialization https://review.opendev.org/713132 | |
| 21:09:31 | openstack | bug 1849409 in OpenStack Compute (nova) ocata "openstack server list --deleted --limit -1 hangs" [High,In progress] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem) | |
| 21:09:31 | openstackgerrit | Merged openstack/nova stable/ocata: Add functional regression test for bug 1849409 https://review.opendev.org/690746 | |
| 21:09:36 | openstackgerrit | Merged openstack/nova stable/ocata: Fix listing deleted servers with a marker https://review.opendev.org/690747 | |
| 21:09:41 | openstackgerrit | Merged openstack/nova stable/ocata: Fix incompatible version handling in BuildRequest https://review.opendev.org/660456 | |
| 21:09:48 | openstackgerrit | Merged openstack/nova stable/ocata: Hook resource_tracker to remove stale node information https://review.opendev.org/676467 | |
| 21:14:53 | openstackgerrit | melanie witt proposed openstack/nova stable/stein: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713982 | |
| 21:17:09 | openstackgerrit | Merged openstack/nova master: Switching new default roles in os-volumes-attachments policies https://review.opendev.org/710190 | |
| 21:48:53 | melwitt | zzzeek: I've been looking at some debug logs in gate runs and I notice a lot of ROLLBACK logged, would you think it's concerning? https://zuul.opendev.org/t/openstack/build/b4f6a11e4b0940e0a9513f7113bfb95c/log/logs/screen-n-sch.txt#1691 | |
| 21:50:12 | zzzeek | melwitt: when a connection is done being used, it is returned to the pool where a ROLLBACK is emitted. if you have lots of short single-statement transactions then you'd see a lot of these | |
| 21:50:33 | melwitt | zzzeek: oh, I see. thanks | |
| 21:50:49 | zzzeek | melwitt: these can be a problem performance wise | |
| 21:51:03 | zzzeek | but only if yo're observing that | |
| 21:53:01 | melwitt | zzzeek: yeah ... not yet sure what we're seeing, but this is the file where we have the gate bug where we wait 60 seconds without getting rows back from the database. could be a bug on our side. but I'm studying the logs looking for anything that seems like it could cause trouble. what would be a way of avoiding the rollbacks? for the sake of experiment | |
| 22:02:14 | zzzeek | melwitt: there's a parameter "rollback_on_return" to craete_engine | |
| 22:02:35 | zzzeek | oops, that's not it, one sec | |
| 22:03:01 | melwitt | rollback_reader_sessions ? | |
| 22:03:11 | zzzeek | reset_on_return | |
| 22:03:19 | melwitt | oh k thanks | |
| 22:03:19 | zzzeek | https://docs.sqlalchemy.org/en/13/core/engines.html?highlight=reset_on_return#sqlalchemy.create_engine.params.pool_reset_on_return | |
| 22:03:46 | zzzeek | but...it's dangerous to use this | |
| 22:04:20 | zzzeek | in fact it can't actually work correctly, because there is always a transaction with MySQL InnoDB | |
| 22:04:29 | zzzeek | so you will have very poor effects from using htis | |