| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-04 | |||
| 16:18:49 | mriedem | see https://review.opendev.org/#/c/679274/ | |
| 16:20:26 | mriedem | you could also take a look at this https://review.opendev.org/#/c/640516/ | |
| 16:20:55 | mriedem | "If someone comes forward saying "yes this fixes a thing I'm hitting" then I'd be willing to restore it." | |
| 16:21:38 | mriedem | that second patch won't help you on queens since it relies on the force_refresh kwarg from the previous patch | |
| 16:21:53 | sean-k-mooney | mriedem: for what its worth we have backported only the periodic heal downstream | |
| 16:22:06 | mriedem | super | |
| 16:22:09 | sean-k-mooney | we did not backport the data migration | |
| 16:22:18 | sean-k-mooney | because that caused other issues | |
| 16:22:19 | mriedem | and canonical is going to as well it looks like | |
| 16:23:02 | sean-k-mooney | we have it all the way back to newton at this point | |
| 16:23:31 | sean-k-mooney | although we only did newton in the last month | |
| 16:23:34 | salmankhan | So you saying apply this 679274, that could fix on queens? | |
| 16:24:18 | sean-k-mooney | only if you have fixed teh underlying network issue | |
| 16:24:51 | sean-k-mooney | this only happens if there was an external failure that casued the cache to be curropted | |
| 16:25:43 | salmankhan | I messed up with neutron ports policy and that probably corrupted the cache, I think. | |
| 16:25:57 | sean-k-mooney | yep that is one of the know ways to cause it | |
| 16:25:57 | salmankhan | though I have fixed that policy | |
| 16:26:27 | sean-k-mooney | if the neutron policy.json is miss configured an empty list will be returned form neutron | |
| 16:26:33 | sean-k-mooney | and then we update the caceh with that | |
| 16:26:50 | sean-k-mooney | if that has been fixed the back can fix that | |
| 16:27:11 | salmankhan | ok, :finger_crossed: going to apply 679274 and see. | |
| 16:32:55 | stephenfin | dansmith: Any idea what could cause this test to fail? https://zuul.opendev.org/t/openstack/build/c598788184854780b2494a45b42a9718/console | |
| 16:33:07 | stephenfin | (from https://review.opendev.org/697156) | |
| 16:33:58 | dansmith | mucking around with the registry | |
| 16:34:01 | dansmith | oh | |
| 16:34:14 | stephenfin | I thought maybe I'd removed an object that was referenced from another object, but it doesn't seem like I have | |
| 16:34:22 | dansmith | stephenfin: did you maybe remove or not register an object? | |
| 16:34:39 | stephenfin | I removed objects, yes | |
| 16:34:47 | stephenfin | But I've removed all references to those objects, fwict | |
| 16:34:48 | dansmith | did you remove one that you didn't remove from the test_objects list of hashes? | |
| 16:35:44 | dansmith | unrelated to your failure, did we talk about removing objects already? | |
| 16:35:48 | stephenfin | grep doesn't bring up any matches for the removed objects in 'nova/tests/unit/objects/' | |
| 16:36:18 | stephenfin | I think so. You've to keep the DB models if it's persistent for a cycle or two, but the objects can go | |
| 16:36:21 | stephenfin | I think... | |
| 16:37:53 | dansmith | so, removing these objects will mean that if anything sends us one over the wire, we will completely fail, | |
| 16:38:08 | dansmith | and obviously things stop being able to run remotable methods if they're not upgraded | |
| 16:38:55 | stephenfin | Then I will triple check, but I think the only thing that could have sent these over the wire was nova-network, and that has been non-functional for at least a cycle now | |
| 16:39:13 | dansmith | stephenfin: no, I think api would send them in some cases wouldn't it? | |
| 16:39:44 | dansmith | note I'm not saying we should keep them necessarily, just trying to think aloud | |
| 16:40:02 | stephenfin | yup, I get ya. Checking now myself | |
| 16:41:41 | stephenfin | Yeah, leave it with me. I'll check my queue of patches and see if any of those objects could get through to the APIs. I'll state whatever I find in the commit message | |
| 16:56:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: ironic: report a custom trait for the node owner https://review.opendev.org/697184 | |
| 16:56:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add node owner pre-filter https://review.opendev.org/697187 | |
| 16:56:47 | mriedem | efried: i wrote a functional test for ^ and it shows what i suspected - traits aren't going to work | |
| 16:56:49 | mriedem | not enough context | |
| 16:57:11 | efried | mriedem: yeah, I was noodling that yesterday and I concur it would be pretty tough to do. | |
| 16:57:50 | efried | mriedem: If you had a I_AM_IRONIC trait, and Placement had a ?required_if=CUSTOM_OWNER_XXX:I_AM_IRONIC syntax.... | |
| 16:58:19 | mriedem | and bingo bango you have metadata in placement :) | |
| 17:05:41 | sean-k-mooney | im pretty sure when i raised the idea of key:value traits in the past it was shot down pretty hard | |
| 17:10:59 | dansmith | please god no | |
| 17:11:47 | sean-k-mooney | yep that was the reaction back then too | |
| 17:18:57 | efried | I was not seriously proposing that. | |
| 17:19:13 | efried | forgot my /s | |
| 17:26:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add NodeOwnerFilter https://review.opendev.org/697331 | |
| 18:51:16 | salmankhan | sean-k-mooney: Thanks mate! This fix worked for me https://review.opendev.org/#/c/679274/ | |
| 18:58:35 | eandersson | Off topic but who do you poke to get Debian packages updated? | |
| 18:59:10 | frickler | eandersson: zigo | |
| 18:59:21 | eandersson | Thanks frickler | |
| 19:13:48 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: Spec: Ussuri: Encrypted Emulated Virtual TPM https://review.opendev.org/686804 | |
| 19:14:12 | efried | gibi, stephenfin, sean-k-mooney: Thanks for those reviews. Hopefully addressed everything ^ | |
| 19:19:54 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698 | |
| 19:31:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add NodeOwnerFilter https://review.opendev.org/697331 | |
| 19:35:02 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in Admin Actions https://review.opendev.org/657823 | |
| 21:04:04 | openstackgerrit | Merged openstack/nova stable/pike: Hook resource_tracker to remove stale node information https://review.opendev.org/676461 | |
| #openstack-nova - 2019-12-05 | |||
| 00:11:49 | openstackgerrit | Merged openstack/nova master: vmware: Remove vestigial nova-network support https://review.opendev.org/696503 | |
| 00:21:35 | openstackgerrit | sean mooney proposed openstack/nova master: Block rebuild when NUMA topology changed https://review.opendev.org/687957 | |
| 00:21:35 | openstackgerrit | sean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild https://review.opendev.org/689861 | |
| 00:30:01 | openstackgerrit | Merged openstack/nova master: zvm: Remove vestigial nova-network support https://review.opendev.org/696504 | |
| 01:24:31 | openstackgerrit | Merged openstack/nova master: Cache security group driver https://review.opendev.org/697122 | |
| 01:36:43 | openstackgerrit | jichenjc proposed openstack/nova master: libvirt: avoid cpu check at s390x arch https://review.opendev.org/696228 | |
| 01:51:03 | openstackgerrit | melanie witt proposed openstack/nova stable/queens: delete consumers which no longer have allocations https://review.opendev.org/697398 | |
| 04:19:58 | eandersson | mnaser what version of RabbitMQ are you running now? | |
| 06:47:23 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.opendev.org/694717 | |
| 07:07:49 | eandersson | I think I figured out the main issue we were having with instances getting stuck in BUILD. Many of our RabbitMQ Clusters got stuck into a bad state and because of how our scheduling works instances kept getting scheduled to the same computes. These computes had bad queues, but because it's almost impossible to know which compute(s) are affected. | |
| 07:08:40 | eandersson | Because the database is not yet updated to reflect the compute hosting the instance (or at least it's not easy, or obvious to find the compute without digging deep into the database) | |
| 07:09:16 | eandersson | I am still baffled on why so many RabbitMQ Clusters broke within a short period of time. Seems to be caused by some sort of memory leak on one of the RabbitMQ servers. | |
| 07:10:44 | eandersson | But we also saw it happen on one cluster after a network partition. | |
| 07:12:46 | eandersson | The only thing we could really find on RabbitMQ was messages stuck un'acked (even without consumers), but unsure if that was just a display issue. | |
| 08:11:14 | openstackgerrit | Luyao Zhong proposed openstack/nova-specs master: support live migration with virtual persistent memory https://review.opendev.org/695863 | |
| 08:20:57 | openstackgerrit | Luyao Zhong proposed openstack/nova-specs master: support live migration with virtual persistent memory https://review.opendev.org/695863 | |
| 10:08:54 | stephenfin | bauzas: Can you send this on its way? https://review.opendev.org/#/c/665518/ | |
| 10:09:14 | bauzas | stephenfin: sure | |
| 10:10:13 | stephenfin | Thanks | |
| 10:14:29 | bauzas | stephenfin: any chance to add a new tox target that would wrap pre-commit ? | |
| 10:14:55 | stephenfin | I could, but it doesn't really make sense. They're really different tools | |
| 10:15:50 | stephenfin | pre-commit "installs" things by cloning git repos and creating its own venv. It wouldn't benefit from tox | |
| 10:16:05 | zbr_ | stephenfin: it makes a lot of sense, haptics being one of them | |
| 10:16:17 | stephenfin | haptics? | |
| 10:16:26 | zbr_ | yeah, or habbits | |
| 10:16:44 | zbr_ | most devs are used to call "tox -e linters", as most projects in opestack have them. | |
| 10:16:46 | bauzas | stephenfin: i have to admit I'm new to pre-commit, but I'll give it a try | |
| 10:17:09 | stephenfin | zbr_: to be clear, this doesn't replace that - you can still call that | |
| 10:17:12 | bauzas | stephenfin: that said, I'm maybe a bit torn about saying we *could* deprecate fast8 | |
| 10:17:16 | zbr_ | mainly you change how we orchestrate linters without changing the user interface | |
| 10:17:17 | stephenfin | all that pre-commit adds is a way to call it automatically | |
| 10:17:27 | zbr_ | tox -> linters -> pre-commit -> .... | |
| 10:17:42 | stephenfin | no, it's more like | |
| 10:17:47 | zbr_ | mainly the only dependency listed in linters environment of tox would be "pre-commit" | |
| 10:17:49 | stephenfin | tox -> flake8 (hacking) | |
| 10:17:58 | stephenfin | pre-commit -> flake8 (hacking) | |