| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-25 | |||
| 21:58:00 | gouthamr | dansmith: yes | |
| 21:58:02 | dansmith | resource contention like memory/ | |
| 21:58:10 | gouthamr | yes, and disk | |
| 21:58:26 | dansmith | so there's a tweak in devstack for memory that has been helping a lot of jobs, and we run with it enabled in the nova ceph jobs | |
| 21:58:39 | gouthamr | oh? i'd love to know! | |
| 21:58:43 | dansmith | drops mysql usage by about half, which is ~400MiB on these jobs | |
| 21:58:47 | gouthamr | nice | |
| 21:58:55 | dansmith | gouthamr: problem is you have to pay me a royalty per job execution to use it | |
| 21:59:17 | dansmith | gouthamr: https://github.com/openstack/nova/blob/master/.zuul.yaml#L422 | |
| 21:59:33 | gouthamr | :D i'd be poor if i was betting on ceph using less memory ever | |
| 21:59:38 | dansmith | haha | |
| 22:00:03 | gouthamr | (2) there's also a way to turn off "cephadm" after the install -- we should set that option on the CI - there's no use to keep it running | |
| 22:00:06 | dansmith | so based on our success with that, if you're cool with it, I'd say we turn that on for these jobs anyway | |
| 22:00:24 | dansmith | gouthamr: meaning before we start running tempest? | |
| 22:01:30 | gouthamr | yes, the plugin will do that for you after the ceph cluster deployment is done: https://github.com/openstack/devstack-plugin-ceph/blob/563cb5deeb21815ce0c62fa30249e85e886c783a/devstack/lib/cephadm#L28 | |
| 22:02:12 | dansmith | oh so why is that not being set? | |
| 22:02:33 | dansmith | I don't even know what that means.. cephadm is a tool I thought, but you're saying it stays running even though the services are started or something? | |
| 22:02:36 | gouthamr | (here's an example from the only good cephadm job at the moment: https://github.com/openstack/manila-tempest-plugin/blob/ad0db6359c8c51c4521ac6660c8014981b2f1dea/zuul.d/manila-tempest-jobs.yaml#L413) | |
| 22:02:49 | dansmith | ack | |
| 22:02:56 | gouthamr | yes; we need it for day2 operations on the cluster | |
| 22:03:07 | gouthamr | so if you're running this on your local devstack, it's useful | |
| 22:03:13 | dansmith | ah, sure, okay | |
| 22:03:24 | dansmith | we should set that in these jobs that everyone inherits from | |
| 22:03:36 | gouthamr | ++ | |
| 22:03:42 | dansmith | so maybe a follow-on to this to set that, the mysql thing, and make this job voting | |
| 22:04:00 | dansmith | you know, if and when it starts working :) | |
| 22:04:40 | gouthamr | ^^ +1 | |
| 22:06:02 | dansmith | hmm, gouthamr I just noticed that we're still set to release=pacific for the distro-package-based job | |
| 22:06:11 | dansmith | is that maybe setting stuff that is non-ideal for quincy that could be related? | |
| 22:06:38 | gouthamr | i think the plugin ignores it, let me check | |
| 22:06:43 | dansmith | okay | |
| 22:06:59 | dansmith | https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315/11/.zuul.yaml#46 | |
| 22:08:43 | dansmith | okay just used for package repos anyway it looks like | |
| 22:08:51 | gouthamr | ack; we should get that opt out of the job because its confusing | |
| 22:08:53 | gouthamr | https://github.com/openstack/devstack-plugin-ceph/blob/563cb5deeb21815ce0c62fa30249e85e886c783a/devstack/lib/ceph#L981-L989 | |
| 22:09:43 | dansmith | ack, so I assume we're configuring a repo but just not installing anything from it on jammy, right? | |
| 22:10:02 | dansmith | I'll add it to my follow-on patch to add the optimization devstack vars | |
| 22:10:08 | dansmith | add .. the removal of it, I mean :) | |
| 22:10:13 | gouthamr | ack ty | |
| 22:10:20 | gouthamr | i may have messed this up in my last patch | |
| 22:10:21 | gouthamr | https://github.com/openstack/devstack-plugin-ceph/blob/563cb5deeb21815ce0c62fa30249e85e886c783a/devstack/lib/ceph#L1067-L1094 | |
| 22:10:47 | gouthamr | we're not invoking that method at all for ubuntu anymore.. | |
| 22:11:27 | dansmith | ack | |
| 22:11:45 | dansmith | well, there's a bunch of focal-specific stuff to clean up in there regardless | |
| 22:14:58 | dansmith | hmm, I think it's about to fail a test | |
| 22:15:11 | dansmith | been stopped for going on five minutes, assuming retrying a detach | |
| 22:15:38 | gouthamr | yes.. it was a head-scratcher; think vkmc and i noticed that our override to use download.ceph.com stopped working with focal at some point since ubuntu default-enabled the ubuntu ceph repos.. so dropping it made no difference, we ended up using/testing with the distro provided packages | |
| 22:15:43 | gouthamr | oh | |
| 22:16:19 | gouthamr | test_rebuild_server_with_volume_attached? | |
| 22:16:25 | dansmith | dunno yet | |
| 22:16:40 | gouthamr | ah | |
| 22:16:43 | dansmith | but it's in the rebuld group | |
| 22:16:46 | dansmith | yep | |
| 22:17:14 | dansmith | test_rebuild_server_with_volume_attached [430.305393s] ... FAILED | |
| 22:17:18 | dansmith | ugh | |
| 22:18:19 | dansmith | so the other variable here is the version of qemu and qemu's block-rbd driver are different in jammy of course, compared to what we've been testing | |
| 22:18:42 | dansmith | so could be a bug in one of those, especially since it's related to the detach in the guest | |
| 22:19:35 | gouthamr | ack; another thing to try would be to bump the ceph image to the latest quincy: https://github.com/openstack/devstack-plugin-ceph/blob/563cb5deeb21815ce0c62fa30249e85e886c783a/devstack/lib/cephadm#L32 | |
| 22:20:02 | dansmith | okay | |
| 22:20:17 | gouthamr | they've published v17.2.6 today, and v17.2.5 a month ago | |
| 22:21:01 | gouthamr | https://quay.io/repository/ceph/ceph?tab=tags | |
| 22:21:45 | dansmith | ack, I hate this sort of "version minesweeper" game.. if we're that sensitive to version, it feels like we're doing something wrong | |
| 22:22:39 | gouthamr | agreed; but since this stuff hasn't worked before on our ci, its worth a try | |
| 22:22:58 | dansmith | yeah for sure | |
| 22:23:18 | dansmith | volume resize passed | |
| 22:23:32 | dansmith | maybe we'll find that this is fewer fails or something | |
| 22:25:18 | dansmith | actually, that one didn't fail before | |
| 22:28:07 | gouthamr | okay, this may be good news? devstack-plugin-ceph-tempest-py3 is going to pass | |
| 22:28:34 | dansmith | no, really? | |
| 22:28:52 | dansmith | third recheck's a charm? | |
| 22:28:55 | gouthamr | :D | |
| 22:42:28 | dansmith | more fails on this cephadm job | |
| 22:42:48 | dansmith | so I guess it's not something fundamental, but maybe just massively less stable or we're hitting some race easier? | |
| 22:43:26 | dansmith | maybe it is memory-related and we're stressed more here | |
| 22:43:41 | dansmith | maybe I should try turning on the two optimizations here to see if that makes things more stable | |
| 22:43:54 | gouthamr | ++ | |
| 22:45:47 | gouthamr | "DISABLE_CEPHADM_POST_DEPLOY: true" and "MYSQL_REDUCE_MEMORY: true" for the rescue; we can iterate after with the concurrency if these failures reduce | |
| 22:45:54 | gouthamr | or go away | |
| 22:46:11 | opendevreview | Artom Lifshitz proposed openstack/nova master: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/862967 | |
| 22:46:12 | opendevreview | Artom Lifshitz proposed openstack/nova master: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/862964 | |
| 22:46:21 | dansmith | gouthamr: ack, will put those in here and see | |
| 22:53:15 | opendevreview | Merged openstack/nova master: Remove focal job for 2023.2 https://review.opendev.org/c/openstack/nova/+/881409 | |
| 23:25:39 | dansmith | gouthamr: okay, it's off and running with those flags | |
| 23:25:53 | dansmith | I'm burnt out so I'll circle back tomorrow | |
| 23:27:32 | gouthamr | dansmith++ works; good evening! :) | |
| #openstack-nova - 2023-04-26 | |||
| 00:01:40 | opendevreview | Merged openstack/nova master: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/862967 | |
| 07:55:12 | gibi | auniyal, bauzas, sean-k-mooney my recent finding about test not waiting for sshable before attach/detach are in https://bugs.launchpad.net/nova/+bug/1998148 but I have no cycles to push tempest changes so feel free to jump on it | |
| 07:56:33 | bauzas | gibi: ack, but I'm atm still digesting dansmith's and gouthamr efforts on ceph/Jammy jobs | |
| 08:38:25 | gibi | sure. I just wanted to be explict about the fact that I won't push a solution | |
| 08:51:41 | bauzas | gibi: I'll try to help this afternoon then | |
| 08:51:49 | bauzas | thanks for the findings again | |
| 11:31:16 | ykarel | Hi bauzas | |
| 11:31:29 | ykarel | reported https://blueprints.launchpad.net/nova/+spec/libvirt-tb-cache-size as discussed yesterday, can you please check | |
| 11:32:34 | bauzas | ykarel: ack, approving it then | |
| 11:34:00 | bauzas | and done | |
| 11:34:41 | ykarel | Thanks bauzas | |
| 11:34:55 | bauzas | np | |
| 11:37:23 | sean-k-mooney | ykarel: once we get the ceph/py38 issues resolved im fine with prioritsing getting this landed too | |
| 11:37:59 | ykarel | sean-k-mooney, ack | |
| 11:38:42 | opendevreview | Elod Illes proposed openstack/nova master: Drop py38 based zuul jobs https://review.opendev.org/c/openstack/nova/+/881339 | |