| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-24 | |||
| 19:08:40 | gouthamr | saw your comment: "There are errors from one of the nova computes that indicate that it's not getting access to the ceph cluster properly, which means there is more work to do here." | |
| 19:09:08 | gouthamr | so assumed that was on the CI somewhere; ashrodri can enable the base job in the repo with her changes and debug that | |
| 19:09:58 | dansmith | gouthamr: oh, that's probably from the nova test patch, which likely has some logs still because it has a multinode job | |
| 19:10:05 | dansmith | hang on | |
| 19:10:34 | dansmith | gouthamr: https://zuul.opendev.org/t/openstack/build/588de1c62e564a4e93d4e94d1e9a5216 | |
| 19:11:18 | dansmith | yeah so all the migration tests fail which I assume is because the other node can't access the ceph cluster | |
| 19:11:25 | dansmith | (sorry I had purged all this from my cache) | |
| 19:12:34 | dansmith | ceph conf and keyrings are there on the subnode | |
| 19:13:00 | dansmith | but nova can't see it: https://zuul.opendev.org/t/openstack/build/588de1c62e564a4e93d4e94d1e9a5216/log/compute1/logs/screen-n-cpu.txt#8038 | |
| 19:18:10 | gouthamr | dansmith: it looks like we created two ceph clusters there :) | |
| 19:18:29 | gouthamr | with REMOTE_CEPH set to true, we should skip the ceph install on the subnode | |
| 19:18:38 | gouthamr | which we'll try with ashrodri's patch.. | |
| 19:19:32 | dansmith | gouthamr: ack, makes sense | |
| #openstack-nova - 2023-05-25 | |||
| 02:13:04 | gmann | dansmith: can you re-review this, removed glance scope setting also as that is also now enabled in devastack by default https://review.opendev.org/c/openstack/nova/+/883557 | |
| 02:13:32 | gmann | gibi: bauzas: ^^ can either of you check this RBAC old default new job | |
| 04:37:08 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | |
| 04:37:09 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 04:37:09 | opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | |
| 04:37:10 | opendevreview | melanie witt proposed openstack/nova master: Support resize with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870933 | |
| 04:37:10 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 04:37:11 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 04:37:11 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 04:37:12 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | |
| 04:37:12 | opendevreview | melanie witt proposed openstack/nova master: Support rescue with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/873675 | |
| 04:37:13 | opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | |
| 04:37:13 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 04:37:15 | opendevreview | melanie witt proposed openstack/nova master: block_device: Add encryption attributes to swap disks https://review.opendev.org/c/openstack/nova/+/884312 | |
| 04:37:15 | opendevreview | melanie witt proposed openstack/nova master: WIP raw imagebackend support https://review.opendev.org/c/openstack/nova/+/884313 | |
| 05:01:10 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 13:40:07 | dansmith | gmann: yep | |
| 14:17:09 | kashyap | dansmith: That CirrOS 0.5.2 is fixed now: https://github.com/cirros-dev/cirros/issues/102 | |
| 14:17:19 | kashyap | They bumped the kernel. | |
| 14:17:33 | sean-k-mooney | finally | |
| 14:17:34 | dansmith | kashyap: hah, we just bumped to 6.0 :) | |
| 14:17:39 | kashyap | LOL | |
| 14:17:52 | dansmith | but yeah, I need to survey for guest kernel crashes since the bump | |
| 14:17:56 | sean-k-mooney | we are on 6.1 no? | |
| 14:18:21 | kashyap | Oh, wait - there are more guest kernel crashes after the bump? | |
| 14:18:21 | sean-k-mooney | i asked them to bump the kernel like 2 years ago to fix some of the kernel issues | |
| 14:18:27 | dansmith | I meant 6.x | |
| 14:19:11 | sean-k-mooney | thats a diffent kernel fix then the one i asked them to bump for before | |
| 14:19:53 | sean-k-mooney | but in anycase we are not using that anymore but it might be useful on stable i guess | |
| 14:20:08 | dansmith | kashyap: no, I'm saying we bumped to cirros 6.x a few days ago and I need to survey to see if the kernel crashes have stopped | |
| 14:20:27 | kashyap | Aaah, like that | |
| 14:20:30 | kashyap | Makes sense, yes. | |
| 14:21:34 | kashyap | sean-k-mooney: Which one did you ask? We asked this update to solve these sort of crashes: https://kashyapc.fedorapeople.org/CirrOS_segfault/guest-boot-console-and-segfault.txt | |
| 14:21:48 | sean-k-mooney | kashyap: it was related to https://github.com/cirros-dev/cirros/issues/69 | |
| 14:23:04 | sean-k-mooney | we worked aroudn it with https://github.com/lyarwood/cirros/commit/c8972ca3dcde2bedc00cbeda0e0ccb8fcbf92883 but chattign to lee i found a fixed kernel bug that had been released in an newer verison | |
| 14:23:18 | kashyap | Oh, this, the IO-APIC one. | |
| 14:23:47 | sean-k-mooney | ya so there was a fix for the apic issue in a later kernel package | |
| 14:23:56 | sean-k-mooney | we worked around it with the kernel parmater | |
| 14:24:09 | sean-k-mooney | but the package never got updated at least not at the time | |
| 14:24:17 | sean-k-mooney | it may have been since | |
| 14:24:43 | kashyap | Ah, nod | |
| 15:35:25 | dansmith | kashyap: from this morning: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f3e/883602/2/gate/nova-ceph-multistore/f3e5585/testr_results.html | |
| 15:35:35 | dansmith | guest kernel crash, but it's the IO-APIC variant | |
| 15:36:11 | kashyap | dansmith: Ohh, is it with CirrOS 6.1? | |
| 15:36:25 | dansmith | should be, it's from this morning on master | |
| 15:36:57 | kashyap | If it's IO-APIC variant one; then the immediate thing that comes to mind is does this image also needs "no_timer_check" to its kernel parameter | |
| 15:37:01 | dansmith | yep https://zuul.opendev.org/t/openstack/build/f3e558513e3f4a6c8e9c0ceb50a023e1/log/job-output.txt#2056 | |
| 15:37:29 | dansmith | well, it ran a hundred other tests with that image and configuration of course | |
| 15:37:42 | dansmith | but if that's something we should always be using in devstack, you should add that :) | |
| 15:39:19 | kashyap | Understood; I've added it to my plate to see if there's any other way around here, besides the kernel parameter | |
| 15:40:53 | frickler | kashyap: I missed that kernel bump issue because I was on holiday. iiuc it will still need a release, like 0.5.3? | |
| 15:41:12 | kashyap | dansmith: Why do you mention the hundred other tests? Is it as a "contrasting" success of what tests are actually passing? | |
| 15:41:36 | dansmith | kashyap: no I'm saying it's clearly not "this requires this flag to work at all" | |
| 15:41:38 | kashyap | frickler: See the issue mentioned here: https://github.com/cirros-dev/cirros/issues/102 | |
| 15:41:59 | frickler | kashyap: yes, I don't see a release for that | |
| 15:42:40 | frickler | I'l check with smoser | |
| 15:42:59 | kashyap | dansmith: Aah, like that. Noted. I was only thinking of th IO-APIC + timer panic | |
| 15:43:25 | kashyap | frickler: smoser seems to point me to a build? | |
| 15:43:43 | kashyap | frickler: Please mention your thoughts on that ticket, to keep everything in one place | |
| 15:45:58 | frickler | yes, that's kind of a testing build. updated the issue, good point | |
| 19:53:09 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Translate VF network capabilities to port binding https://review.opendev.org/c/openstack/nova/+/884439 | |
| 22:19:13 | opendevreview | melanie witt proposed openstack/nova master: WIP raw imagebackend support https://review.opendev.org/c/openstack/nova/+/884313 | |
| 22:20:34 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| #openstack-nova - 2023-05-26 | |||
| 00:25:19 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to convert_image https://review.opendev.org/c/openstack/nova/+/870934 | |
| 00:25:19 | opendevreview | melanie witt proposed openstack/nova master: Add hw_ephemeral_encryption_secret_uuid image property https://review.opendev.org/c/openstack/nova/+/870935 | |
| 00:25:20 | opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | |
| 00:25:20 | opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/870937 | |
| 00:25:21 | opendevreview | melanie witt proposed openstack/nova master: Support rescue with ephemeral encryption https://review.opendev.org/c/openstack/nova/+/873675 | |
| 00:25:21 | opendevreview | melanie witt proposed openstack/nova master: Update driver BDMs with ephemeral encryption image properties https://review.opendev.org/c/openstack/nova/+/870939 | |
| 00:25:22 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | |
| 00:25:22 | opendevreview | melanie witt proposed openstack/nova master: WIP raw imagebackend support https://review.opendev.org/c/openstack/nova/+/884313 | |
| 00:32:25 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 00:43:39 | opendevreview | Wenping Song proposed openstack/nova stable/victoria: Trival Change: Remove redundant code in instance delete https://review.opendev.org/c/openstack/nova/+/796912 | |
| 02:40:09 | opendevreview | melanie witt proposed openstack/nova master: DNM test ephemeral encryption + resize: qcow2, raw https://review.opendev.org/c/openstack/nova/+/862416 | |
| 14:37:30 | lowercase | sean-k-mooney: It appears we are hitting a bug where instances are not honoring availability zone placement even though we have cinder_Cross_az attach set to false. Is this ringing any bells or anything? | |
| 14:37:36 | lowercase | Xena release | |
| 15:01:41 | bauzas | lowercase: are you sure that the instances are not pinned ? | |
| 15:05:25 | lowercase | bauzas: yes, we don't do cpu pinning. | |
| 15:06:10 | bauzas | lowercase: no sorry, I mean, pinned to an AZ | |
| 15:07:05 | lowercase | how would we pin them to an AZ? | |
| 15:08:34 | lowercase | so what is happening is that a when we attempt to resize an instance, the nova scheduler can (in our experience) pick a host outside of the AZ where the instance is currently is, because we have cinder_cross_az attach set to false | |
| 15:08:47 | lowercase | because we have multiple ceph clusters, 1 ceph cluster per AZ | |
| 15:13:31 | lowercase | https://docs.openstack.org/nova/xena/admin/availability-zones.html#implications-for-moving-servers | |
| 15:13:38 | lowercase | #4 is our condition | |
| 15:30:58 | sean-k-mooney | lowercase: the schgduler is allowed ot pick an instance outside the current az | |
| 15:31:14 | sean-k-mooney | lowercase: it is only not allowe to do that if the instnace is create reqeusting a specific AZ | |
| 15:31:55 | lowercase | is there anyway I can retro-give an instance the paramter to keep the instance in a az | |