| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-05-01 | |||
| 13:01:10 | opendevreview | sean mooney proposed openstack/nova master: [WIP] use alpine instead of cirros https://review.opendev.org/c/openstack/nova/+/881912 | |
| 13:04:28 | opendevreview | sean mooney proposed openstack/nova master: [WIP] use alpine instead of cirros https://review.opendev.org/c/openstack/nova/+/881912 | |
| 13:31:56 | dansmith | sean-k-mooney: so check this out: https://264a86dd518c5142b8a6-3b04393c7506ab488b4fde073fc22e36.ssl.cf1.rackcdn.com/881764/1/check/cinder-tempest-plugin-lvm-multiattach/d28afe7/testr_results.html | |
| 13:32:01 | dansmith | maybe kashyap too ^ | |
| 13:32:25 | dansmith | that almost kinda looks like we attached the disk to the guest and the kernel crashed while probing partitions/filesystems, right? | |
| 13:33:10 | dansmith | in sysfs_add_file_mode_ns() | |
| 13:33:49 | dansmith | or perhaps while creating sysfs entries for the disk perhaps? | |
| 16:30:56 | dansmith | eharney: you around by chance? | |
| 16:45:20 | eharney | dansmith: yes | |
| 16:45:43 | dansmith | eharney: so, I seem to have gotten the nova ceph job down to a single repeatable failure | |
| 16:45:56 | dansmith | it is in the volume extend test, and it fails during cleanup | |
| 16:46:25 | dansmith | it's trying to, I guess, detach the volume from the server before deleting it and then before deleting the volume | |
| 16:46:49 | dansmith | it does *not* fail locally, so I don't think it's something fundamentally broken with new ceph or anything like that | |
| 16:46:54 | dansmith | https://955f32f8268e5d475e65-6c8f4c6e546a0854b4c11cc7c78829ca.ssl.cf5.rackcdn.com/881585/6/check/nova-ceph-multistore/1ecb09a/testr_results.html | |
| 16:47:25 | eharney | dansmith: let me take a look through the logs | |
| 16:47:49 | eharney | detach failing isn't something i'm familiar with (other than it being mentioned here the other day) | |
| 16:48:04 | dansmith | I've just been tracing through the code looking for how this works and it seems to me like everything is working, but perhaps it's just legitimately that the guest doesn't let go of the volume when we detach after the resize happened | |
| 16:48:17 | dansmith | okay, volume detach is without a doubt our most common failure | |
| 16:48:34 | dansmith | eharney: yeah appreciate if you could see if you spot anything in the logs | |
| 17:00:13 | dansmith | I see the guest saw the size change on vdb and also mentions that it's resizing the filesystem on it... that looks like more than just the block device size change, | |
| 17:00:41 | dansmith | so I wonder if it is literally doing a resize2fs on it and that is still happening when we try to issue the detach and that gets us stuck | |
| 17:01:02 | dansmith | because we start the detach less than half a second after the resize happens | |
| 17:03:32 | eharney | dansmith: yeah, i was also just looking down the path of whether the libvirt block resize call is synchronous or not (it looks like nova assumes it is?) | |
| 17:04:47 | dansmith | eharney: synchronous with what? It's synchronous to libvirt from compute, but I don't know that it waits to return until it's delivered to the guest, but definitely stuff like resizing filesystems would happen after that returns | |
| 17:05:08 | dansmith | even still, the test polls for completion of the operation as far as nova is concerned, but the guest stuff would all be async | |
| 17:05:34 | eharney | i see | |
| 17:05:56 | dansmith | I think maybe I should make the test ssh to the guest and see if it's mounted, and perhaps try to unmount it before the test ends or something, to avoid racing with the detach | |
| 17:11:10 | dansmith | the resize happens before cirros is even done with its startup stuff, | |
| 17:11:36 | dansmith | so I wonder on a slow emulated guest we mount all the filesystems we find, which means it's mounted in the guest when we resize, so it does the resize2fs activity automatically | |
| 17:11:58 | dansmith | but on a fast non-nested local run, it finishes startup before we do the attach and thus doesn't end up with it mounted during the resize | |
| 17:12:29 | eharney | is resize2fs etc triggered by qemu-guest-agent? | |
| 17:13:08 | dansmith | could be.. does cirros have the guest agent in it? I assumed not | |
| 17:13:22 | eharney | i don't think so | |
| 17:13:25 | dansmith | on regular systems I've never had resize2fs triggered automatically for me, so I'm not really sure why that would happen | |
| 17:13:30 | dansmith | but it seems like it is here | |
| 17:14:00 | eharney | i would have guessed that resize2fs doesn't happen in these test jobs | |
| 17:14:10 | dansmith | me too | |
| 17:14:12 | dansmith | you see it in the output though right? | |
| 17:14:26 | eharney | no, where is that? | |
| 17:14:27 | dansmith | [ 48.156649] EXT4-fs (vda1): resizing filesystem from 25600 to 259835 blocks | |
| 17:14:27 | dansmith | [ 48.255755] EXT4-fs (vda1): resized filesystem to 259835 | |
| 17:14:40 | dansmith | in the guest console dump | |
| 17:14:45 | dansmith | oh damn | |
| 17:14:53 | dansmith | that is vda, nevermind! | |
| 17:15:08 | dansmith | that's cirros resizing its root disk on startup, not the attached volume | |
| 17:15:10 | eharney | ah, right | |
| 17:15:21 | dansmith | mah bad | |
| 17:17:19 | dansmith | so, without a doubt the most common failure in nova jobs is failing to detach volumes | |
| 17:17:33 | dansmith | we've been trying to get a handle on it for a long time, | |
| 17:17:42 | eharney | does it show up on non-rbd volumes? | |
| 17:17:48 | dansmith | yeah | |
| 17:18:13 | dansmith | there is some assertion that if we attach to an instance before it is far enough along during boot, then it might prevent it from being detached later | |
| 17:18:26 | dansmith | I'm slightly skeptical of that, but we've been adding "wait for sshable" checks everywhere | |
| 17:19:01 | eharney | i guess i'm not sure what kind of conditions in the libvirt area would prevent detach from completing | |
| 17:19:17 | dansmith | I just added that for this test recently (merged on friday) which passes normally (and passes locally) but with this rbd job it seems to fail.. it passed the gate on the focal-based rbd, but not on new ceph and jammy | |
| #openstack-nova - 2023-05-02 | |||
| 08:07:20 | kashyap | dansmith: I was off yesterday (public holiday). Looking at the scrollback now about "tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest" | |
| 09:38:13 | opendevreview | Elod Illes proposed openstack/nova master: Add nova-tox-functional-py310 to gate jobs https://review.opendev.org/c/openstack/nova/+/881339 | |
| 11:19:55 | sfinucan | bauzas: addressed your comments on https://review.opendev.org/c/openstack/python-openstackclient/+/881822 lemme know if they work | |
| 11:57:57 | sean-k-mooney | sfinucan: im ok wiht osc generating keypairs client side | |
| 11:58:17 | sean-k-mooney | althoug it proably should be in its own command but i understand why we might wasnt to proxy it form the nova one | |
| 11:59:23 | sean-k-mooney | sfinucan: can i suggest we do both. proceed with bauzas patch as is and then add your change on top to add client side generation | |
| 11:59:39 | sean-k-mooney | the keypair type would be changing so they are not really interchangable | |
| 12:00:05 | sean-k-mooney | they kind of are but old operating systsms wont supprot ssh-ed25519 | |
| 12:00:25 | sean-k-mooney | i.e. centos 7 and newer OSs wont support RSA i.e. centos 9 | |
| 12:02:42 | sean-k-mooney | so i see you are proposing taking the approch of always generating htem client side isntaed fo doing this based on the microversion | |
| 12:03:21 | sean-k-mooney | that actully has some benifits in that the behaivor wont be "incorrect" for the new microverion | |
| 12:03:50 | sean-k-mooney | i.e. the client wont be giving the perception of the api generating the key for the latest microversion | |
| 12:04:02 | sean-k-mooney | since now it will be alwasy client side. | |
| 12:56:59 | sfinucan | sean-k-mooney: yeah, I'd rather not have different behaviour depending on the microversion in use | |
| 12:58:22 | bauzas | sfinucan: sean-k-mooney: I'm OK with generating private keys by default in OSC even if the API microversion is older than 2.92 | |
| 12:58:50 | bauzas | since the API was able to provide a private key by passing it, I'm ok | |
| 13:03:13 | sean-k-mooney | ok | |
| 13:03:19 | sean-k-mooney | so how do you want to proceed | |
| 13:03:24 | sean-k-mooney | just go with sfinucan patch | |
| 13:03:33 | sean-k-mooney | or merge both? | |
| 13:04:42 | sfinucan | I think it's a case of one or the other | |
| 13:06:25 | stephenfin | sean-k-mooney: if we start doing stuff client-side, there's no reason not to do it for all microversions. If we stick server-side, we're obliged to drop key generation functionality for newer or all microversions | |
| 13:11:19 | stephenfin | sean-k-mooney: bauzas: btw, do you have +2 on OSC now? You should? | |
| 13:12:48 | bauzas | stephenfin: my only concern if we merge your patch is that enduser wouldn' longer see that they need to create their own keypairs | |
| 13:13:26 | bauzas | stephenfin: so if they use openstacksdk directly, they would see then | |
| 13:13:33 | sean-k-mooney | i can check | |
| 13:13:46 | stephenfin | wdym? They don't. We're still doing it for them, only on the client rather than the server | |
| 13:13:48 | stephenfin | Ah | |
| 13:14:08 | stephenfin | I mean, I think that's okay. We do a whole load of helpful things in OSC that don't happen in SDK | |
| 13:15:36 | stephenfin | Like the 'server ssh' command. There's no equivalent for that in SDK, mainly because it's not needed. We could add a docstring in SDK noting that the private key field must be explicitly provided in newer microversions | |
| 13:18:33 | sean-k-mooney | stephenfin: for osc i do not have +2 currently | |
| 13:18:54 | sean-k-mooney | likely the same for sdk i have not actuly looked | |
| 13:26:08 | bauzas | ditto, I'm not osc-core (and not sure I want it :) ) | |
| 13:27:19 | dansmith | kashyap: thanks for looking..that looks like a kernel bug to me and so it'd be good to know if it's been fixed as it's not uncommon to see it | |
| 13:27:31 | stephenfin | Ah, okay, it seems I'd misunderstood what 'python-openstackclient-service-core' was being used for. I'd thought it would contain all the service core groups by default. Evidently not. | |
| 13:28:27 | stephenfin | bauzas: https://review.opendev.org/c/openstack/openstacksdk/+/881965 | |
| 13:29:15 | kashyap | dansmith: Yeah, it looks like "somehow" the loading of the kernel modules itself is failing: "failed loading these modules: nls_ascii nls_iso8859-1 nls_utf8 ip_tables ahci" | |
| 13:30:21 | kashyap | Also, I hate to say this, but I wonder if it's CirrOS-specific | |
| 13:30:57 | kashyap | dansmith: I take it that this segfault is consistently reproducible | |
| 13:31:54 | dansmith | kashyap: I can't push a button and make it happen, but I see it fairly often | |
| 13:32:21 | kashyap | dansmith: Noted; I'm trying to wade through the logs to also find the QEMU command-line of this. And the host kernel version | |
| 13:32:28 | dansmith | kashyap: entirely possible that it's cirros related, but since this there is not much going on in userland and this is a kernel crash, I imagine it's not the actual fault of cirros | |
| 13:32:36 | kashyap | Have you got the instance name / UUID, per chance? | |
| 13:35:42 | dansmith | kashyap: if you look right above the top of the console dump you'll see requests tempest was making to watch the status and you should find the uuid there | |
| 13:35:59 | kashyap | Okido, this seems the UUID: e0dc4c98-95e2-4421-a6ba-ae07213914a7 | |