| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-01 | |||
| 09:51:38 | sean-k-mooney | do you need to run that job on fedora for a particalar reason by the way? im wondering if ubuntu 22.04 woudl see the same failure | |
| 09:54:05 | slaweq | sean-k-mooney (@sean-k-mooney:matrix.org) we just want to have fedora based periodic job, we run many Ubuntu jobs in check/gate queues already | |
| 09:54:25 | slaweq | maybe we could/should move it to c9s now | |
| 09:54:30 | slaweq | but for now it's fedora | |
| 09:57:26 | sean-k-mooney | slaweq: right i was going to suggest ye move to ubuntu 22.04 or centos 9 stream | |
| 09:57:43 | sean-k-mooney | i strongly dislike haveing fedora based testing in the gate | |
| 09:58:00 | sean-k-mooney | im supportive of c9s testing | |
| 09:59:21 | sean-k-mooney | im looking at the logs currently and waitign for the filters error logs to render | |
| 10:02:10 | sean-k-mooney | actully there are no error level logs in nova | |
| 10:02:28 | sean-k-mooney | so there were no excptions | |
| 10:02:36 | sean-k-mooney | i do see DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"type": "PortNotFound", "message": "Port 1a6c72ae-3e1a-45d6-a70e-1f9964b70756 could not be found.", "detail": ""}} | |
| 10:02:46 | sean-k-mooney | but that proably not related to the failure | |
| 10:03:04 | sean-k-mooney | im only looking at the novac compute currently | |
| 10:04:09 | gibi | slaweq: I see that the instance being snapshotted is crashed | |
| 10:04:11 | gibi | https://zuul.openstack.org/build/4a7f284f32eb436da6b5ef59d46e615d/log/controller/logs/screen-n-cpu.txt#19235 | |
| 10:04:26 | gibi | Instance instance-0000002e disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid | |
| 10:04:44 | gibi | https://zuul.openstack.org/build/4a7f284f32eb436da6b5ef59d46e615d/log/controller/logs/libvirt/libvirt/qemu/instance-0000002e_log.txt | |
| 10:04:50 | gibi | 2022-06-01 03:13:33.853+0000: shutting down, reason=crashed | |
| 10:05:02 | gibi | but I don't see any reaonse why it is crashed | |
| 10:05:09 | sean-k-mooney | its paused but i have nto gotten to where it crahsed yet | |
| 10:06:03 | sean-k-mooney | Instance instance-0000002e disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid '84e8668d-6263-4a85-98f5-15d1a7e38f0d' (instance-0000002e) | |
| 10:07:01 | gibi | it seems that the syslog journal was rotated as it only has later logs in it | |
| 10:16:18 | gibi | slaweq: as far as I see this is job failing since 2022-05-07 with the same issue | |
| 10:17:45 | gibi | but older runs has no logs any more | |
| 10:18:01 | gibi | so potentially it is failing a lot longer than 05-07 | |
| 10:26:09 | gibi | slaweq: I found the rotated yournal | |
| 10:26:13 | gibi | qemu is segfaulted | |
| 10:26:14 | gibi | https://paste.opendev.org/show/bIxQstqasZeJiuyDRHxD/ | |
| 10:27:23 | gibi | better paste https://paste.opendev.org/show/byrrh32nwTYvGXtw52ax/ | |
| 10:29:17 | sean-k-mooney | nice find | |
| 10:29:25 | sean-k-mooney | so likely a fedora bug | |
| 10:29:34 | gibi | probably worth taking up with the downstream virt team | |
| 10:29:43 | gibi | as my knowledge ends here | |
| 10:30:17 | sean-k-mooney | ya perhaps although they dont really care about the tcg backend for qemu | |
| 10:30:25 | sean-k-mooney | so if this does not happen with kvm | |
| 10:30:32 | sean-k-mooney | they may or may not help | |
| 10:32:20 | sean-k-mooney | hum there is nothing useful in the qemu instance log | |
| 10:33:08 | sean-k-mooney | it looks like a pretty normal vm | |
| 10:33:12 | sean-k-mooney | -machine pc-i440fx-6.1,accel=tcg,usb=off,dump-guest-core=off,memory-backend=pc.ram | |
| 10:33:24 | sean-k-mooney | so still pc machine type too | |
| 10:50:09 | opendevreview | Merged openstack/nova master: Fix typos in help messages https://review.opendev.org/c/openstack/nova/+/843843 | |
| 10:50:17 | opendevreview | Merged openstack/nova master: Fix typos https://review.opendev.org/c/openstack/nova/+/843127 | |
| 10:50:27 | opendevreview | Merged openstack/nova stable/train: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840777 | |
| 11:28:15 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Drop py36 and py37 support https://review.opendev.org/c/openstack/osc-placement/+/844281 | |
| 11:29:09 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545 | |
| 11:36:40 | slaweq | thx sean-k-mooney (@sean-k-mooney:matrix.org) and gibi for checking tha | |
| 11:36:42 | slaweq | *that | |
| 11:42:17 | sean-k-mooney | slaweq: assuming this is just the qemu bug then you shoudl be able to swap the nodeset to c9s and that hopefully will resolve the problem unless that but has already made it into c9s | |
| 11:46:38 | gibi | elodilles, melwitt: I see a new blocker on stable/ussuri https://zuul.opendev.org/t/openstack/build/034890df09f94bcba63b97f1586e9a48/log/job-output.txt#43311 | |
| 11:46:57 | gibi | added to the tracking etherpad https://etherpad.opendev.org/p/nova-stable-branch-ci L26 | |
| 11:47:28 | gibi | gmann: ^^ fyi maybe you have a view on this too | |
| 11:49:45 | elodilles | gibi: there's already a patch for that: https://review.opendev.org/c/openstack/openstacksdk/+/843978 | |
| 11:51:35 | gibi | thanks, then now that is a blocker for ussuri as the job fails 100% | |
| 11:51:54 | gibi | due to master upper-constraint was bumped recently | |
| 11:52:38 | elodilles | yepp | |
| 11:54:16 | gibi | elodilles: thanks for the info | |
| 12:01:02 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: func: Increase rpc_response_timeout in TestMultiCellMigrate tests https://review.opendev.org/c/openstack/nova/+/844200 | |
| 12:03:40 | gibi | elodilles, melwitt: I saw this issue ^^ in wallaby (newer branches are already fixed) | |
| 12:09:05 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Add Python3 zed unit tests https://review.opendev.org/c/openstack/osc-placement/+/835369 | |
| 12:09:06 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545 | |
| 12:15:59 | opendevreview | Alexey Stupnikov proposed openstack/nova master: [minor]Remove unused argument from _fake_do_delete https://review.opendev.org/c/openstack/nova/+/844285 | |
| 12:17:38 | gibi | elodilles, melwitt: another blocker on stable/train grenade is failing in devstack-plugin-ceph https://zuul.opendev.org/t/openstack/build/a6e02e6d4b5d40f794c188c852214002/log/job-output.txt#5771-5780 https://zuul.opendev.org/t/openstack/build/a6e02e6d4b5d40f794c188c852214002/log/job-output.txt#5771-5780 | |
| 12:17:50 | gibi | it seems it started two days ago https://paste.opendev.org/show/bxCRqAFCVzM28hDqlWix/ | |
| 12:25:29 | elodilles | gibi: wallaby patch is +2'd. the train blocker is interesting :S i was focused on ussuri and victoria and haven't noticed we have a blocker in train :S | |
| 12:28:46 | opendevreview | Rico Lin proposed openstack/nova master: libvirt: Add vIOMMU device to guest https://review.opendev.org/c/openstack/nova/+/830646 | |
| 12:32:53 | gibi | elodilles: seems to be a fresh one | |
| 12:33:31 | gibi | elodilles: thanks for the review on the wallaby one | |
| 12:40:05 | elodilles | gibi: about the train failure: it is strange, it seems 'source' command is missing (?) from the node :-o | |
| 12:45:52 | gibi | it could be that "" is not found | |
| 12:45:54 | gibi | by source | |
| 12:46:00 | gibi | as the next line is | |
| 12:46:04 | gibi | /bin/sh: 6: install_ceph_remote: not found | |
| 12:46:23 | gibi | or meh | |
| 12:46:37 | elodilles | i guess that should come from the sourced code | |
| 12:47:39 | gibi | https://github.com/openstack/devstack-plugin-ceph/commit/67da31fa430b99bbe13dc53bbcf81d0e7688523f | |
| 12:47:55 | gibi | this did some change 6 days ago on ussuri and victoria | |
| 12:47:59 | gibi | so this might be related | |
| 12:50:46 | gibi | other than that I don't see any new thing in the plugin repo | |
| 13:04:02 | gibi | elodilles: btw, you were right in https://review.opendev.org/c/openstack/nova/+/839354/2#message-8f0247bc92c2d82ed37c05c88db0769fa1f69fa0 I fixed up the cherry pick | |
| 13:04:15 | gibi | you have eagle eyes | |
| 13:22:40 | elodilles | gibi: diffing the diffs helps a lot :D | |
| 13:23:23 | elodilles | gibi: it was strange that in one we have CPUPinning while in the other we have CPUUnpinning :) | |
| 13:23:59 | elodilles | i'll review that again when i get there :) | |
| 13:25:15 | elodilles | btw, i think the problem is with the train thing is that we don't have bash (thus 'source' command). somehow now it runs /bin/sh instead of /bin/bash. (at least that is what i suspect) | |
| 13:26:20 | sean-k-mooney | elodilles: odd thing i noticed on pop os 22.04 but might be the same on ubuntu 22.04 /bin/sh is actully dash... | |
| 13:26:35 | sean-k-mooney | you shoud never depend on /bin/sh for portabl scripts | |
| 13:27:15 | gibi | elodilles: interesting... | |
| 13:27:42 | sean-k-mooney | . <thing> | |
| 13:27:53 | sean-k-mooney | is more porable then "source thing" | |
| 13:28:42 | sean-k-mooney | but ya i hit the /bin/sh -> /bin/dash issue because pushd was not a command | |
| 13:29:04 | sean-k-mooney | in the script that uses /bin/sh | |
| 13:29:58 | sean-k-mooney | wehre is /bin/sh being used | |
| 13:30:00 | elodilles | sean-k-mooney: what i see is we expicitly ask for /bin/bash: https://opendev.org/openstack/nova/src/branch/stable/train/gate/live_migration/hooks/ceph.sh#L10 | |
| 13:30:02 | sean-k-mooney | is it in the devstack plugin | |
| 13:30:31 | sean-k-mooney | ah yep we do | |
| 13:30:37 | elodilles | sean-k-mooney: but then we got errors like '/bin/sh: .: install_ceph_remote: not found' | |
| 13:31:11 | elodilles | sean-k-mooney: or '/bin/sh: 5: source: not found' | |
| 13:31:27 | sean-k-mooney | why are we using ansible with raw there | |
| 13:31:30 | sean-k-mooney | and not shell | |
| 13:31:55 | elodilles | so it seems the subnode does not have 'bash' | |