| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-01 | |||
| 10:04:26 | gibi | Instance instance-0000002e disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid | |
| 10:04:44 | gibi | https://zuul.openstack.org/build/4a7f284f32eb436da6b5ef59d46e615d/log/controller/logs/libvirt/libvirt/qemu/instance-0000002e_log.txt | |
| 10:04:50 | gibi | 2022-06-01 03:13:33.853+0000: shutting down, reason=crashed | |
| 10:05:02 | gibi | but I don't see any reaonse why it is crashed | |
| 10:05:09 | sean-k-mooney | its paused but i have nto gotten to where it crahsed yet | |
| 10:06:03 | sean-k-mooney | Instance instance-0000002e disappeared while taking snapshot of it: [Error Code 42] Domain not found: no domain with matching uuid '84e8668d-6263-4a85-98f5-15d1a7e38f0d' (instance-0000002e) | |
| 10:07:01 | gibi | it seems that the syslog journal was rotated as it only has later logs in it | |
| 10:16:18 | gibi | slaweq: as far as I see this is job failing since 2022-05-07 with the same issue | |
| 10:17:45 | gibi | but older runs has no logs any more | |
| 10:18:01 | gibi | so potentially it is failing a lot longer than 05-07 | |
| 10:26:09 | gibi | slaweq: I found the rotated yournal | |
| 10:26:13 | gibi | qemu is segfaulted | |
| 10:26:14 | gibi | https://paste.opendev.org/show/bIxQstqasZeJiuyDRHxD/ | |
| 10:27:23 | gibi | better paste https://paste.opendev.org/show/byrrh32nwTYvGXtw52ax/ | |
| 10:29:17 | sean-k-mooney | nice find | |
| 10:29:25 | sean-k-mooney | so likely a fedora bug | |
| 10:29:34 | gibi | probably worth taking up with the downstream virt team | |
| 10:29:43 | gibi | as my knowledge ends here | |
| 10:30:17 | sean-k-mooney | ya perhaps although they dont really care about the tcg backend for qemu | |
| 10:30:25 | sean-k-mooney | so if this does not happen with kvm | |
| 10:30:32 | sean-k-mooney | they may or may not help | |
| 10:32:20 | sean-k-mooney | hum there is nothing useful in the qemu instance log | |
| 10:33:08 | sean-k-mooney | it looks like a pretty normal vm | |
| 10:33:12 | sean-k-mooney | -machine pc-i440fx-6.1,accel=tcg,usb=off,dump-guest-core=off,memory-backend=pc.ram | |
| 10:33:24 | sean-k-mooney | so still pc machine type too | |
| 10:50:09 | opendevreview | Merged openstack/nova master: Fix typos in help messages https://review.opendev.org/c/openstack/nova/+/843843 | |
| 10:50:17 | opendevreview | Merged openstack/nova master: Fix typos https://review.opendev.org/c/openstack/nova/+/843127 | |
| 10:50:27 | opendevreview | Merged openstack/nova stable/train: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840777 | |
| 11:28:15 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Drop py36 and py37 support https://review.opendev.org/c/openstack/osc-placement/+/844281 | |
| 11:29:09 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545 | |
| 11:36:40 | slaweq | thx sean-k-mooney (@sean-k-mooney:matrix.org) and gibi for checking tha | |
| 11:36:42 | slaweq | *that | |
| 11:42:17 | sean-k-mooney | slaweq: assuming this is just the qemu bug then you shoudl be able to swap the nodeset to c9s and that hopefully will resolve the problem unless that but has already made it into c9s | |
| 11:46:38 | gibi | elodilles, melwitt: I see a new blocker on stable/ussuri https://zuul.opendev.org/t/openstack/build/034890df09f94bcba63b97f1586e9a48/log/job-output.txt#43311 | |
| 11:46:57 | gibi | added to the tracking etherpad https://etherpad.opendev.org/p/nova-stable-branch-ci L26 | |
| 11:47:28 | gibi | gmann: ^^ fyi maybe you have a view on this too | |
| 11:49:45 | elodilles | gibi: there's already a patch for that: https://review.opendev.org/c/openstack/openstacksdk/+/843978 | |
| 11:51:35 | gibi | thanks, then now that is a blocker for ussuri as the job fails 100% | |
| 11:51:54 | gibi | due to master upper-constraint was bumped recently | |
| 11:52:38 | elodilles | yepp | |
| 11:54:16 | gibi | elodilles: thanks for the info | |
| 12:01:02 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: func: Increase rpc_response_timeout in TestMultiCellMigrate tests https://review.opendev.org/c/openstack/nova/+/844200 | |
| 12:03:40 | gibi | elodilles, melwitt: I saw this issue ^^ in wallaby (newer branches are already fixed) | |
| 12:09:05 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Add Python3 zed unit tests https://review.opendev.org/c/openstack/osc-placement/+/835369 | |
| 12:09:06 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545 | |
| 12:15:59 | opendevreview | Alexey Stupnikov proposed openstack/nova master: [minor]Remove unused argument from _fake_do_delete https://review.opendev.org/c/openstack/nova/+/844285 | |
| 12:17:38 | gibi | elodilles, melwitt: another blocker on stable/train grenade is failing in devstack-plugin-ceph https://zuul.opendev.org/t/openstack/build/a6e02e6d4b5d40f794c188c852214002/log/job-output.txt#5771-5780 https://zuul.opendev.org/t/openstack/build/a6e02e6d4b5d40f794c188c852214002/log/job-output.txt#5771-5780 | |
| 12:17:50 | gibi | it seems it started two days ago https://paste.opendev.org/show/bxCRqAFCVzM28hDqlWix/ | |
| 12:25:29 | elodilles | gibi: wallaby patch is +2'd. the train blocker is interesting :S i was focused on ussuri and victoria and haven't noticed we have a blocker in train :S | |
| 12:28:46 | opendevreview | Rico Lin proposed openstack/nova master: libvirt: Add vIOMMU device to guest https://review.opendev.org/c/openstack/nova/+/830646 | |
| 12:32:53 | gibi | elodilles: seems to be a fresh one | |
| 12:33:31 | gibi | elodilles: thanks for the review on the wallaby one | |
| 12:40:05 | elodilles | gibi: about the train failure: it is strange, it seems 'source' command is missing (?) from the node :-o | |
| 12:45:52 | gibi | it could be that "" is not found | |
| 12:45:54 | gibi | by source | |
| 12:46:00 | gibi | as the next line is | |
| 12:46:04 | gibi | /bin/sh: 6: install_ceph_remote: not found | |
| 12:46:23 | gibi | or meh | |
| 12:46:37 | elodilles | i guess that should come from the sourced code | |
| 12:47:39 | gibi | https://github.com/openstack/devstack-plugin-ceph/commit/67da31fa430b99bbe13dc53bbcf81d0e7688523f | |
| 12:47:55 | gibi | this did some change 6 days ago on ussuri and victoria | |
| 12:47:59 | gibi | so this might be related | |
| 12:50:46 | gibi | other than that I don't see any new thing in the plugin repo | |
| 13:04:02 | gibi | elodilles: btw, you were right in https://review.opendev.org/c/openstack/nova/+/839354/2#message-8f0247bc92c2d82ed37c05c88db0769fa1f69fa0 I fixed up the cherry pick | |
| 13:04:15 | gibi | you have eagle eyes | |
| 13:22:40 | elodilles | gibi: diffing the diffs helps a lot :D | |
| 13:23:23 | elodilles | gibi: it was strange that in one we have CPUPinning while in the other we have CPUUnpinning :) | |
| 13:23:59 | elodilles | i'll review that again when i get there :) | |
| 13:25:15 | elodilles | btw, i think the problem is with the train thing is that we don't have bash (thus 'source' command). somehow now it runs /bin/sh instead of /bin/bash. (at least that is what i suspect) | |
| 13:26:20 | sean-k-mooney | elodilles: odd thing i noticed on pop os 22.04 but might be the same on ubuntu 22.04 /bin/sh is actully dash... | |
| 13:26:35 | sean-k-mooney | you shoud never depend on /bin/sh for portabl scripts | |
| 13:27:15 | gibi | elodilles: interesting... | |
| 13:27:42 | sean-k-mooney | . <thing> | |
| 13:27:53 | sean-k-mooney | is more porable then "source thing" | |
| 13:28:42 | sean-k-mooney | but ya i hit the /bin/sh -> /bin/dash issue because pushd was not a command | |
| 13:29:04 | sean-k-mooney | in the script that uses /bin/sh | |
| 13:29:58 | sean-k-mooney | wehre is /bin/sh being used | |
| 13:30:00 | elodilles | sean-k-mooney: what i see is we expicitly ask for /bin/bash: https://opendev.org/openstack/nova/src/branch/stable/train/gate/live_migration/hooks/ceph.sh#L10 | |
| 13:30:02 | sean-k-mooney | is it in the devstack plugin | |
| 13:30:31 | sean-k-mooney | ah yep we do | |
| 13:30:37 | elodilles | sean-k-mooney: but then we got errors like '/bin/sh: .: install_ceph_remote: not found' | |
| 13:31:11 | elodilles | sean-k-mooney: or '/bin/sh: 5: source: not found' | |
| 13:31:27 | sean-k-mooney | why are we using ansible with raw there | |
| 13:31:30 | sean-k-mooney | and not shell | |
| 13:31:55 | elodilles | so it seems the subnode does not have 'bash' | |
| 13:32:19 | elodilles | sean-k-mooney: good question :) | |
| 13:32:28 | sean-k-mooney | it should im not sure that executable=/bin/bash is the correc t syntx | |
| 13:32:50 | sean-k-mooney | that is defining a varable in the "script" | |
| 13:33:11 | sean-k-mooney | its not actully part of the argument to the ansibel module | |
| 13:33:51 | sean-k-mooney | as in that is not telling the raw module to use bash as far as i can see | |
| 13:34:17 | sean-k-mooney | its just storing /bin/bash in the executable variable in the shell that is creted by raw | |
| 13:35:02 | sean-k-mooney | executable is a parmater it can accpet | |
| 13:35:04 | sean-k-mooney | https://docs.ansible.com/ansible/latest/collections/ansible/builtin/raw_module.html | |
| 13:35:16 | sean-k-mooney | but that does not look liek the correct syntax to me | |
| #openstack-nova - 2022-06-02 | |||
| 07:50:00 | bauzas | hi nova | |
| 07:51:35 | gibi | o/ | |
| 08:56:13 | bauzas | wow, missed one day and world became crazy | |
| 08:56:30 | bauzas | was hoping to do reviews this morning, apparently I was wrong :/ | |
| 08:57:44 | gibi | which crazyness you observe? | |
| 09:08:56 | bauzas | gibi: just a lot of things arrived in my inbox that require a bit of priority :) | |