| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-01 | |||
| 11:54:16 | gibi | elodilles: thanks for the info | |
| 12:01:02 | opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: func: Increase rpc_response_timeout in TestMultiCellMigrate tests https://review.opendev.org/c/openstack/nova/+/844200 | |
| 12:03:40 | gibi | elodilles, melwitt: I saw this issue ^^ in wallaby (newer branches are already fixed) | |
| 12:09:05 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Add Python3 zed unit tests https://review.opendev.org/c/openstack/osc-placement/+/835369 | |
| 12:09:06 | opendevreview | Balazs Gibizer proposed openstack/osc-placement master: Support microversion 1.39 https://review.opendev.org/c/openstack/osc-placement/+/828545 | |
| 12:15:59 | opendevreview | Alexey Stupnikov proposed openstack/nova master: [minor]Remove unused argument from _fake_do_delete https://review.opendev.org/c/openstack/nova/+/844285 | |
| 12:17:38 | gibi | elodilles, melwitt: another blocker on stable/train grenade is failing in devstack-plugin-ceph https://zuul.opendev.org/t/openstack/build/a6e02e6d4b5d40f794c188c852214002/log/job-output.txt#5771-5780 https://zuul.opendev.org/t/openstack/build/a6e02e6d4b5d40f794c188c852214002/log/job-output.txt#5771-5780 | |
| 12:17:50 | gibi | it seems it started two days ago https://paste.opendev.org/show/bxCRqAFCVzM28hDqlWix/ | |
| 12:25:29 | elodilles | gibi: wallaby patch is +2'd. the train blocker is interesting :S i was focused on ussuri and victoria and haven't noticed we have a blocker in train :S | |
| 12:28:46 | opendevreview | Rico Lin proposed openstack/nova master: libvirt: Add vIOMMU device to guest https://review.opendev.org/c/openstack/nova/+/830646 | |
| 12:32:53 | gibi | elodilles: seems to be a fresh one | |
| 12:33:31 | gibi | elodilles: thanks for the review on the wallaby one | |
| 12:40:05 | elodilles | gibi: about the train failure: it is strange, it seems 'source' command is missing (?) from the node :-o | |
| 12:45:52 | gibi | it could be that "" is not found | |
| 12:45:54 | gibi | by source | |
| 12:46:00 | gibi | as the next line is | |
| 12:46:04 | gibi | /bin/sh: 6: install_ceph_remote: not found | |
| 12:46:23 | gibi | or meh | |
| 12:46:37 | elodilles | i guess that should come from the sourced code | |
| 12:47:39 | gibi | https://github.com/openstack/devstack-plugin-ceph/commit/67da31fa430b99bbe13dc53bbcf81d0e7688523f | |
| 12:47:55 | gibi | this did some change 6 days ago on ussuri and victoria | |
| 12:47:59 | gibi | so this might be related | |
| 12:50:46 | gibi | other than that I don't see any new thing in the plugin repo | |
| 13:04:02 | gibi | elodilles: btw, you were right in https://review.opendev.org/c/openstack/nova/+/839354/2#message-8f0247bc92c2d82ed37c05c88db0769fa1f69fa0 I fixed up the cherry pick | |
| 13:04:15 | gibi | you have eagle eyes | |
| 13:22:40 | elodilles | gibi: diffing the diffs helps a lot :D | |
| 13:23:23 | elodilles | gibi: it was strange that in one we have CPUPinning while in the other we have CPUUnpinning :) | |
| 13:23:59 | elodilles | i'll review that again when i get there :) | |
| 13:25:15 | elodilles | btw, i think the problem is with the train thing is that we don't have bash (thus 'source' command). somehow now it runs /bin/sh instead of /bin/bash. (at least that is what i suspect) | |
| 13:26:20 | sean-k-mooney | elodilles: odd thing i noticed on pop os 22.04 but might be the same on ubuntu 22.04 /bin/sh is actully dash... | |
| 13:26:35 | sean-k-mooney | you shoud never depend on /bin/sh for portabl scripts | |
| 13:27:15 | gibi | elodilles: interesting... | |
| 13:27:42 | sean-k-mooney | . <thing> | |
| 13:27:53 | sean-k-mooney | is more porable then "source thing" | |
| 13:28:42 | sean-k-mooney | but ya i hit the /bin/sh -> /bin/dash issue because pushd was not a command | |
| 13:29:04 | sean-k-mooney | in the script that uses /bin/sh | |
| 13:29:58 | sean-k-mooney | wehre is /bin/sh being used | |
| 13:30:00 | elodilles | sean-k-mooney: what i see is we expicitly ask for /bin/bash: https://opendev.org/openstack/nova/src/branch/stable/train/gate/live_migration/hooks/ceph.sh#L10 | |
| 13:30:02 | sean-k-mooney | is it in the devstack plugin | |
| 13:30:31 | sean-k-mooney | ah yep we do | |
| 13:30:37 | elodilles | sean-k-mooney: but then we got errors like '/bin/sh: .: install_ceph_remote: not found' | |
| 13:31:11 | elodilles | sean-k-mooney: or '/bin/sh: 5: source: not found' | |
| 13:31:27 | sean-k-mooney | why are we using ansible with raw there | |
| 13:31:30 | sean-k-mooney | and not shell | |
| 13:31:55 | elodilles | so it seems the subnode does not have 'bash' | |
| 13:32:19 | elodilles | sean-k-mooney: good question :) | |
| 13:32:28 | sean-k-mooney | it should im not sure that executable=/bin/bash is the correc t syntx | |
| 13:32:50 | sean-k-mooney | that is defining a varable in the "script" | |
| 13:33:11 | sean-k-mooney | its not actully part of the argument to the ansibel module | |
| 13:33:51 | sean-k-mooney | as in that is not telling the raw module to use bash as far as i can see | |
| 13:34:17 | sean-k-mooney | its just storing /bin/bash in the executable variable in the shell that is creted by raw | |
| 13:35:02 | sean-k-mooney | executable is a parmater it can accpet | |
| 13:35:04 | sean-k-mooney | https://docs.ansible.com/ansible/latest/collections/ansible/builtin/raw_module.html | |
| 13:35:16 | sean-k-mooney | but that does not look liek the correct syntax to me | |
| #openstack-nova - 2022-06-02 | |||
| 07:50:00 | bauzas | hi nova | |
| 07:51:35 | gibi | o/ | |
| 08:56:13 | bauzas | wow, missed one day and world became crazy | |
| 08:56:30 | bauzas | was hoping to do reviews this morning, apparently I was wrong :/ | |
| 08:57:44 | gibi | which crazyness you observe? | |
| 09:08:56 | bauzas | gibi: just a lot of things arrived in my inbox that require a bit of priority :) | |
| 09:09:23 | bauzas | don't worry, I'm french, I'm used to complain | |
| 09:09:30 | gibi | :) | |
| 09:46:39 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reject AZ changes during aggregate add / remove host https://review.opendev.org/c/openstack/nova/+/821423 | |
| 11:11:18 | opendevreview | Rajesh Tailor proposed openstack/nova master: Remove unnecessary if condition https://review.opendev.org/c/openstack/nova/+/844418 | |
| 11:48:01 | opendevreview | Rico Lin proposed openstack/nova master: libvirt: Add vIOMMU device to guest https://review.opendev.org/c/openstack/nova/+/830646 | |
| 13:41:53 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Optimize _local_delete calls by compute unit tests https://review.opendev.org/c/openstack/nova/+/844285 | |
| 14:24:32 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Optimize _local_delete calls by compute unit tests https://review.opendev.org/c/openstack/nova/+/844285 | |
| 14:55:24 | opendevreview | Balazs Gibizer proposed openstack/nova master: Unparent PciDeviceSpec from PciAddressSpec https://review.opendev.org/c/openstack/nova/+/844491 | |
| 15:20:54 | dansmith | kashyap: slaweq is seeing a qemu segv on their fedora periodic job.. could you help us examine and open a bug for the qemu type folks to look at? | |
| 15:21:14 | kashyap | dansmith: Hiya; sure. | |
| 15:21:44 | kashyap | dansmith: Got a link for it? I'm on a call right now, but can look at the errors (I wonder which version of Fedora) | |
| 15:21:44 | dansmith | kashyap: thanks, we're still in meeting, but I imagine slaweq will be around here with a job link shortly | |
| 15:21:50 | kashyap | Sure | |
| 15:22:14 | slaweq | kashyap dansmith here's failed job https://zuul.openstack.org/build/4a7f284f32eb436da6b5ef59d46e615d/logs | |
| 15:22:34 | slaweq | I know that @gibi was looking briefly into it yesterday | |
| 15:23:08 | slaweq | but interesting thing is that today this job passed https://zuul.openstack.org/build/4c1f894e55f84447b8b0b0f14c774c89/logs | |
| 15:23:17 | kashyap | So it's a bit intermittent | |
| 15:23:40 | slaweq | kashyap it was failing every day in at least last week, except today | |
| 15:23:55 | slaweq | this is periodic job so we can check how it will be tomorrow | |
| 15:23:59 | kashyap | slaweq: Can you link me to the exact error, pls? | |
| 15:24:37 | slaweq | kashyap https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4a7/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/4a7f284/controller/logs/libvirt/libvirt/qemu/instance-0000002e_log.txt | |
| 15:25:49 | kashyap | slaweq: Interesting ... I wonder if there's a `coredumpctl list` output, then we can get the crashdumps right away | |
| 15:26:13 | slaweq | I don't think there is anything like that in the job's logs | |
| 15:26:22 | slaweq | all logs are here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4a7/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/4a7f284/controller/logs/index.html | |
| 15:27:42 | kashyap | slaweq: Thanks; looking while on a call | |
| 15:27:47 | dansmith | kashyap: if it's just something we run post-crash we can add that as a one-off | |
| 15:27:56 | slaweq | kashyap sure, I'm on call now too | |
| 15:28:40 | kashyap | dansmith: What would be good is to capture both: `coredumpctl list | grep qemu`, and then for each QEMU PID log `coredumpctl info $PID` (I know ... I'm asking too much) | |
| 15:29:09 | kashyap | dansmith: E.g. see at the bottom here for the example output of `coredumpctl info $PID` - https://www.freedesktop.org/software/systemd/man/coredumpctl.html | |
| 15:29:59 | kashyap | The reason I ask is, I've successfully found several root-cause stack trace from it in the past. | |
| 15:31:02 | kashyap | dansmith: slaweq: Ah, scratch the above, we could even just get this post-crash: `coredumpctl -o qemu.coredump dump /usr/bin/qemu-system-x86_64` | |
| 15:32:17 | kashyap | slaweq: A quick question: the instance simply crashes when launching it? | |
| 15:32:51 | slaweq | kashyap I think it crashed during snapshoting | |
| 15:32:55 | slaweq | it was spawned properly | |
| 15:33:00 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Optimize _local_delete calls by compute unit tests https://review.opendev.org/c/openstack/nova/+/844285 | |
| 15:33:51 | dansmith | kashyap: ah, running it on a specific pid would be much harder | |
| 15:34:26 | dansmith | running it like you describe is something we could hack in as a post job | |
| 15:34:32 | kashyap | dansmith: Nah, we can disregad the per-PID thing | |
| 15:34:47 | kashyap | Yeah, binary is easier indeed | |
| 15:35:23 | dansmith | okay after call(s) I can help hack that in if we need, but if it's pretty repeatable it might be easier to just try to repro locally | |