Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-06-01
13:29:58 sean-k-mooney wehre is /bin/sh being used
13:30:00 elodilles sean-k-mooney: what i see is we expicitly ask for /bin/bash: https://opendev.org/openstack/nova/src/branch/stable/train/gate/live_migration/hooks/ceph.sh#L10
13:30:02 sean-k-mooney is it in the devstack plugin
13:30:31 sean-k-mooney ah yep we do
13:30:37 elodilles sean-k-mooney: but then we got errors like '/bin/sh: .: install_ceph_remote: not found'
13:31:11 elodilles sean-k-mooney: or '/bin/sh: 5: source: not found'
13:31:27 sean-k-mooney why are we using ansible with raw there
13:31:30 sean-k-mooney and not shell
13:31:55 elodilles so it seems the subnode does not have 'bash'
13:32:19 elodilles sean-k-mooney: good question :)
13:32:28 sean-k-mooney it should im not sure that executable=/bin/bash is the correc t syntx
13:32:50 sean-k-mooney that is defining a varable in the "script"
13:33:11 sean-k-mooney its not actully part of the argument to the ansibel module
13:33:51 sean-k-mooney as in that is not telling the raw module to use bash as far as i can see
13:34:17 sean-k-mooney its just storing /bin/bash in the executable variable in the shell that is creted by raw
13:35:02 sean-k-mooney executable is a parmater it can accpet
13:35:04 sean-k-mooney https://docs.ansible.com/ansible/latest/collections/ansible/builtin/raw_module.html
13:35:16 sean-k-mooney but that does not look liek the correct syntax to me
#openstack-nova - 2022-06-02
07:50:00 bauzas hi nova
07:51:35 gibi o/
08:56:13 bauzas wow, missed one day and world became crazy
08:56:30 bauzas was hoping to do reviews this morning, apparently I was wrong :/
08:57:44 gibi which crazyness you observe?
09:08:56 bauzas gibi: just a lot of things arrived in my inbox that require a bit of priority :)
09:09:23 bauzas don't worry, I'm french, I'm used to complain
09:09:30 gibi :)
09:46:39 opendevreview Balazs Gibizer proposed openstack/nova master: Reject AZ changes during aggregate add / remove host https://review.opendev.org/c/openstack/nova/+/821423
11:11:18 opendevreview Rajesh Tailor proposed openstack/nova master: Remove unnecessary if condition https://review.opendev.org/c/openstack/nova/+/844418
11:48:01 opendevreview Rico Lin proposed openstack/nova master: libvirt: Add vIOMMU device to guest https://review.opendev.org/c/openstack/nova/+/830646
13:41:53 opendevreview Alexey Stupnikov proposed openstack/nova master: Optimize _local_delete calls by compute unit tests https://review.opendev.org/c/openstack/nova/+/844285
14:24:32 opendevreview Alexey Stupnikov proposed openstack/nova master: Optimize _local_delete calls by compute unit tests https://review.opendev.org/c/openstack/nova/+/844285
14:55:24 opendevreview Balazs Gibizer proposed openstack/nova master: Unparent PciDeviceSpec from PciAddressSpec https://review.opendev.org/c/openstack/nova/+/844491
15:20:54 dansmith kashyap: slaweq is seeing a qemu segv on their fedora periodic job.. could you help us examine and open a bug for the qemu type folks to look at?
15:21:14 kashyap dansmith: Hiya; sure.
15:21:44 kashyap dansmith: Got a link for it? I'm on a call right now, but can look at the errors (I wonder which version of Fedora)
15:21:44 dansmith kashyap: thanks, we're still in meeting, but I imagine slaweq will be around here with a job link shortly
15:21:50 kashyap Sure
15:22:14 slaweq kashyap dansmith here's failed job https://zuul.openstack.org/build/4a7f284f32eb436da6b5ef59d46e615d/logs
15:22:34 slaweq I know that @gibi was looking briefly into it yesterday
15:23:08 slaweq but interesting thing is that today this job passed https://zuul.openstack.org/build/4c1f894e55f84447b8b0b0f14c774c89/logs
15:23:17 kashyap So it's a bit intermittent
15:23:40 slaweq kashyap it was failing every day in at least last week, except today
15:23:55 slaweq this is periodic job so we can check how it will be tomorrow
15:23:59 kashyap slaweq: Can you link me to the exact error, pls?
15:24:37 slaweq kashyap https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4a7/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/4a7f284/controller/logs/libvirt/libvirt/qemu/instance-0000002e_log.txt
15:25:49 kashyap slaweq: Interesting ... I wonder if there's a `coredumpctl list` output, then we can get the crashdumps right away
15:26:13 slaweq I don't think there is anything like that in the job's logs
15:26:22 slaweq all logs are here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4a7/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/4a7f284/controller/logs/index.html
15:27:42 kashyap slaweq: Thanks; looking while on a call
15:27:47 dansmith kashyap: if it's just something we run post-crash we can add that as a one-off
15:27:56 slaweq kashyap sure, I'm on call now too
15:28:40 kashyap dansmith: What would be good is to capture both: `coredumpctl list | grep qemu`, and then for each QEMU PID log `coredumpctl info $PID` (I know ... I'm asking too much)
15:29:09 kashyap dansmith: E.g. see at the bottom here for the example output of `coredumpctl info $PID` - https://www.freedesktop.org/software/systemd/man/coredumpctl.html
15:29:59 kashyap The reason I ask is, I've successfully found several root-cause stack trace from it in the past.
15:31:02 kashyap dansmith: slaweq: Ah, scratch the above, we could even just get this post-crash: `coredumpctl -o qemu.coredump dump /usr/bin/qemu-system-x86_64`
15:32:17 kashyap slaweq: A quick question: the instance simply crashes when launching it?
15:32:51 slaweq kashyap I think it crashed during snapshoting
15:32:55 slaweq it was spawned properly
15:33:00 opendevreview Alexey Stupnikov proposed openstack/nova master: Optimize _local_delete calls by compute unit tests https://review.opendev.org/c/openstack/nova/+/844285
15:33:51 dansmith kashyap: ah, running it on a specific pid would be much harder
15:34:26 dansmith running it like you describe is something we could hack in as a post job
15:34:32 kashyap dansmith: Nah, we can disregad the per-PID thing
15:34:47 kashyap Yeah, binary is easier indeed
15:35:23 dansmith okay after call(s) I can help hack that in if we need, but if it's pretty repeatable it might be easier to just try to repro locally
15:36:01 kashyap dansmith: Yeah, that's the next thing I'm looking. It looks like it's not any Ceph-based, and just plain local storage, IIRC
15:36:08 dansmith cool
15:37:49 kashyap slaweq: I'm just trying to find the precise test trigger. From looking at the 'n-cpu' log, snapshots seem to happen just fine. I'll look more after I'm done w/ this call
15:38:27 slaweq kashyap but IIUC nova logs, instance is gone during snapshoting process
15:38:37 slaweq please take Your time, it's not urgent for us for sure
15:38:52 dansmith kashyap: it's test_create_backup
15:38:55 dansmith https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_4a7/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-ovs-master-fedora/4a7f284/testr_results.html
15:39:25 kashyap (Yeah, just found it; thx)
15:43:08 kashyap dansmith: Thanks; so the above command I noted above "coredumpctl dump" will dump the most recent core dump. I guess we have to redirect it to a file
15:43:11 opendevreview Merged openstack/osc-placement master: Add Python3 zed unit tests https://review.opendev.org/c/openstack/osc-placement/+/835369
15:43:39 kashyap dansmith: Err, ignore the above comment; the "-o" is the file.
15:43:58 dansmith kashyap: ack, are you going to try to repro locally first?
15:44:33 kashyap dansmith: On F35, just run the Tempest test, or construct a manual libvirt-based repro?
15:44:37 dansmith if it doesn't repro locally that might be interesting to know as well, like whether it's related to having run a lot of tests first, or that one thing always fails in isolation
15:44:55 dansmith kashyap: I just meant devstack, run that one tempest test
15:45:33 kashyap Ah; nod. I can't today; but I can give it a go tomorrow.
15:45:37 dansmith when I'm done here I can work on adding that as a post task, it just might take a bunch of iterations to get it right (based on experience)
15:45:45 dansmith okay, I'll give it a shot at least when I'm done here
15:47:00 kashyap dansmith: When you say "adding that as a post task" -- I take it you mean adding the above "coredumpctl ... dump", yeah?
15:47:08 dansmith yep
15:49:56 kashyap I have a deja vu about this test_create_backup test, reading its code
15:56:14 kashyap slaweq: When you get a minute, can you please file an upstream LP bug to track this? So we can keep all the investigation in one place?
16:01:12 dansmith kashyap: against what, nova?
16:01:26 kashyap Yeah, I'd say so
16:01:33 kashyap dansmith: I just looked at the compressed libvirtd.log
16:01:46 kashyap 2022-06-01 03:35:33.685+0000: 87576: error : qemuMonitorJSONCheckErrorFull:412 : internal error: unable to execute QEMU command 'blockdev-del': Failed to find node with node-name='libvirt-5-storage'
16:01:46 kashyap And I see a familiar libvirt error:
16:02:30 kashyap dansmith: In the past we found the same error earlier this year, and I recall working w/ libvirt folks to get a fix. But that was in a different context: https://listman.redhat.com/archives/libvir-list/2022-February/msg00790.html
16:03:11 kashyap dansmith: slaweq: The root TripleO (actually should've filed for Nova) bug was this where did the analysis: https://bugs.launchpad.net/tripleo/+bug/1959014
16:03:26 kashyap If you open that last link, scroll from bottom for more signal
16:54:24 dansmith kashyap: yeah I remember that one.. so to be clear, you expect this is a different issue right?
17:09:24 dansmith kashyap: this is running, we'll see: https://review.opendev.org/c/openstack/devstack/+/844503
17:10:56 ricolin bauzas: I think https://review.opendev.org/c/openstack/nova/+/830646 is ready for review now, could you kindly remove the -2
17:11:54 bauzas ricolin: sure, lemme look
17:12:13 ricolin bauzas: thanks:)
17:12:33 bauzas ricolin: oh, yeah you created the bp and the spec, ta

Earlier   Later