| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-21 | |||
| 19:10:50 | mriedem | i'm much more interested in making sure the features we've already landed actually work | |
| 19:10:57 | kashyap | mriedem: I'm all with you | |
| 19:10:59 | mriedem | like when live migration fails and we tell you it was ok | |
| 19:11:27 | kashyap | "Stability" over "yeah, let's bake that half-assed feature in, and let it rot after initial enthusiasm wanes" | |
| 19:11:35 | superdan | mriedem: lol, I was trying to help with your doc patches, sorry if that was not that helpful | |
| 19:12:00 | mriedem | superdan: you ruined EVERYTHING! | |
| 19:12:07 | mriedem | superdan: you just have to realize the sfinucan-isms | |
| 19:12:10 | mriedem | with .rst files | |
| 19:12:30 | superdan | mriedem: the one thing I know about sfinucanisms is that he gives a shit about the formatting and I do not | |
| 19:12:33 | superdan | I read it in text format | |
| 19:12:40 | mriedem | i also care about format | |
| 19:12:45 | mriedem | it's why sfinucan and i butt heads | |
| 19:12:54 | superdan | well, sorry, you gonna change it? | |
| 19:13:03 | mriedem | no, i don't really care | |
| 19:13:12 | superdan | ack | |
| 19:13:13 | mriedem | i'm fixing this live migration logging turd | |
| 19:13:23 | superdan | never again shall I do that | |
| 19:13:25 | mriedem | whilst live migration job is broken in ocata and blocking all changes | |
| 19:13:46 | mriedem | and kashyap wants nova injecting qemu command line stuff in production, it's insane :P | |
| 19:13:53 | kashyap | mriedem: No, no! | |
| 19:13:59 | kashyap | You're too eager to misunderstand, aren't you? | |
| 19:14:05 | mriedem | kashyap: i know all too well my friend | |
| 19:14:07 | kashyap | :P | |
| 19:14:32 | kashyap | mriedem: What is the failure? Actually Dave Gilbert from QEMU is also up and awake at this our | |
| 19:14:40 | kashyap | I see the failure you link above is due to a bogus error message | |
| 19:14:46 | kashyap | Is there anything more other than that? | |
| 19:15:08 | kashyap | (s/our/hour/) | |
| 19:15:29 | kashyap | Jeez - "logging turd" | |
| 19:16:33 | openstackgerrit | Samantha Blanco proposed openstack/nova master: Make flavor-rxtx policy more granular flavor-rxtx currently only has one policy it checks against, which is not restrictive enough when used in conjunction with other policies (e.g., flavor-manage). This allows users a greater degree of customization. https://review.openstack.org/444036 | |
| 19:16:38 | kashyap | mriedem: It can be fast-approved, I think? | |
| 19:17:00 | openstackgerrit | Samantha Blanco proposed openstack/nova master: Make flavor-rxtx policy more granular flavor-rxtx currently only has one policy it checks against, which is not restrictive enough when used in conjunction with other policies (e.g., flavor-manage). This allows users a greater degree of customization. https://review.openstack.org/444036 | |
| 19:21:38 | mriedem | kashyap: i don't actually need someone from qemu | |
| 19:21:54 | mriedem | i mean, i don't know why the guest wasn't found that caused this actual failure | |
| 19:21:58 | mriedem | but that's not what i'm fixing atm | |
| 19:22:20 | kashyap | mriedem: Thought, you were also looking at another bug which involves QEMU itself | |
| 19:22:56 | mriedem | well, this live migration job failed | |
| 19:23:15 | mriedem | because during post live migration, when source calls dest to complete the migration on that side, the guest domain wasn't found | |
| 19:23:29 | kashyap | mriedem: Got a link to a log, if you don't mind? | |
| 19:23:30 | mriedem | even though we were in post live migration | |
| 19:23:43 | kashyap | I think I should find one from here: logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/ | |
| 19:23:52 | mriedem | yes | |
| 19:25:07 | kashyap | The root dir (gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/) has destination logs | |
| 19:25:35 | mriedem | http://logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/screen-n-cpu.txt.gz#_2017-04-21_13_54_10_150 | |
| 19:25:47 | mriedem | yes ^ is the dest logs | |
| 19:28:09 | mriedem | http://logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/libvirt/qemu/instance-00000001.txt.gz | |
| 19:28:18 | mriedem | /build/qemu-5OJ39u/qemu-2.8+dfsg/nbd/server.c:nbd_receive_request():L710: read failed 2017-04-21 13:54:08.792+0000: shutting down, reason=failed | |
| 19:29:35 | kashyap | For future note: Incidentally, we're going to get finer-grained details about shutdown reasons | |
| 19:29:44 | kashyap | Whether it is guest-triggered, or host-triggerred | |
| 19:29:48 | kashyap | And what precisely caused it, etc | |
| 19:30:14 | kashyap | (There's a long thread on qemu-devel upstream, to iron out the details.) | |
| 19:31:07 | mriedem | well, /build/qemu-5OJ39u/qemu-2.8+dfsg/nbd/server.c:nbd_receive_request():L710: read failed | |
| 19:31:08 | mriedem | caused it | |
| 19:31:09 | mriedem | i guess | |
| 19:31:24 | kashyap | Yes, I recall seeing that | |
| 19:32:51 | mriedem | /build/qemu-5OJ39u/qemu-2.8+dfsg/nbd/server.c:nbd_receive_request():L710: read failed | |
| 19:32:53 | mriedem | oops | |
| 19:33:00 | mriedem | http://logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/libvirt/libvirtd.txt.gz#_2017-04-21_13_54_08_792 | |
| 19:33:00 | kashyap | mriedem: Yep, I recall it | |
| 19:33:13 | kashyap | I even reported it recently here: http://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg01086.html | |
| 19:33:40 | kashyap | Quoting myself: | |
| 19:33:42 | kashyap | on the destination, _without_ gracefully completing or cancelling the | |
| 19:33:42 | kashyap | Just writing for completeness' sake, if you try to stop the NBD server | |
| 19:33:43 | kashyap | `drive-mirror` job on the source, you see the below: | |
| 19:33:53 | kashyap | /home/stack/src/qemu/nbd/server.c:nbd_receive_request():L706: read failed | |
| 19:33:53 | kashyap | {"return": {}} | |
| 19:33:53 | kashyap | {"execute":"nbd-server-stop"} | |
| 19:33:53 | kashyap | [...] | |
| 19:33:56 | kashyap | [...] | |
| 19:35:05 | mriedem | kashyap: so this is a known bug in qemu? | |
| 19:35:22 | kashyap | mriedem: It's actually benign | |
| 19:35:24 | kashyap | Because: | |
| 19:35:54 | kashyap | It means the source has gone away unexpectedly or that you stopped the NBD server before the source was disconnected | |
| 19:36:02 | kashyap | I know Nova shouldn't care this level of detail. | |
| 19:36:26 | kashyap | I'm checking if my discussion resulted in another patch | |
| 19:37:27 | mriedem | around that same time in syslog http://logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/syslog.txt.gz#_Apr_21_13_54_08 | |
| 19:37:59 | kashyap | mriedem: What were you trying to quote in the previous libvirtd log? Your link seems broken here | |
| 19:38:20 | mriedem | that's where it failed | |
| 19:38:26 | mriedem | 2017-04-21 13:54:08.792+0000: 12165: debug : qemuProcessStop:5941 : Shutting down vm=0x7f918c009d40 name=instance-00000001 id=2 pid=5792, reason=failed, asyncJob=migration in, flags=1 | |
| 19:39:48 | mriedem | Apr 21 13:54:08ubuntu-xenial-2-node-osic-cloud1-s3500-8527282 virtlogd[12188]: End of file while reading data: Input/output error | |
| 19:39:48 | mriedem | then around that time in syslog | |
| 19:40:19 | kashyap | But _what_ caused the guest shutdown... | |
| 19:40:25 | mriedem | no idea | |
| 19:42:34 | kashyap | mriedem: Checking with Eric Blake & Max, both should know this code in libvirt / QEMU respectively | |
| 19:52:08 | cdent | thank you mriedem, someone had to respond to leakypipes | |
| 19:53:41 | kashyap | mriedem: Is this mig. failure blocking Gate? | |
| 19:55:22 | kashyap | At least, I found out (w/ help from Max) the commit that introduced it. | |
| 19:56:45 | mriedem | cdent: ha | |
| 19:56:53 | mriedem | kashyap: i think this is probably random failure | |
| 19:57:10 | kashyap | mriedem: Ah, okay, good. | |
| 19:57:21 | kashyap | http://logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/subnode-2/libvirt/qemu/instance-00000001.txt.gz | |
| 19:57:21 | kashyap | The actual culprit failure is on the source host: | |
| 19:57:25 | mriedem | stable/ocata is blocked b/c of the live migration job, and to fix that i need to get this other change in, but i think this is just an intermittent failure | |
| 19:57:42 | mriedem | oh fun | |
| 19:57:43 | mriedem | qemu-system-x86_64: /build/qemu-5OJ39u/qemu-2.8+dfsg/block/io.c:1514: bdrv_co_pwritev: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed. | |
| 19:57:47 | kashyap | Yep | |
| 19:57:50 | mriedem | i have no idea what that means | |
| 20:02:44 | kashyap | mriedem: It means (according to the QEMU commit that introduced it): | |
| 20:03:08 | kashyap | Something tried to write to the image file on the source, _while_ it is being migrated | |
| 20:03:40 | kashyap | Anyway, it's unreasonably late here, I'll get to this on Monday | |
| 20:04:11 | kashyap | Cool; Gerrit is also offline on time. Everything is in harmony, and the earth keeps spinning | |