| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-21 | |||
| 18:47:43 | mriedem | oh fun that info message is in the compute manager | |
| 18:47:48 | mriedem | where i might not be using libvirt at all | |
| 18:54:53 | mriedem | oh no wonder this goes back to diablo in a commit with no explanation | |
| 18:56:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove archaic reference to QEMU errors during post live migration https://review.openstack.org/458935 | |
| 18:57:39 | mriedem | also | |
| 18:57:40 | mriedem | Post live migration at destination ubuntu-xenial-2-node-osic-cloud1-s3500-8527282 failed | |
| 18:57:51 | mriedem | Migrating instance to ubuntu-xenial-2-node-osic-cloud1-s3500-8527282 finished successfully. | |
| 18:58:00 | mriedem | wtf | |
| 18:58:59 | kashyap | mriedem: Oops - that error message, specific to driver | |
| 18:59:11 | mriedem | kashyap: well, it's probably from when libvirt was the only driver | |
| 18:59:38 | kashyap | Hmm. Are other drivers, besides Xen, are well-maintained? | |
| 19:00:59 | mriedem | hypev | |
| 19:01:01 | mriedem | *hyperv | |
| 19:01:27 | kashyap | Yeah. | |
| 19:01:53 | kashyap | So, will Nova support Moby Containers? :P | |
| 19:03:42 | kashyap | Was debugging a "fun" issue where exposing a CPU "L3" cache (via QEMU's "-cpu host,l3-cache=on") to a guest will improve performance by 10% or so for some SAP workloads! | |
| 19:04:04 | mriedem | kashyap: you should talk to cfriesen then | |
| 19:04:11 | mriedem | he's been looking for someone that loves qemu and libvirt for the last 3 days | |
| 19:04:13 | kashyap | mriedem: Found out the root cause, actually | |
| 19:04:21 | kashyap | But luckily, no action item needed for upstream Nova, Because: | |
| 19:04:46 | kashyap | We "get it for free", with QEMU 2.8 and above. | |
| 19:04:52 | kashyap | As it is part of the _default_ machine type | |
| 19:05:19 | kashyap | So, if anyone is hitting it with Nova, using an older QEMU, still "no problem", we could use the always-handy (but voids support warranty) workaround: | |
| 19:05:46 | kashyap | Libvirt's XML namespace <qemu:commandline> [...] </qemu:commandline> | |
| 19:06:44 | kashyap | While upstream libvirt is trying to wire it into its QEMU command-line generator. So all good. And I can wind up for the night | |
| 19:06:50 | mriedem | is that like injecting stuff into rabbit at runtime using erlang? | |
| 19:07:00 | kashyap | Hehe | |
| 19:07:08 | kashyap | cfriesen: You might be interested in this -- https://www.redhat.com/archives/libvir-list/2017-April/msg00815.html | |
| 19:07:18 | kashyap | Unfortunately, the link won't work :-( | |
| 19:07:29 | kashyap | As there's a major outage in one of the critical datacenters | |
| 19:08:21 | kashyap | But rest assured, somewhere some tired admin is sweating to hell and back trying to fix the outage ... | |
| 19:09:17 | kashyap | mriedem: But more seriously, that namespace is very handy -- we (Nova) could test interesting features and can decide: "Okay, this and that combination of QEMU command-line gives Nova users some advantage -- so let's file an RFE to get that wired up into official libvirt APIs" | |
| 19:09:22 | mriedem | kashyap: maybe it's down b/c he was injecting stuff via qemu command line to the infra running the server | |
| 19:09:41 | kashyap | mriedem: Yeah, that workaround should be a one-off hack only. | |
| 19:10:01 | kashyap | Because, when using that workaroud, in logs, you'd see the instances to be marked as "tainted" (rightfully) | |
| 19:10:19 | kashyap | Warning the opertors that: "Look, your libvirt / QEMU upgrade might break your existing guests" | |
| 19:10:41 | kashyap | And of course, to help support people identify if someone is using unofficial command-line | |
| 19:10:50 | mriedem | i'm much more interested in making sure the features we've already landed actually work | |
| 19:10:57 | kashyap | mriedem: I'm all with you | |
| 19:10:59 | mriedem | like when live migration fails and we tell you it was ok | |
| 19:11:27 | kashyap | "Stability" over "yeah, let's bake that half-assed feature in, and let it rot after initial enthusiasm wanes" | |
| 19:11:35 | superdan | mriedem: lol, I was trying to help with your doc patches, sorry if that was not that helpful | |
| 19:12:00 | mriedem | superdan: you ruined EVERYTHING! | |
| 19:12:07 | mriedem | superdan: you just have to realize the sfinucan-isms | |
| 19:12:10 | mriedem | with .rst files | |
| 19:12:30 | superdan | mriedem: the one thing I know about sfinucanisms is that he gives a shit about the formatting and I do not | |
| 19:12:33 | superdan | I read it in text format | |
| 19:12:40 | mriedem | i also care about format | |
| 19:12:45 | mriedem | it's why sfinucan and i butt heads | |
| 19:12:54 | superdan | well, sorry, you gonna change it? | |
| 19:13:03 | mriedem | no, i don't really care | |
| 19:13:12 | superdan | ack | |
| 19:13:13 | mriedem | i'm fixing this live migration logging turd | |
| 19:13:23 | superdan | never again shall I do that | |
| 19:13:25 | mriedem | whilst live migration job is broken in ocata and blocking all changes | |
| 19:13:46 | mriedem | and kashyap wants nova injecting qemu command line stuff in production, it's insane :P | |
| 19:13:53 | kashyap | mriedem: No, no! | |
| 19:13:59 | kashyap | You're too eager to misunderstand, aren't you? | |
| 19:14:05 | mriedem | kashyap: i know all too well my friend | |
| 19:14:07 | kashyap | :P | |
| 19:14:32 | kashyap | mriedem: What is the failure? Actually Dave Gilbert from QEMU is also up and awake at this our | |
| 19:14:40 | kashyap | I see the failure you link above is due to a bogus error message | |
| 19:14:46 | kashyap | Is there anything more other than that? | |
| 19:15:08 | kashyap | (s/our/hour/) | |
| 19:15:29 | kashyap | Jeez - "logging turd" | |
| 19:16:33 | openstackgerrit | Samantha Blanco proposed openstack/nova master: Make flavor-rxtx policy more granular flavor-rxtx currently only has one policy it checks against, which is not restrictive enough when used in conjunction with other policies (e.g., flavor-manage). This allows users a greater degree of customization. https://review.openstack.org/444036 | |
| 19:16:38 | kashyap | mriedem: It can be fast-approved, I think? | |
| 19:17:00 | openstackgerrit | Samantha Blanco proposed openstack/nova master: Make flavor-rxtx policy more granular flavor-rxtx currently only has one policy it checks against, which is not restrictive enough when used in conjunction with other policies (e.g., flavor-manage). This allows users a greater degree of customization. https://review.openstack.org/444036 | |
| 19:21:38 | mriedem | kashyap: i don't actually need someone from qemu | |
| 19:21:54 | mriedem | i mean, i don't know why the guest wasn't found that caused this actual failure | |
| 19:21:58 | mriedem | but that's not what i'm fixing atm | |
| 19:22:20 | kashyap | mriedem: Thought, you were also looking at another bug which involves QEMU itself | |
| 19:22:56 | mriedem | well, this live migration job failed | |
| 19:23:15 | mriedem | because during post live migration, when source calls dest to complete the migration on that side, the guest domain wasn't found | |
| 19:23:29 | kashyap | mriedem: Got a link to a log, if you don't mind? | |
| 19:23:30 | mriedem | even though we were in post live migration | |
| 19:23:43 | kashyap | I think I should find one from here: logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/ | |
| 19:23:52 | mriedem | yes | |
| 19:25:07 | kashyap | The root dir (gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/) has destination logs | |
| 19:25:35 | mriedem | http://logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/screen-n-cpu.txt.gz#_2017-04-21_13_54_10_150 | |
| 19:25:47 | mriedem | yes ^ is the dest logs | |
| 19:28:09 | mriedem | http://logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/libvirt/qemu/instance-00000001.txt.gz | |
| 19:28:18 | mriedem | /build/qemu-5OJ39u/qemu-2.8+dfsg/nbd/server.c:nbd_receive_request():L710: read failed 2017-04-21 13:54:08.792+0000: shutting down, reason=failed | |
| 19:29:35 | kashyap | For future note: Incidentally, we're going to get finer-grained details about shutdown reasons | |
| 19:29:44 | kashyap | Whether it is guest-triggered, or host-triggerred | |
| 19:29:48 | kashyap | And what precisely caused it, etc | |
| 19:30:14 | kashyap | (There's a long thread on qemu-devel upstream, to iron out the details.) | |
| 19:31:07 | mriedem | well, /build/qemu-5OJ39u/qemu-2.8+dfsg/nbd/server.c:nbd_receive_request():L710: read failed | |
| 19:31:08 | mriedem | caused it | |
| 19:31:09 | mriedem | i guess | |
| 19:31:24 | kashyap | Yes, I recall seeing that | |
| 19:32:51 | mriedem | /build/qemu-5OJ39u/qemu-2.8+dfsg/nbd/server.c:nbd_receive_request():L710: read failed | |
| 19:32:53 | mriedem | oops | |
| 19:33:00 | mriedem | http://logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/libvirt/libvirtd.txt.gz#_2017-04-21_13_54_08_792 | |
| 19:33:00 | kashyap | mriedem: Yep, I recall it | |
| 19:33:13 | kashyap | I even reported it recently here: http://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg01086.html | |
| 19:33:40 | kashyap | Quoting myself: | |
| 19:33:42 | kashyap | on the destination, _without_ gracefully completing or cancelling the | |
| 19:33:42 | kashyap | Just writing for completeness' sake, if you try to stop the NBD server | |
| 19:33:43 | kashyap | `drive-mirror` job on the source, you see the below: | |