| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-21 | |||
| 16:21:51 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/458834 | |
| 16:46:55 | openstackgerrit | Dan Smith proposed openstack/nova master: Tell people that the nova-cells man page is for cells v1 https://review.openstack.org/456925 | |
| 16:46:56 | openstackgerrit | Dan Smith proposed openstack/nova master: Add release note and update cell install guide for multi-cell limitations https://review.openstack.org/456923 | |
| 17:01:31 | coffee_cat | lyarwood: I had that thought rushing through my head as well re swap volume and reserve the new one | |
| 17:02:38 | coffee_cat | coffee_cat: it was my laziness at this point that I didn't think it through, I just wanted to get something up what we can argue over :) | |
| 17:16:19 | cfriesen | does it make sense to allow a rebuild of a boot-from-volume to a different image, given that we don't write the new image to the volume? | |
| 18:02:19 | mnestratov | sdague: hi, around? | |
| 18:02:26 | sdague | mnestratov: yeh, what's up? | |
| 18:02:39 | mnestratov | can you take a look at https://review.openstack.org/#/c/458912/ | |
| 18:08:54 | sdague | mnestratov: my only concern with that is that we're going to need to add it back in as soon as we roll to packages | |
| 18:09:10 | sdague | and we're going to fail some upgrades in the process | |
| 18:10:48 | mnestratov | sdague: hmm, and when do we plan this? as soon as packages are ready? | |
| 18:14:07 | sdague | mnestratov: yeh, centos already has new enough packages in epel, it might switch back over if epel comes back on | |
| 18:14:27 | sdague | ubuntu has packages, but the socket interface isn't stable yet in them | |
| 18:15:54 | mnestratov | I see, well, looks we shouldn't merge this at all then | |
| 18:31:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: encryptors: Switch to os-brick encryptor classes https://review.openstack.org/391597 | |
| 18:33:27 | mriedem | i don't know how to debug this https://bugs.launchpad.net/nova/+bug/1685333 | |
| 18:33:27 | openstack | Launchpad bug 1685333 in OpenStack Compute (nova) "Fatal Python error: Cannot recover from stack overflow. - in py35 unit test job" [High,Confirmed] | |
| 18:43:11 | mriedem | interesting | |
| 18:43:26 | mriedem | live migration job failed because the destination said the guest wasn't found | |
| 18:43:37 | mriedem | but on the source side, the logs say "You may see the error "libvirt: QEMU error: Domain not found: no domain with matching name." This error can be safely ignored." | |
| 18:47:43 | mriedem | oh fun that info message is in the compute manager | |
| 18:47:48 | mriedem | where i might not be using libvirt at all | |
| 18:54:53 | mriedem | oh no wonder this goes back to diablo in a commit with no explanation | |
| 18:56:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove archaic reference to QEMU errors during post live migration https://review.openstack.org/458935 | |
| 18:57:39 | mriedem | also | |
| 18:57:40 | mriedem | Post live migration at destination ubuntu-xenial-2-node-osic-cloud1-s3500-8527282 failed | |
| 18:57:51 | mriedem | Migrating instance to ubuntu-xenial-2-node-osic-cloud1-s3500-8527282 finished successfully. | |
| 18:58:00 | mriedem | wtf | |
| 18:58:59 | kashyap | mriedem: Oops - that error message, specific to driver | |
| 18:59:11 | mriedem | kashyap: well, it's probably from when libvirt was the only driver | |
| 18:59:38 | kashyap | Hmm. Are other drivers, besides Xen, are well-maintained? | |
| 19:00:59 | mriedem | hypev | |
| 19:01:01 | mriedem | *hyperv | |
| 19:01:27 | kashyap | Yeah. | |
| 19:01:53 | kashyap | So, will Nova support Moby Containers? :P | |
| 19:03:42 | kashyap | Was debugging a "fun" issue where exposing a CPU "L3" cache (via QEMU's "-cpu host,l3-cache=on") to a guest will improve performance by 10% or so for some SAP workloads! | |
| 19:04:04 | mriedem | kashyap: you should talk to cfriesen then | |
| 19:04:11 | mriedem | he's been looking for someone that loves qemu and libvirt for the last 3 days | |
| 19:04:13 | kashyap | mriedem: Found out the root cause, actually | |
| 19:04:21 | kashyap | But luckily, no action item needed for upstream Nova, Because: | |
| 19:04:46 | kashyap | We "get it for free", with QEMU 2.8 and above. | |
| 19:04:52 | kashyap | As it is part of the _default_ machine type | |
| 19:05:19 | kashyap | So, if anyone is hitting it with Nova, using an older QEMU, still "no problem", we could use the always-handy (but voids support warranty) workaround: | |
| 19:05:46 | kashyap | Libvirt's XML namespace <qemu:commandline> [...] </qemu:commandline> | |
| 19:06:44 | kashyap | While upstream libvirt is trying to wire it into its QEMU command-line generator. So all good. And I can wind up for the night | |
| 19:06:50 | mriedem | is that like injecting stuff into rabbit at runtime using erlang? | |
| 19:07:00 | kashyap | Hehe | |
| 19:07:08 | kashyap | cfriesen: You might be interested in this -- https://www.redhat.com/archives/libvir-list/2017-April/msg00815.html | |
| 19:07:18 | kashyap | Unfortunately, the link won't work :-( | |
| 19:07:29 | kashyap | As there's a major outage in one of the critical datacenters | |
| 19:08:21 | kashyap | But rest assured, somewhere some tired admin is sweating to hell and back trying to fix the outage ... | |
| 19:09:17 | kashyap | mriedem: But more seriously, that namespace is very handy -- we (Nova) could test interesting features and can decide: "Okay, this and that combination of QEMU command-line gives Nova users some advantage -- so let's file an RFE to get that wired up into official libvirt APIs" | |
| 19:09:22 | mriedem | kashyap: maybe it's down b/c he was injecting stuff via qemu command line to the infra running the server | |
| 19:09:41 | kashyap | mriedem: Yeah, that workaround should be a one-off hack only. | |
| 19:10:01 | kashyap | Because, when using that workaroud, in logs, you'd see the instances to be marked as "tainted" (rightfully) | |
| 19:10:19 | kashyap | Warning the opertors that: "Look, your libvirt / QEMU upgrade might break your existing guests" | |
| 19:10:41 | kashyap | And of course, to help support people identify if someone is using unofficial command-line | |
| 19:10:50 | mriedem | i'm much more interested in making sure the features we've already landed actually work | |
| 19:10:57 | kashyap | mriedem: I'm all with you | |
| 19:10:59 | mriedem | like when live migration fails and we tell you it was ok | |
| 19:11:27 | kashyap | "Stability" over "yeah, let's bake that half-assed feature in, and let it rot after initial enthusiasm wanes" | |
| 19:11:35 | superdan | mriedem: lol, I was trying to help with your doc patches, sorry if that was not that helpful | |
| 19:12:00 | mriedem | superdan: you ruined EVERYTHING! | |
| 19:12:07 | mriedem | superdan: you just have to realize the sfinucan-isms | |
| 19:12:10 | mriedem | with .rst files | |
| 19:12:30 | superdan | mriedem: the one thing I know about sfinucanisms is that he gives a shit about the formatting and I do not | |
| 19:12:33 | superdan | I read it in text format | |
| 19:12:40 | mriedem | i also care about format | |
| 19:12:45 | mriedem | it's why sfinucan and i butt heads | |
| 19:12:54 | superdan | well, sorry, you gonna change it? | |
| 19:13:03 | mriedem | no, i don't really care | |
| 19:13:12 | superdan | ack | |
| 19:13:13 | mriedem | i'm fixing this live migration logging turd | |
| 19:13:23 | superdan | never again shall I do that | |
| 19:13:25 | mriedem | whilst live migration job is broken in ocata and blocking all changes | |
| 19:13:46 | mriedem | and kashyap wants nova injecting qemu command line stuff in production, it's insane :P | |
| 19:13:53 | kashyap | mriedem: No, no! | |
| 19:13:59 | kashyap | You're too eager to misunderstand, aren't you? | |
| 19:14:05 | mriedem | kashyap: i know all too well my friend | |
| 19:14:07 | kashyap | :P | |
| 19:14:32 | kashyap | mriedem: What is the failure? Actually Dave Gilbert from QEMU is also up and awake at this our | |
| 19:14:40 | kashyap | I see the failure you link above is due to a bogus error message | |
| 19:14:46 | kashyap | Is there anything more other than that? | |
| 19:15:08 | kashyap | (s/our/hour/) | |
| 19:15:29 | kashyap | Jeez - "logging turd" | |
| 19:16:33 | openstackgerrit | Samantha Blanco proposed openstack/nova master: Make flavor-rxtx policy more granular flavor-rxtx currently only has one policy it checks against, which is not restrictive enough when used in conjunction with other policies (e.g., flavor-manage). This allows users a greater degree of customization. https://review.openstack.org/444036 | |
| 19:16:38 | kashyap | mriedem: It can be fast-approved, I think? | |
| 19:17:00 | openstackgerrit | Samantha Blanco proposed openstack/nova master: Make flavor-rxtx policy more granular flavor-rxtx currently only has one policy it checks against, which is not restrictive enough when used in conjunction with other policies (e.g., flavor-manage). This allows users a greater degree of customization. https://review.openstack.org/444036 | |
| 19:21:38 | mriedem | kashyap: i don't actually need someone from qemu | |
| 19:21:54 | mriedem | i mean, i don't know why the guest wasn't found that caused this actual failure | |
| 19:21:58 | mriedem | but that's not what i'm fixing atm | |
| 19:22:20 | kashyap | mriedem: Thought, you were also looking at another bug which involves QEMU itself | |
| 19:22:56 | mriedem | well, this live migration job failed | |
| 19:23:15 | mriedem | because during post live migration, when source calls dest to complete the migration on that side, the guest domain wasn't found | |
| 19:23:29 | kashyap | mriedem: Got a link to a log, if you don't mind? | |
| 19:23:30 | mriedem | even though we were in post live migration | |
| 19:23:43 | kashyap | I think I should find one from here: logs.openstack.org/43/458843/1/check/gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/ | |
| 19:23:52 | mriedem | yes | |
| 19:25:07 | kashyap | The root dir (gate-tempest-dsvm-multinode-live-migration-ubuntu-xenial/697a501/logs/) has destination logs | |