Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-06
15:28:05 mriedem odyssey4me: left a comment, i think this might fail
15:28:08 mriedem we'll see
15:28:23 openstackgerrit Dan Smith proposed openstack/nova master: Clarify/correct the ordering of API and Cell database schema updates https://review.openstack.org/559357
15:28:26 superdan ooh, look how many words!
15:29:39 jgwentworth you dun good
15:30:46 odyssey4me mriedem question to clarify before I revise
15:30:51 odyssey4me (in review)
15:33:50 tssurya https://review.openstack.org/#/c/559358/
15:35:45 superdan tssurya: comments on the commit message
15:35:53 tssurya looking
15:36:02 superdan also probably need jgwentworth's sign-off on the final version before we put it in
15:36:14 superdan we usually try to require that for things like this in grenade
15:36:55 tssurya sure :)
15:37:04 jgwentworth I shall ack it
15:38:37 superdan tssurya: if my friday-speak wasn't clear enough on that review let me know and I'll try to plain it up a little
15:39:50 sususuryashines superdan: yes please
15:40:04 jgwentworth hah, good friday nick
15:40:14 sususuryashines for starters did you only want a change in explanation or the title also seems odd to you ?
15:40:30 superdan no, the title seems okay
15:40:33 sususuryashines jgwentworth: heh thanks ;)
15:40:39 superdan let me just write what I think it should be
15:41:55 mriedem sususuryashines: you could just lift from this https://review.openstack.org/#/c/559357/
15:42:47 sususuryashines mriedem: yes I thought so at first, but that commit message is a little different from superdan's recommendations on my commit message.
15:43:27 mriedem it's different in that it doesn't have all the meh and stuff :)
15:44:23 superdan sususuryashines: left a suggested block of text
15:44:52 sususuryashines superdan: historical :)
15:45:03 sususuryashines ack, will read and change
15:45:04 sususuryashines thanks
15:45:08 superdan sususuryashines: it's a lot like my doc patch, but a little more grenade-y.. feel free to re-write it in your words, I just want it to sound more like "oops, this was wrong all along" and not "we decided to change this when something broke"
15:45:29 sususuryashines superdan: ack
16:12:57 mriedem fried_rice: thanks for going over https://review.openstack.org/#/c/558001/ - i left some replies, will wait on you before making any changes there
16:13:06 fried_rice mriedem: ack
16:13:37 mriedem i think at this point it's likely easiest to say, "if you use OVS everywhere, just set this to True"
16:14:04 mriedem if you're mnaser and enjoy pain, then leave it False
16:14:10 mnaser hi
16:14:23 mriedem just making a little ODL joke
16:14:26 mriedem at your expense
16:14:54 mnaser oh
16:15:10 mnaser mriedem: if you're really into pain then you're run nova with multiple network backends
16:15:26 mnaser s/run/running/
16:15:29 mnaser s/running/trying to run/
16:15:41 mriedem yeah, that's what the new config option in that patch is for
16:15:56 mnaser thanks for the ping about it though
16:19:38 mriedem kashyap: ever seen this live migration failure? http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-multinode-live-migration/8de6e74/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_05_21_48_43_258043 i've seen this in a few different live migration runs this week
16:19:43 mriedem VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8
16:19:53 mriedem error while loading state for instance 0x0 of device '0000:00:04.0/virtio-blk'
16:22:44 mriedem http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-multinode-live-migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz
16:24:23 mriedem found https://bugzilla.redhat.com/show_bug.cgi?id=1450524
16:24:23 openstack bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang
16:24:44 arvindn05 mriedem: i added comments to https://review.openstack.org/#/c/541507/9
16:24:56 fried_rice mriedem: Responded
16:25:30 arvindn05 also had a question on https://review.openstack.org/#/c/546357/ - Skip placement on rebuild in same host
16:26:00 arvindn05 why are we allowing the skipping placement given we know the image can now contain qualitative aspects?
16:26:46 arvindn05 also what if the qualitative aspects in the flavor changes? would we want to run placement in this case as well??
16:28:35 mriedem arvindn05: for one, we aren't yet allowing images with required traits
16:28:47 mriedem that patch fixes a bug which has to go back to pike
16:28:58 mriedem two, rebuild doesn't take a new flavor
16:29:28 arvindn05 ahh...missed that it was pike...
16:29:39 mriedem rebuild does take a new image, and the image meta could change, so we have to re-run some scheduler filters
16:29:49 mriedem this patch is master, but fixes a bug that was introduced in pike
16:30:11 arvindn05 for #2, it doesnt need to take a new flavor right? if the flavor was updated for the instance, then we might want to run placement
16:30:21 mriedem arvindn05: my point was, once we do start accepting images with required qualitative traits, the logic in this bug fix will likely need to change, as part of your blueprint
16:30:38 mriedem arvindn05: the flavor doesn't get updated for the instance unless you resize the instance
16:30:47 mriedem in which case you're not doing a rebuild, you're doing a cold migration with a new flavor
16:30:53 mriedem to a new host
16:31:25 kashyap mriedem: Was AFK; looking
16:31:39 fried_rice jroll: Was this just a suggestion, or is this the way we're supposed to be setting these up? https://review.openstack.org/#/c/555961/3/tox.ini@243
16:32:15 arvindn05 got it...will keep in mind to change the patch
16:32:21 fried_rice jroll: I don't know if you're involved in the whole requirements shift thingy, or if you're just able to look at it and know that that will dtrt
16:32:33 jroll fried_rice: that's how the other tox environments wind up running tests, look at the testenv:py35 section of that file
16:32:36 kashyap mriedem: That "Guest index YYYYY inconsistent with Host index" error rings a bell. And there _was_ a QEMU fix, let me look
16:32:41 jroll fried_rice: this is assuming the goal is to run unit tests there
16:33:11 jroll fried_rice: I'm not involved, just saw you mention here that you don't know what you need there, and thought I'd help :)
16:33:13 fried_rice jroll: Right, I get that it will do a thing, and that that thing is probably useful; but I didn't know if ^ assumption is actually true yet.
16:33:20 mriedem kashyap: yeah you're on https://bugzilla.redhat.com/show_bug.cgi?id=1450524
16:33:20 openstack bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang
16:33:28 kashyap Am I?
16:33:35 jroll fried_rice: ah, gotcha. question for someone else then :)
16:33:58 fried_rice jroll: Okay, thanks. I'll wait for dhellmann to respond (either saying, "verily, do as jroll says," or "yeah, we don't want to do that yet - amend the commit message").
16:34:19 jroll fried_rice: though based on the commit message, I'd guess unit tests were the goal: "Create a tox environment for running the unit tests against the lower bounds of the dependencies."
16:34:55 fried_rice jroll: yuh; but the email chain hints that maybe we're just setting up the framework right now. I dunno. Was just trying to ease Doug's burden a little.
16:35:17 jroll ¯\_(ツ)_/¯
16:35:18 arvindn05 mriedem: i might be missing something....lets say you have instance X with flavor F, while X is running, you go in and change the flavor F to F'(only adding 1 trait, no resource changes), so when its rebuild are we still using F or F'?
16:35:42 mriedem arvindn05: F
16:35:55 mriedem the flavor used to originally create the instance is stored in the db with the instance,
16:36:09 openstackgerrit Merged openstack/nova master: Use a pythonic delete, with a retry. https://review.openstack.org/554793
16:36:10 mriedem so if the admin deletes or changes extra specs on the original flavor, the instance is still using what it was created with
16:36:25 openstackgerrit Merged openstack/nova master: Clarify/correct the ordering of API and Cell database schema updates https://review.openstack.org/559357
16:36:30 mriedem i believe it's also persisted with the request spec
16:36:33 arvindn05 ahh...didnt know that. so basically we have a snapshot of the flavor with the instance....makes more sense now :)
16:36:37 mriedem yes
16:37:57 kashyap mriedem: So ... this seems to be the instance that crashed:
16:37:58 kashyap http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-multinode-live-migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz
16:38:00 mriedem kashyap: tracking in nova with https://bugs.launchpad.net/nova/+bug/1761798
16:38:00 openstack Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed]
16:38:03 mriedem kashyap: yup
16:39:25 kashyap mriedem: So, that RHBZ bug was closed with this (unsatisfying) conclusion:
16:39:28 kashyap [quote]
16:39:30 kashyap "Well, we know half of their problem was the old virtio-balloon bug - that's already fixed.
16:39:33 kashyap There is an unexplained 'virtio-net' problem they were seeing - but we've never managed to recreate it, so I don't know if there's anything we can do.

Earlier   Later