Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-06
16:32:21 fried_rice jroll: I don't know if you're involved in the whole requirements shift thingy, or if you're just able to look at it and know that that will dtrt
16:32:33 jroll fried_rice: that's how the other tox environments wind up running tests, look at the testenv:py35 section of that file
16:32:36 kashyap mriedem: That "Guest index YYYYY inconsistent with Host index" error rings a bell. And there _was_ a QEMU fix, let me look
16:32:41 jroll fried_rice: this is assuming the goal is to run unit tests there
16:33:11 jroll fried_rice: I'm not involved, just saw you mention here that you don't know what you need there, and thought I'd help :)
16:33:13 fried_rice jroll: Right, I get that it will do a thing, and that that thing is probably useful; but I didn't know if ^ assumption is actually true yet.
16:33:20 mriedem kashyap: yeah you're on https://bugzilla.redhat.com/show_bug.cgi?id=1450524
16:33:20 openstack bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang
16:33:28 kashyap Am I?
16:33:35 jroll fried_rice: ah, gotcha. question for someone else then :)
16:33:58 fried_rice jroll: Okay, thanks. I'll wait for dhellmann to respond (either saying, "verily, do as jroll says," or "yeah, we don't want to do that yet - amend the commit message").
16:34:19 jroll fried_rice: though based on the commit message, I'd guess unit tests were the goal: "Create a tox environment for running the unit tests against the lower bounds of the dependencies."
16:34:55 fried_rice jroll: yuh; but the email chain hints that maybe we're just setting up the framework right now. I dunno. Was just trying to ease Doug's burden a little.
16:35:17 jroll ¯\_(ツ)_/¯
16:35:18 arvindn05 mriedem: i might be missing something....lets say you have instance X with flavor F, while X is running, you go in and change the flavor F to F'(only adding 1 trait, no resource changes), so when its rebuild are we still using F or F'?
16:35:42 mriedem arvindn05: F
16:35:55 mriedem the flavor used to originally create the instance is stored in the db with the instance,
16:36:09 openstackgerrit Merged openstack/nova master: Use a pythonic delete, with a retry. https://review.openstack.org/554793
16:36:10 mriedem so if the admin deletes or changes extra specs on the original flavor, the instance is still using what it was created with
16:36:25 openstackgerrit Merged openstack/nova master: Clarify/correct the ordering of API and Cell database schema updates https://review.openstack.org/559357
16:36:30 mriedem i believe it's also persisted with the request spec
16:36:33 arvindn05 ahh...didnt know that. so basically we have a snapshot of the flavor with the instance....makes more sense now :)
16:36:37 mriedem yes
16:37:57 kashyap mriedem: So ... this seems to be the instance that crashed:
16:37:58 kashyap http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-multinode-live-migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz
16:38:00 mriedem kashyap: tracking in nova with https://bugs.launchpad.net/nova/+bug/1761798
16:38:00 openstack Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed]
16:38:03 mriedem kashyap: yup
16:39:25 kashyap mriedem: So, that RHBZ bug was closed with this (unsatisfying) conclusion:
16:39:28 kashyap [quote]
16:39:30 kashyap "Well, we know half of their problem was the old virtio-balloon bug - that's already fixed.
16:39:33 kashyap There is an unexplained 'virtio-net' problem they were seeing - but we've never managed to recreate it, so I don't know if there's anything we can do.
16:39:36 kashyap [/quote]
16:39:39 kashyap "their
16:39:39 kashyap (Err, "their" == the reporter)
16:41:03 mriedem yeah,
16:41:10 mriedem hoping that maybe this goes away when we start using the queens UCA packages
16:41:19 mriedem so far it's only 4 hits in 7 days
16:41:33 kashyap Nod; I'm trying to see which version is that fix in
16:41:41 kashyap ("that fix" alluded to in the above quote)
16:42:06 kashyap mriedem: Look at that comment: https://bugzilla.redhat.com/show_bug.cgi?id=1450524#c21
16:42:06 openstack bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang
16:42:19 kashyap "I'm not having any luck replicating this here; it's ran ~230 iterations of a migration [...]"
16:43:08 mriedem kashyap: yeah, don't really care about how often someone ran something in parallel in isolation
16:43:10 kashyap (I recall trying to get this reproducing; but was nigh impossible .)
16:43:32 kashyap Yeah; OpenStack Gate is singularly unique in identifying weird bugs
16:43:33 mriedem with openstack ci the tests are run concurrently on the same 2 hosts, and get run thousands of times per day
16:44:02 kashyap Yep, /me remembers the horrors of the "live snapshot bug" that was only reproducible on the Gate
16:44:09 mriedem anyway, not high priority, i wasn't sure if there was any link from the rhev package versions to versions of qemu we use
16:44:22 mriedem i don't know how the rhev package versions translate
16:44:43 kashyap ('qemu-kvm-rhev' is what CentOS uses)
16:45:10 mriedem otherwise the versions are the same?
16:45:13 kashyap It is the "enterprise" version that has all the 'advanced' features. (And 'qemu-kvm' is the plain binary.)
16:45:49 kashyap mriedem: Not exactly; I'll figure out the version translations and how they map to Ubuntu
16:45:53 kashyap (And write a comment)
16:46:00 mriedem thanks. we're hitting this with qemu 2.10
16:46:09 kashyap The eagle-eyed would notice the "NB2" noted by DanPB here: https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix
16:46:12 kashyap At the end of the page
16:47:05 mriedem i haven't had to care about rhev since i was at ibm and even then, about 8 years ago
16:47:18 kashyap mriedem: Me neither; the "-rhev" is just the QEMU binary split
16:47:19 mriedem superdan: gimme some MCP builds!
16:47:41 kashyap Real People use raw QEMU directly
16:47:42 superdan mriedem: no.
16:48:54 superdan mtreinish: can you hook us up? https://review.openstack.org/#/c/559358/3
16:51:35 arvindn05 mriedem: you got me thinking on instance migration as welll
16:51:52 mriedem arvindn05: images don't change on instance migration
16:52:15 arvindn05 mriedem: is the placement api called anytime during the migrations?
16:52:15 mriedem but, when migrating to a new host, the scheduler will validate the image used to create the instance as well
16:52:20 mriedem yes
16:52:25 mriedem as part of scheduling to find a new dest host
16:52:36 mriedem rebuild is weird in that the instance doesn't move
16:52:50 mriedem but the image can change so we have to re-validate that the new image can work on the same host
16:53:19 mriedem the original image used to create the instance is stored in the request_spec for that instance
16:53:31 mriedem so that's what the scheduler uses during a move operation
16:54:23 arvindn05 so when you say the scheduler will validate the image, the placement API will be requested for a matching compute host?
16:54:57 arvindn05 i see there is a "Manual selection of the destination host" per https://docs.openstack.org/nova/pike/admin/live-migration-usage.html
16:55:03 mriedem yes. scheduling asks placement for compute node resource providers that can fit the flavor constraints, and optionally qualitative traits defined in the flavor extra specs
16:55:37 mriedem arvindn05: yes, but that still runs through the scheduler but with the requested host
16:55:54 mriedem if you specify the force parameter to live migration with a host, then we bypass the scheduler and, well, you get what you ask for
16:56:17 arvindn05 ok...and we would get no valid host if when run through the scheduler, the host does not match
16:56:24 mriedem correct
16:57:22 arvindn05 can you point me to the code where the live migration calls the scheduler?(assuming it calls select destination method)
16:58:14 mriedem arvindn05: https://github.com/openstack/nova/blob/0de6ebd086656066c4dcc5c093b5cc7e92e035ac/nova/conductor/tasks/live_migrate.py#L319
16:59:29 arvindn05 mriedem: thank you very much :)
17:01:57 arvindn05 mriedem: btw https://review.openstack.org/#/c/557795/ -Update ImageMetaProp object to expose traits is ready for review as well
17:06:35 mriedem arvindn05: yeah i wanted to post this https://review.openstack.org/#/c/557795/14/nova/objects/image_meta.py@560 and get the ack from superdan and leakypipes first so we're on the same page
17:13:56 mtreinish superdan: is that ordering documented anywhere?
17:14:09 superdan mtreinish: it's documented in the right order
17:14:12 superdan mtreinish: this matches the docs
17:14:13 mriedem mtreinish: it is now https://review.openstack.org/#/c/559357/
17:14:18 superdan mriedem: dude, shsssshh
17:14:21 mriedem ha
17:14:51 mtreinish heh, ok well as long as it's written down now
17:15:17 mriedem mtreinish is like the substitute teacher on a friday that lets things slide
17:15:19 mtreinish +A
17:15:23 mriedem see!?
17:15:35 superdan hah
17:17:12 mtreinish lol
17:17:12 sean-k-mooney by the way when ye refer to cells in that doc does the ordering of api first apply equally to v1 and v2
17:20:06 sean-k-mooney i generally try to not know how cells v1 works in case anyone will ask me to help fix somthing with cellsv1 in the furutre but just taught i would ask.

Earlier   Later