Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-06
16:36:37 mriedem yes
16:37:57 kashyap mriedem: So ... this seems to be the instance that crashed:
16:37:58 kashyap http://logs.openstack.org/37/522537/20/check/legacy-tempest-dsvm-multinode-live-migration/8de6e74/logs/subnode-2/libvirt/qemu/instance-00000002.txt.gz
16:38:00 openstack Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed]
16:38:00 mriedem kashyap: tracking in nova with https://bugs.launchpad.net/nova/+bug/1761798
16:38:03 mriedem kashyap: yup
16:39:25 kashyap mriedem: So, that RHBZ bug was closed with this (unsatisfying) conclusion:
16:39:28 kashyap [quote]
16:39:30 kashyap "Well, we know half of their problem was the old virtio-balloon bug - that's already fixed.
16:39:33 kashyap There is an unexplained 'virtio-net' problem they were seeing - but we've never managed to recreate it, so I don't know if there's anything we can do.
16:39:36 kashyap [/quote]
16:39:39 kashyap (Err, "their" == the reporter)
16:39:39 kashyap "their
16:41:03 mriedem yeah,
16:41:10 mriedem hoping that maybe this goes away when we start using the queens UCA packages
16:41:19 mriedem so far it's only 4 hits in 7 days
16:41:33 kashyap Nod; I'm trying to see which version is that fix in
16:41:41 kashyap ("that fix" alluded to in the above quote)
16:42:06 openstack bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang
16:42:06 kashyap mriedem: Look at that comment: https://bugzilla.redhat.com/show_bug.cgi?id=1450524#c21
16:42:19 kashyap "I'm not having any luck replicating this here; it's ran ~230 iterations of a migration [...]"
16:43:08 mriedem kashyap: yeah, don't really care about how often someone ran something in parallel in isolation
16:43:10 kashyap (I recall trying to get this reproducing; but was nigh impossible .)
16:43:32 kashyap Yeah; OpenStack Gate is singularly unique in identifying weird bugs
16:43:33 mriedem with openstack ci the tests are run concurrently on the same 2 hosts, and get run thousands of times per day
16:44:02 kashyap Yep, /me remembers the horrors of the "live snapshot bug" that was only reproducible on the Gate
16:44:09 mriedem anyway, not high priority, i wasn't sure if there was any link from the rhev package versions to versions of qemu we use
16:44:22 mriedem i don't know how the rhev package versions translate
16:44:43 kashyap ('qemu-kvm-rhev' is what CentOS uses)
16:45:10 mriedem otherwise the versions are the same?
16:45:13 kashyap It is the "enterprise" version that has all the 'advanced' features. (And 'qemu-kvm' is the plain binary.)
16:45:49 kashyap mriedem: Not exactly; I'll figure out the version translations and how they map to Ubuntu
16:45:53 kashyap (And write a comment)
16:46:00 mriedem thanks. we're hitting this with qemu 2.10
16:46:09 kashyap The eagle-eyed would notice the "NB2" noted by DanPB here: https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix
16:46:12 kashyap At the end of the page
16:47:05 mriedem i haven't had to care about rhev since i was at ibm and even then, about 8 years ago
16:47:18 kashyap mriedem: Me neither; the "-rhev" is just the QEMU binary split
16:47:19 mriedem superdan: gimme some MCP builds!
16:47:41 kashyap Real People use raw QEMU directly
16:47:42 superdan mriedem: no.
16:48:54 superdan mtreinish: can you hook us up? https://review.openstack.org/#/c/559358/3
16:51:35 arvindn05 mriedem: you got me thinking on instance migration as welll
16:51:52 mriedem arvindn05: images don't change on instance migration
16:52:15 mriedem but, when migrating to a new host, the scheduler will validate the image used to create the instance as well
16:52:15 arvindn05 mriedem: is the placement api called anytime during the migrations?
16:52:20 mriedem yes
16:52:25 mriedem as part of scheduling to find a new dest host
16:52:36 mriedem rebuild is weird in that the instance doesn't move
16:52:50 mriedem but the image can change so we have to re-validate that the new image can work on the same host
16:53:19 mriedem the original image used to create the instance is stored in the request_spec for that instance
16:53:31 mriedem so that's what the scheduler uses during a move operation
16:54:23 arvindn05 so when you say the scheduler will validate the image, the placement API will be requested for a matching compute host?
16:54:57 arvindn05 i see there is a "Manual selection of the destination host" per https://docs.openstack.org/nova/pike/admin/live-migration-usage.html
16:55:03 mriedem yes. scheduling asks placement for compute node resource providers that can fit the flavor constraints, and optionally qualitative traits defined in the flavor extra specs
16:55:37 mriedem arvindn05: yes, but that still runs through the scheduler but with the requested host
16:55:54 mriedem if you specify the force parameter to live migration with a host, then we bypass the scheduler and, well, you get what you ask for
16:56:17 arvindn05 ok...and we would get no valid host if when run through the scheduler, the host does not match
16:56:24 mriedem correct
16:57:22 arvindn05 can you point me to the code where the live migration calls the scheduler?(assuming it calls select destination method)
16:58:14 mriedem arvindn05: https://github.com/openstack/nova/blob/0de6ebd086656066c4dcc5c093b5cc7e92e035ac/nova/conductor/tasks/live_migrate.py#L319
16:59:29 arvindn05 mriedem: thank you very much :)
17:01:57 arvindn05 mriedem: btw https://review.openstack.org/#/c/557795/ -Update ImageMetaProp object to expose traits is ready for review as well
17:06:35 mriedem arvindn05: yeah i wanted to post this https://review.openstack.org/#/c/557795/14/nova/objects/image_meta.py@560 and get the ack from superdan and leakypipes first so we're on the same page
17:13:56 mtreinish superdan: is that ordering documented anywhere?
17:14:09 superdan mtreinish: it's documented in the right order
17:14:12 superdan mtreinish: this matches the docs
17:14:13 mriedem mtreinish: it is now https://review.openstack.org/#/c/559357/
17:14:18 superdan mriedem: dude, shsssshh
17:14:21 mriedem ha
17:14:51 mtreinish heh, ok well as long as it's written down now
17:15:17 mriedem mtreinish is like the substitute teacher on a friday that lets things slide
17:15:19 mtreinish +A
17:15:23 mriedem see!?
17:15:35 superdan hah
17:17:12 sean-k-mooney by the way when ye refer to cells in that doc does the ordering of api first apply equally to v1 and v2
17:17:12 mtreinish lol
17:20:06 sean-k-mooney i generally try to not know how cells v1 works in case anyone will ask me to help fix somthing with cellsv1 in the furutre but just taught i would ask.
17:21:36 mriedem don't ask
17:21:47 mriedem the rule with "helping people with cells v1" is, don't help them
17:22:01 sean-k-mooney :)
17:22:02 mriedem failing that, goto superdan or jgwentworth
17:23:59 sean-k-mooney i approch cellsv1 the opisitway i approch most of the rest of the codebase. i want to know how most of the codebase works so i can help others but with cellsv1 i tend to follow the above advice
17:25:10 mriedem finucannot: wtf does this mean? http://logs.openstack.org/70/559370/1/check/build-sphinx-docs/bda526f/job-output.txt.gz#_2018-04-06_16_47_02_376553
17:25:13 mriedem and how do i fix it?
17:27:04 ildikov mriedem: it looks a little like when the 'todo' extension was missing from the config
17:27:09 openstackgerrit Matt Riedemann proposed openstack/nova master: Cleanup _get_request_spec_for_select_destinations for live migrate https://review.openstack.org/559381
17:28:07 mriedem ildikov: it's something with this new PTI docs change i think, but i don't have the heart to read that thread right now
17:36:18 sean-k-mooney mriedem: do you need the "-" in query: >- also gr say sphinx>=1.6.2,!=1.6.6,!=1.6.7 # BSD but the test requirements in elastic-rechec is sphinx>=1.1.2,<1.2
17:37:03 sean-k-mooney mriedem: so maybe just use a non achient version of sphinx?
17:38:46 sean-k-mooney mriedem: it does not look like the openstack requirement proposal bot keeps the repo in sync so when they bumped to new pbr on march 1st perhaps that has issues since pbr is trying to move to sphinx >1.6
17:41:24 sean-k-mooney lower constratings also reqiures 1.6.5 https://github.com/openstack/requirements/blob/master/lower-constraints.txt#L359 so the 1.2 max in openstack-infra/elastic-recheck is proable the issue
17:43:44 arvindn05 mriedem: thanks. jaypipes said he would approve as well...will work on my next patch for the BP and also start thinking about the resize issue for rocky
17:55:21 sean-k-mooney mriedem: https://review.openstack.org/559390 fixes your issue. do you want me to rebase your patch on top.
18:01:18 mriedem sean-k-mooney: i can, was looking at the pti email and doing some stuff, but if this works, sure
18:02:40 sean-k-mooney mriedem: locally at least this fixes it for me. i cherry picked you patch locallly and that passed the docs job too after i bumped the version
18:03:09 cdent leakypipes, fried_rolls, figleaf : this might be of interest, a sort of placement test env: https://github.com/cdent/placecat (not fully fleshed out yet, just playing so far)
18:04:05 sean-k-mooney mriedem: by the way pti do you mean the "[openstack-dev] Following the new PTI for document build, broken local builds" thread from finucannot
18:04:18 leakypipes cdent: nice :)
18:06:31 sean-k-mooney mriedem: hum the chages for pti seam simple ill try making those instead

Earlier   Later