| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-19 | |||
| 10:28:31 | openstackgerrit | Merged openstack/nova master: Fix the formatting for 2.56 in the compute REST API history doc https://review.openstack.org/528114 | |
| 10:41:02 | lyarwood | kashyap: http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/logs/subnode-2/screen-n-cpu.txt.gz#_2017-12-18_20_20_13_894 - seeing this on stable/newton, LM failure for a single paused instance, looks like the remote libvirtd didn't respond in time, have you seen this before? | |
| 10:41:09 | lyarwood | kashyap: http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/job-output.txt.gz#_2017-12-18_20_20_56_230121 is the tempest failure | |
| 10:42:04 | kashyap | lyarwood: Is this only stable/newton? | |
| 10:42:29 | lyarwood | kashyap: that's the only place I've seen it thus far | |
| 10:50:20 | kashyap | lyarwood: So I looked at all the logs, one thing that potentially jumps out at me is in the source QEMU log: | |
| 10:50:23 | kashyap | [...] | |
| 10:50:24 | kashyap | [...] | |
| 10:50:27 | kashyap | warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] | |
| 10:50:30 | kashyap | main-loop: WARNING: I/O thread spun for 1000 iterations | |
| 10:50:52 | kashyap | Now, that warning isn't really an egregious error (& upstream QEMU is aware of it; it's a hard thing to fix), but that might be contributing to it | |
| 10:51:04 | kashyap | Where "it" being the timeout we see: | |
| 10:51:05 | kashyap | 2017-12-18 20:20:13.880+0000: 16820: error : virKeepAliveTimerInternal:143 : internal error: connection closed due to keepalive timeout | |
| 10:51:08 | kashyap | 2017-12-18 20:20:13.881+0000: 16825: error : virKeepAliveTimerInternal:143 : internal error: connection closed due to keepalive timeout | |
| 10:51:11 | kashyap | (From the source) | |
| 10:52:10 | lyarwood | kashyap: kk, so the dest was stuck, didn't send a keepalive and everything dies? | |
| 10:52:32 | kashyap | lyarwood: Yeah, from the destination livirtd log: | |
| 10:52:35 | kashyap | 2017-12-18 20:20:19.394+0000: 23816: error : qemuMonitorIOWrite:545 : Unable to write to monitor: Broken pipe | |
| 10:53:01 | kashyap | The above means libvirt lost access to the QMP socket connection, i.e. VM died | |
| 10:53:39 | lyarwood | kashyap: it's paused, that shouldn't cause the QMP socket to die however right? | |
| 10:55:27 | kashyap | lyarwood: Yeah, clearly something is wonky. I'll check this w/ Dave Gilbert as he meditates on migration | |
| 10:55:53 | kashyap | But the process _is_ killed, as we see from the destination (http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/logs/subnode-2/libvirt/qemu/instance-00000004.txt.gz): | |
| 10:55:58 | kashyap | 2017-12-18T20:17:51.612132Z qemu-system-x86_64: terminating on signal 15 from pid 16820 | |
| 10:58:06 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: libvirt: don't call sync_guest_time if qga is not enabled https://review.openstack.org/524836 | |
| 10:58:26 | lyarwood | kashyap: kk, thanks, I'm going to recheck this change and see if we can hit it again | |
| 10:58:37 | kashyap | So it is the live block migration, right | |
| 10:58:43 | kashyap | (To pause it) | |
| 10:58:47 | kashyap | LiveMigrationTest.test_live_block_migration_paused | |
| 10:59:10 | lyarwood | kashyap: yup, LM of a paused instance without shared storage | |
| 11:01:22 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | |
| 11:01:23 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | |
| 11:03:00 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | |
| 11:03:01 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | |
| 11:13:27 | openstackgerrit | Merged openstack/nova master: Implement query param schema for migration index https://review.openstack.org/518644 | |
| 11:14:16 | kashyap | lyarwood: So a couple of things from interacting w/ Dan & Dave from QEMU: | |
| 11:14:54 | kashyap | (1) You see the _later_ log messages (on destination) are 3 before the earlier log message. | |
| 11:15:23 | kashyap | So that's some weird timestamps there. | |
| 11:17:08 | openstack | bugzilla.redhat.com bug 1367620 in libvirt "storage migration fails due to keepalive timeout" [High,Assigned] - Assigned to jdenemar | |
| 11:17:08 | kashyap | (2) We're not the first to hit this case; there's this existing bug https://bugzilla.redhat.com/show_bug.cgi?id=1367620 ("storage migration fails due to keepalive timeout") | |
| 11:18:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose BDM uuid to drivers https://review.openstack.org/529037 | |
| 11:19:24 | kashyap | lyarwood: Wonder if you have a link to how many times this was hit in the past / this week? | |
| 11:21:52 | lyarwood | kashyap: I don't have one to hand now but I can create one, and an upstream bug for this | |
| 11:22:04 | mdbooth | lyarwood: Don't know if you're still looking at it, but I've been messing with that series this morning. | |
| 11:22:06 | lyarwood | assuming we've seen it more than once | |
| 11:22:09 | mdbooth | Not quite finished yet. | |
| 11:22:32 | lyarwood | mdbooth: kk I stopped after the earlier change sorry | |
| 11:23:00 | mdbooth | lyarwood: NP. I'd have been messing you about anyway. | |
| 11:23:13 | kashyap | lyarwood: I'm creating a potential reproducer, and can file one with that | |
| 11:23:23 | kashyap | lyarwood: But if you've already drafted a bug / issue, go ahead & submit it | |
| 11:27:58 | openstackgerrit | Merged openstack/nova master: Remove 'nova-manage shell' command https://review.openstack.org/521835 | |
| 11:32:15 | openstackgerrit | Matthew Booth proposed openstack/nova master: Give volume DriverBlockDevice classes a common prefix https://review.openstack.org/526346 | |
| 11:32:16 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347 | |
| 11:34:34 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | |
| 11:34:34 | openstackgerrit | Matthew Booth proposed openstack/nova master: Use real block_device_info data in test_blockinfo https://review.openstack.org/527916 | |
| 11:34:35 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | |
| 11:34:35 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | |
| 11:34:36 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | |
| 11:53:45 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Change 'InstancePCIRequest' spec field https://review.openstack.org/449257 | |
| 11:53:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Format NIC features using os-traits definitions https://review.openstack.org/466051 | |
| 11:53:46 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add Neutron port capabilities to devspec in request https://review.openstack.org/451777 | |
| 12:00:28 | stephenfin | ralonsoh: If you have the time to address it, I rebased and left a question on https://review.openstack.org/#/c/449257/ | |
| 12:00:31 | openstackgerrit | Merged openstack/nova master: Pass mountpoint to volume attachment_update https://review.openstack.org/527468 | |
| 12:02:09 | ralonsoh | stephenfin: I'll take a look to those patches on Friday. This patch depends on other two patches, 449257 and 451777 | |
| 12:02:30 | stephenfin | ralonsoh: 449257 is the one I'm referring to :) | |
| 12:02:44 | ralonsoh | stephenfin: https://review.openstack.org/#/c/466051/ shouldn't be on top of master | |
| 12:03:18 | ralonsoh | stephenfin: ok, thanks! I'll take a look at those patches on Friday | |
| 12:03:24 | stephenfin | ralonsoh: It isn't - it's on top of 449257 and 451777 | |
| 12:03:58 | ralonsoh | stephenfin: I was lookin at the wrong patch | |
| 12:04:03 | ralonsoh | looking | |
| 12:04:24 | openstack | Launchpad bug 1739013 in OpenStack Compute (nova) "nova.tests.functional.test_server_group.ServerGroupTest*.test_evacuate_with_anti_affinity does not validate that evacuation really happens" [Undecided,New] | |
| 12:04:24 | gibi | jaypipes: some extra madness for the server_group functional tests https://bugs.launchpad.net/nova/+bug/1739013 | |
| 12:07:11 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/python-novaclient master: Microversion 2.58 - Instance actions list pagination https://review.openstack.org/528601 | |
| 12:17:59 | jaypipes | gibi: not sure how much more madness you can get in that... :) | |
| 12:18:07 | ebbex | I've created a server with swap, where both root and swap are rbd, yet I have a "huge" swap-file on my compute node under nova/instances/_base/, and I see in the logs a "nova-rootwrap touch -c ...ova/instances/_base/swap_16384" going off about once a minute on that compute node. Where can I read up on the code that creates that file, and how long is the swap-file supposed to stay there? | |
| 12:19:17 | jaypipes | ebbex: the swap file should stay there for the life of the VM (since it's the swap content for the image...) | |
| 12:20:05 | jaypipes | ebbex: though I'm not sure why you'd see the touch -c command show up more than once. that's odd... | |
| 12:22:04 | ebbex | "virsh domblklist instance-00000041" gives: vdb vms/6e366e4b-2d87-48ac-a99c-999706e7e4f0_disk.swap, (on the ceph cluster) which I take it is where the instance gets to write swap to, right? No actual writes going to the _base/swap | |
| 12:24:20 | ebbex | I'm afraid that we might end up with a full disk thanks to swap images on our computenode as we have really small disks there. Yet vast amounts of storage on ceph. | |
| 12:25:57 | jaypipes | ebbex: hmm, I'm not sure. sure... mdbooth you around? | |
| 12:26:01 | lyarwood | mdbooth: ^ that smells like a bug, looksing at the code the fetch_func for creating swap is always _create_swap in nova/virt/libvirt/driver.py | |
| 12:26:10 | lyarwood | looksing | |
| 12:26:11 | lyarwood | :| | |
| 12:26:16 | jaypipes | mdbooth: does swap file get fulfilled by local disk even when ceph is used? | |
| 12:26:22 | jaypipes | oh, hey lyarwood :) | |
| 12:26:33 | lyarwood | \o_ morning | |
| 12:28:45 | ebbex | jaypipes: Yeah, I think it's kinda odd touching the file every minute, if the ImageCache tries something like _remove_old_enough*. I don't really understand how it's all supposed to hang together. | |
| 12:29:40 | jaypipes | ebbex: we've sent up the bat-signal for mdbooth :) hopefully he can share his insight on this (I'm afraid I'm not proficient enough in this area of the codebase) | |
| 12:30:41 | ebbex | :) | |
| 12:42:53 | jaypipes | stephenfin: I'd just go ahead and take over that InstancePCIRequest patch... ralonsoh, you cool with that? | |
| 12:47:01 | ralonsoh | jaypipes, stephenfin: but I've been taking care of my remaining patches. Anyway, if doing this we can have this 8 months patch merged, is ok | |
| 12:48:08 | openstackgerrit | Claudiu Belu proposed openstack/nova master: tests: autospecs all the mock.patch usages https://review.openstack.org/470775 | |
| 12:52:46 | jaypipes | ralonsoh: cool. it's just that stephenfin has been making some other changes around InstancePCIRequest object to support NUMA PCI affinity policy, so I thought it would be easier to have him take it over. | |
| 12:54:00 | jaypipes | gibi: are you planning on pushing a patch around that test_server_groups.py bug? | |
| 13:07:14 | gibi | jaypipes: yes, I'm working on that right now | |
| 13:07:26 | jaypipes | gibi: cool. | |
| 13:12:15 | openstackgerrit | Merged openstack/nova master: Deprecate configurable Hide Server Address Feature https://review.openstack.org/526297 | |
| 13:30:41 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix false positive server group functional tests https://review.openstack.org/529063 | |
| 13:30:45 | gibi | jaypipes: ^^ | |
| 13:30:55 | jaypipes | cool, thanks | |
| 13:34:39 | cdent | gibi: since you've spent a lot of time in the servers functional tests, can you recall how good the coverage is for the various migrations? I'm hoping that existing tests cover https://review.openstack.org/#/c/528089/ | |