| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-19 | |||
| 02:48:30 | rybridges | ok | |
| 02:49:07 | mriedem | if that works, penick owes me a ginger ale in dublin | |
| 02:49:36 | mriedem | either way i'm hanging it up for the night | |
| 02:50:41 | rybridges | haha | |
| 02:50:46 | rybridges | i will be in ireland for ptg in feb | |
| 02:50:54 | rybridges | so ill get ou a ginger ale too | |
| 04:52:29 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Verify parameters in servers.inc https://review.openstack.org/528201 | |
| 04:52:34 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Verify parameters in servers.inc https://review.openstack.org/528201 | |
| 05:21:21 | openstackgerrit | Nakanishi Tomotaka proposed openstack/nova master: Use Placement API to check resource usage https://review.openstack.org/528953 | |
| 06:05:58 | openstackgerrit | Rajesh Tailor proposed openstack/nova master: Host addition host-aggregate should be case-sensitive https://review.openstack.org/498334 | |
| 06:06:33 | openstackgerrit | Rajesh Tailor proposed openstack/nova master: Fix case-sensitivity for metadata keys https://review.openstack.org/504885 | |
| 07:00:31 | openstackgerrit | jichenjc proposed openstack/nova master: Remove 'nova-manage shell' command https://review.openstack.org/521835 | |
| 07:00:32 | openstackgerrit | jichenjc proposed openstack/nova master: Remove 'nova-manage logs' command https://review.openstack.org/522133 | |
| 07:00:32 | openstackgerrit | jichenjc proposed openstack/nova master: Remove 'nova-manage account' and 'nova-manage project' https://review.openstack.org/521833 | |
| 07:12:56 | Guest72028 | Hello All | |
| 07:19:17 | Guest72028 | I have a questions for nova specialists about nova ressources: is it possible to have compute nodes spares ? is it possible to reserve compute ressources , prioritize rebuild order of VM while evacuation process ? | |
| 08:25:12 | openstackgerrit | 龚肖 proposed openstack/nova stable/pike: compute: Catch binding failed exception while init host https://review.openstack.org/528985 | |
| 08:35:02 | openstackgerrit | TommyLike proposed openstack/nova master: Remove redundant try/except block when authorize https://review.openstack.org/528991 | |
| 08:35:38 | openstackgerrit | Mr Rambo proposed openstack/nova master: Fix the problems that volume-backed server rebuild https://review.openstack.org/528994 | |
| 09:05:26 | lyarwood | mdbooth: thanks for the reviews yesterday btw, is your uuid series ready for review? | |
| 09:07:00 | mdbooth | lyarwood: Mostly, yes. | |
| 09:07:17 | mdbooth | lyarwood: Well, most of it | |
| 09:08:22 | mdbooth | lyarwood: I'd very much like your input on this one: https://review.openstack.org/#/c/528363/ | |
| 09:08:41 | mdbooth | But I'm about to rebase that into the main series, because I need it for the next patch | |
| 09:09:33 | mdbooth | The series is here: https://review.openstack.org/#/q/topic:bp/local-disk-serial-numbers+(status:open+OR+status:merged) | |
| 09:10:36 | lyarwood | mdbooth: ack, looking | |
| 09:13:27 | mdbooth | lyarwood: Thanks | |
| 09:37:46 | openstackgerrit | Mr Rambo proposed openstack/nova master: Fix the problems that volume-backed server rebuild https://review.openstack.org/528740 | |
| 09:41:28 | maciejjozefczyk | gibi jaypipes: Hello, thanks for your comments :) Please check my response https://review.openstack.org/#/c/520024/ Thank you! | |
| 10:02:49 | lyarwood | mdbooth: so the DriverVolumeBlockDevice change LGTM, could we set self.connection_info earlier and just pass that around within the method? | |
| 10:05:03 | mdbooth | lyarwood: Which function are you referring to specifically? | |
| 10:05:51 | lyarwood | mdbooth: _legacy_volume_attach or _volume_attach in block_device.py | |
| 10:06:28 | mdbooth | Sec, just reading and digesting your review comment | |
| 10:07:07 | mdbooth | lyarwood: Ah, you're talking about *not* changing the interface? | |
| 10:07:17 | mdbooth | And continuing to pass connection info explicitly? | |
| 10:07:46 | lyarwood | mdbooth: no, just setting self.connection_info when we actually fetch it from cinder | |
| 10:07:56 | mdbooth | Ah... | |
| 10:08:45 | mdbooth | Yep, that would be cleaner. | |
| 10:08:54 | mdbooth | It would make the patch a bit noisier, though... | |
| 10:08:59 | mdbooth | And it's already pretty noisy. | |
| 10:09:01 | mdbooth | Hmm... | |
| 10:09:06 | lyarwood | yeah I assumed that's why you didn't do it tbh | |
| 10:16:06 | norman | hi all, anyone know why domainxml of the live-migrated instance has <features> under <cpu> section, but new booted instances not | |
| 10:17:04 | norman | I'd trying go through the code ,failed to find clues. I am still using the Mitaka, not sure the new version is Ok or not | |
| 10:17:56 | Tahvok | Hey guys, I'm unable to find a good example of ComputeCapabilitiesFilter. Do you apply it on the flavor's metadata or on the compute host somehow? If it should me on the compute host, so where exactly do I specify my 'capabilities' filter? | |
| 10:28:25 | openstackgerrit | Merged openstack/nova stable/ocata: Make request_spec.spec MediumText https://review.openstack.org/528332 | |
| 10:28:31 | openstackgerrit | Merged openstack/nova master: Fix the formatting for 2.56 in the compute REST API history doc https://review.openstack.org/528114 | |
| 10:41:02 | lyarwood | kashyap: http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/logs/subnode-2/screen-n-cpu.txt.gz#_2017-12-18_20_20_13_894 - seeing this on stable/newton, LM failure for a single paused instance, looks like the remote libvirtd didn't respond in time, have you seen this before? | |
| 10:41:09 | lyarwood | kashyap: http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/job-output.txt.gz#_2017-12-18_20_20_56_230121 is the tempest failure | |
| 10:42:04 | kashyap | lyarwood: Is this only stable/newton? | |
| 10:42:29 | lyarwood | kashyap: that's the only place I've seen it thus far | |
| 10:50:20 | kashyap | lyarwood: So I looked at all the logs, one thing that potentially jumps out at me is in the source QEMU log: | |
| 10:50:23 | kashyap | [...] | |
| 10:50:24 | kashyap | [...] | |
| 10:50:27 | kashyap | warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] | |
| 10:50:30 | kashyap | main-loop: WARNING: I/O thread spun for 1000 iterations | |
| 10:50:52 | kashyap | Now, that warning isn't really an egregious error (& upstream QEMU is aware of it; it's a hard thing to fix), but that might be contributing to it | |
| 10:51:04 | kashyap | Where "it" being the timeout we see: | |
| 10:51:05 | kashyap | 2017-12-18 20:20:13.880+0000: 16820: error : virKeepAliveTimerInternal:143 : internal error: connection closed due to keepalive timeout | |
| 10:51:08 | kashyap | 2017-12-18 20:20:13.881+0000: 16825: error : virKeepAliveTimerInternal:143 : internal error: connection closed due to keepalive timeout | |
| 10:51:11 | kashyap | (From the source) | |
| 10:52:10 | lyarwood | kashyap: kk, so the dest was stuck, didn't send a keepalive and everything dies? | |
| 10:52:32 | kashyap | lyarwood: Yeah, from the destination livirtd log: | |
| 10:52:35 | kashyap | 2017-12-18 20:20:19.394+0000: 23816: error : qemuMonitorIOWrite:545 : Unable to write to monitor: Broken pipe | |
| 10:53:01 | kashyap | The above means libvirt lost access to the QMP socket connection, i.e. VM died | |
| 10:53:39 | lyarwood | kashyap: it's paused, that shouldn't cause the QMP socket to die however right? | |
| 10:55:27 | kashyap | lyarwood: Yeah, clearly something is wonky. I'll check this w/ Dave Gilbert as he meditates on migration | |
| 10:55:53 | kashyap | But the process _is_ killed, as we see from the destination (http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/logs/subnode-2/libvirt/qemu/instance-00000004.txt.gz): | |
| 10:55:58 | kashyap | 2017-12-18T20:17:51.612132Z qemu-system-x86_64: terminating on signal 15 from pid 16820 | |
| 10:58:06 | openstackgerrit | Chen Hanxiao proposed openstack/nova master: libvirt: don't call sync_guest_time if qga is not enabled https://review.openstack.org/524836 | |
| 10:58:26 | lyarwood | kashyap: kk, thanks, I'm going to recheck this change and see if we can hit it again | |
| 10:58:37 | kashyap | So it is the live block migration, right | |
| 10:58:43 | kashyap | (To pause it) | |
| 10:58:47 | kashyap | LiveMigrationTest.test_live_block_migration_paused | |
| 10:59:10 | lyarwood | kashyap: yup, LM of a paused instance without shared storage | |
| 11:01:22 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | |
| 11:01:23 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | |
| 11:03:00 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | |
| 11:03:01 | openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | |
| 11:13:27 | openstackgerrit | Merged openstack/nova master: Implement query param schema for migration index https://review.openstack.org/518644 | |
| 11:14:16 | kashyap | lyarwood: So a couple of things from interacting w/ Dan & Dave from QEMU: | |
| 11:14:54 | kashyap | (1) You see the _later_ log messages (on destination) are 3 before the earlier log message. | |
| 11:15:23 | kashyap | So that's some weird timestamps there. | |
| 11:17:08 | openstack | bugzilla.redhat.com bug 1367620 in libvirt "storage migration fails due to keepalive timeout" [High,Assigned] - Assigned to jdenemar | |
| 11:17:08 | kashyap | (2) We're not the first to hit this case; there's this existing bug https://bugzilla.redhat.com/show_bug.cgi?id=1367620 ("storage migration fails due to keepalive timeout") | |
| 11:18:59 | openstackgerrit | Matthew Booth proposed openstack/nova master: Expose BDM uuid to drivers https://review.openstack.org/529037 | |
| 11:19:24 | kashyap | lyarwood: Wonder if you have a link to how many times this was hit in the past / this week? | |
| 11:21:52 | lyarwood | kashyap: I don't have one to hand now but I can create one, and an upstream bug for this | |
| 11:22:04 | mdbooth | lyarwood: Don't know if you're still looking at it, but I've been messing with that series this morning. | |
| 11:22:06 | lyarwood | assuming we've seen it more than once | |
| 11:22:09 | mdbooth | Not quite finished yet. | |
| 11:22:32 | lyarwood | mdbooth: kk I stopped after the earlier change sorry | |
| 11:23:00 | mdbooth | lyarwood: NP. I'd have been messing you about anyway. | |
| 11:23:13 | kashyap | lyarwood: I'm creating a potential reproducer, and can file one with that | |
| 11:23:23 | kashyap | lyarwood: But if you've already drafted a bug / issue, go ahead & submit it | |
| 11:27:58 | openstackgerrit | Merged openstack/nova master: Remove 'nova-manage shell' command https://review.openstack.org/521835 | |
| 11:32:15 | openstackgerrit | Matthew Booth proposed openstack/nova master: Give volume DriverBlockDevice classes a common prefix https://review.openstack.org/526346 | |
| 11:32:16 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347 | |
| 11:34:34 | openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | |
| 11:34:34 | openstackgerrit | Matthew Booth proposed openstack/nova master: Use real block_device_info data in test_blockinfo https://review.openstack.org/527916 | |