| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-25 | |||
| 14:07:52 | mriedem | it's blank right? so why does the *image* cache manager worry about it? | |
| 14:08:04 | mdbooth | mriedem: It's not blank, it's formatted. | |
| 14:08:20 | mriedem | i mean there is no glance image in swap disks yeah? | |
| 14:08:20 | mdbooth | So it's basically a cache of the output of mkswap. | |
| 14:08:31 | mdbooth | No, there's no glance image. | |
| 14:09:01 | mdbooth | It saves a completely inconsequential amount of time during creation, and is likely a performance hit at runtime. | |
| 14:09:07 | mdbooth | And will save essentially no disk space. | |
| 14:09:45 | mdbooth | So we really shouldn't do it all, but cleaning that up would be a significant task. We'd need an upgrade task to rebase disks in use which already use this. | |
| 14:10:24 | mdbooth | However, aging these cached formatted swap files is really unlikely to be of much use to anybody. | |
| 14:10:40 | mdbooth | This user has found it's really expensive, so my suggestion is just don't age them. | |
| 14:11:37 | mdbooth | mriedem: Anyway, gimme a shout if you think ^^^ is an approach you could get behind and I'll fix that patch up properly. | |
| 14:13:09 | mriedem | i just don't really understand the use case for why these are in the image cache | |
| 14:13:22 | mriedem | granted, i'm not super familiar with the image cache manager to begin with | |
| 14:13:48 | mriedem | beyond it's used to cache and (on a timer) age/invalidate images in the cache so new versions can be fetched | |
| 14:14:06 | mriedem | but i thought that was only for root disks | |
| 14:14:14 | mdbooth | mriedem: I don't know the origin, so I can only guess. However ephemeral filesystem are also in there, and there are certain filesystems which *are* expensive to initialise. | |
| 14:14:24 | mdbooth | My guess is swap was simply handled the same way. | |
| 14:14:55 | mdbooth | Although to be clear, I would remove ephemeral filesystems too. We're already not aging them. | |
| 14:17:13 | mriedem | i don't see where ephemerals are in here | |
| 14:18:00 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix ComputeNode ovo compatibility code https://review.openstack.org/631199 | |
| 14:18:23 | mdbooth | ext3 vs ext4 fs creation comparison: http://paste.openstack.org/show/743462/ | |
| 14:19:05 | mdbooth | mriedem: I had to check closely, but I'm confident we don't age ephemeral backing files. | |
| 14:19:19 | mdbooth | reasonably confident :) That code is very hard to read. | |
| 14:20:10 | mdbooth | I believe ext3 was the default, might still be, and ext3 fs creation can be slow. | |
| 14:20:33 | mriedem | yeah i remember a ML thread from robert collins several years ago about changing the default to ext4 | |
| 14:20:41 | mdbooth | I suspect that's how this feature came about. But note that we still have to do this for other imagebackends anyway, and ext4 fs creation is really fast anyway. | |
| 14:22:27 | mriedem | looks like ext4 is the default now | |
| 14:22:30 | openstackgerrit | Merged openstack/nova master: Add missing ws seperator between words https://review.openstack.org/632606 | |
| 14:22:58 | mriedem | https://github.com/openstack/nova/blob/master/nova/privsep/fs.py#L258 | |
| 14:26:33 | kashyap | mdbooth: (Our of curiosity, I ran your same test on an NVMe: http://paste.openstack.org/show/743468/) | |
| 14:27:16 | mdbooth | kashyap: That's consistent, although I doubt either of our tests were actually hitting disk. | |
| 14:27:27 | mdbooth | kashyap: Likely due to RAM/CPU speed differences. | |
| 14:28:17 | kashyap | Oh, indeed. I always forget that point | |
| 14:28:42 | kashyap | (I recall that `fio` is the better way to do benchmarks) | |
| 14:29:03 | kashyap | (Reference: http://blog.vmsplice.net/2017/11/common-disk-benchmarking-mistakes.html) | |
| 14:33:23 | kashyap | mriedem: Completely unrelated, posted the initial investigation from last night here: https://bugs.launchpad.net/nova/+bug/1813192/comments/1 | |
| 14:33:25 | openstack | Launchpad bug 1813192 in OpenStack Compute (nova) "libvirt: instance delete fails with "Cannot destroy instance, operation time out: libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)" in bionic nodes (libvirt 4.0.0, qemu 2.11)" [Medium,Confirmed] | |
| 14:33:44 | kashyap | I did some more digging this morning for an hour, will write it up today. | |
| 14:34:37 | kashyap | The unsolved problem for me here is: how to get tracebacks ("thread all apply bt") from the hung QEMU (or libvirtd) from the CI hosts. | |
| 14:35:13 | kashyap | The only current awful way for that is to abuse 'rootwrap' to elevate privileges and then upload a debug build of libvirt/QEMU. | |
| 14:35:30 | kashyap | Then you can reproduce the problem, because "Heisenbug". | |
| 14:38:59 | mriedem | :/ | |
| 14:39:22 | mriedem | it's only 3 hits in 7 days in the gate so it's not burning us down like some of the other known bugs in the gate | |
| 14:44:15 | kashyap | Yeah, it's not. But I'd like to track down these gnarly bugs. Because if you recall, we had the same problem with the fated "live snapshot" bug | |
| 14:44:23 | kashyap | ... that made us reap the whirlind. | |
| 14:45:02 | kashyap | (Typo earlier: "Then you can" --> "Then you can't") | |
| 14:45:26 | kashyap | mriedem: Mind pointing me to the "known bugs" in the gate? | |
| 14:46:09 | mriedem | kashyap: http://status.openstack.org/elastic-recheck/ | |
| 14:46:17 | mriedem | http://status.openstack.org/elastic-recheck/#1813198 | |
| 14:46:18 | kashyap | Yep, remembered the URL | |
| 14:46:28 | mriedem | http://status.openstack.org/elastic-recheck/#1813147 | |
| 14:46:35 | mriedem | those are 2 big new ones | |
| 14:47:10 | kashyap | Thanks | |
| 14:53:42 | openstackgerrit | Ralf Haferkamp proposed openstack/nova master: compute: reset instance events on reload https://review.openstack.org/633220 | |
| 14:56:40 | mriedem | melwitt: do you expect to start on https://blueprints.launchpad.net/nova/+spec/count-quota-usage-from-placement soon? given you're out next week and we've got 6 weeks left i'm a bit worried about that one | |
| 15:02:42 | melwitt | mriedem: yeah, I hope to have something WIP uploaded by late today | |
| 15:30:34 | openstackgerrit | sean mooney proposed openstack/nova master: Libvirt: do not set mac when unplugging macvtap vf https://review.openstack.org/624842 | |
| 15:30:35 | openstackgerrit | sean mooney proposed openstack/nova master: Allow per-port modification of vnic_type and profile https://review.openstack.org/607365 | |
| 15:30:35 | openstackgerrit | sean mooney proposed openstack/nova master: Add free for claimed, allocated devices https://review.openstack.org/616120 | |
| 15:30:36 | openstackgerrit | sean mooney proposed openstack/nova master: SR-IOV Live migration indirect port support https://review.openstack.org/620115 | |
| 15:30:36 | openstackgerrit | sean mooney proposed openstack/nova master: Add get_instance_pci_request_from_vif https://review.openstack.org/619929 | |
| 15:30:37 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: auto detach/attach sriov ports on migration https://review.openstack.org/629589 | |
| 15:35:37 | openstackgerrit | Jan Gutter proposed openstack/nova master: Add support for vrouter HW datapath offloads https://review.openstack.org/572082 | |
| 15:35:37 | openstackgerrit | Jan Gutter proposed openstack/nova master: Convert vrouter legacy plugging to os-vif https://review.openstack.org/571325 | |
| 15:58:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Raise InstanceFaultRollback for UnableToMigrateToSelf from _prep_resize https://review.openstack.org/633227 | |
| 16:00:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Change InstanceFaultRollback handling in _error_out_instance_on_exception https://review.openstack.org/633212 | |
| 16:00:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Raise InstanceFaultRollback for UnableToMigrateToSelf from _prep_resize https://review.openstack.org/633227 | |
| 16:04:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Change InstanceFaultRollback handling in _error_out_instance_on_exception https://review.openstack.org/633212 | |
| 16:04:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Raise InstanceFaultRollback for UnableToMigrateToSelf from _prep_resize https://review.openstack.org/633227 | |
| 16:11:15 | melwitt | mriedem: I got confused by your comments on https://bugs.launchpad.net/nova/+bug/1811235 is it that cold migrate to same host is only valid for the vmware driver? and compute manager should be kicking it out unilaterally? | |
| 16:11:16 | openstack | Launchpad bug 1811235 in OpenStack Compute (nova) "instance's vm-state becomes error when cold-migrate instance to same host failed" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 16:11:30 | mriedem | stephenfin: i'll likely need you to run it up the flagpole with danpb about why libvirt.sysinfo_serial was never guest-specific | |
| 16:11:39 | mriedem | i don't have the history on that and it wasn't found in the original changes | |
| 16:11:57 | stephenfin | mriedem: Can do | |
| 16:12:02 | melwitt | s/should/shouldn't/ | |
| 16:12:30 | mriedem | the compute manager doesn't kick cold migrate to the same host out unilateraly, the api is | |
| 16:12:38 | mriedem | with that 2.56 change to specify host on cold migrate | |
| 16:12:47 | mriedem | that's a different issue from the bug though, which was reported against ocata | |
| 16:12:52 | mriedem | before 2.56 existed | |
| 16:13:41 | mriedem | i believe migrate_to_same_host exists for the vmware driver b/c you can have a single nova-compute service in your openstack deployment which is actually managing a vcenter cluster with 1000 ESXi hosts in it | |
| 16:13:47 | mriedem | so your only choice to cold migrate is to the same host | |
| 16:13:50 | melwitt | he says he thinks it exists in ocata too from reading the code. right? I'm having trouble parsing this entire bug :( | |
| 16:14:07 | mriedem | my patches above are working on fixing the bug he's hitting which exists in ocata | |
| 16:14:09 | mriedem | 2.56 does not | |
| 16:14:09 | melwitt | comment #2 | |
| 16:14:34 | mriedem | 2.56 was queens | |
| 16:14:55 | mriedem | comment 2 is saying 2 things | |
| 16:14:58 | mriedem | first is the bug i'm fixing above | |
| 16:15:03 | mriedem | second is the 2.56 issue i think | |
| 16:15:06 | mriedem | which is a different bug imo | |
| 16:15:14 | melwitt | yeah, I think he's not saying he's reporting it for ocata. "First, why the latest version of nova not supports migrating instance to same host event through some drivers support that function..Second, if there is any problem when user migrates instance without migration destination host but there is just one compute node." | |
| 16:15:57 | melwitt | eh, I dunno, I think I'm just confused by the wording of these comments | |
| 16:18:33 | mriedem | i left a comment to try and answer the questions from comment 2 | |
| 16:20:11 | melwitt | thanks. I think I see now, he's saying why go to error status when it "fails" because migrating to self doesn't really do anything | |
| 16:21:07 | melwitt | ok that comment makes it clear. good | |
| 16:21:13 | mriedem | correct, but the rejection in the API is not the same as the bug in the compute that sets the instance to error status | |
| 16:21:31 | mriedem | i've got fixes posted above for the latter, not the former | |
| 16:21:54 | melwitt | gotcha. I've got it starred for review | |
| 16:29:30 | mriedem | melwitt: if https://blueprints.launchpad.net/openstack/?searchtext=conf-max-attach-volumes is ready for review again you want to queue it up in runways? | |
| 16:29:46 | mriedem | jangutter's can probably go in the open slot as well | |