Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-01-25
14:09:01 mdbooth It saves a completely inconsequential amount of time during creation, and is likely a performance hit at runtime.
14:09:07 mdbooth And will save essentially no disk space.
14:09:45 mdbooth So we really shouldn't do it all, but cleaning that up would be a significant task. We'd need an upgrade task to rebase disks in use which already use this.
14:10:24 mdbooth However, aging these cached formatted swap files is really unlikely to be of much use to anybody.
14:10:40 mdbooth This user has found it's really expensive, so my suggestion is just don't age them.
14:11:37 mdbooth mriedem: Anyway, gimme a shout if you think ^^^ is an approach you could get behind and I'll fix that patch up properly.
14:13:09 mriedem i just don't really understand the use case for why these are in the image cache
14:13:22 mriedem granted, i'm not super familiar with the image cache manager to begin with
14:13:48 mriedem beyond it's used to cache and (on a timer) age/invalidate images in the cache so new versions can be fetched
14:14:06 mriedem but i thought that was only for root disks
14:14:14 mdbooth mriedem: I don't know the origin, so I can only guess. However ephemeral filesystem are also in there, and there are certain filesystems which *are* expensive to initialise.
14:14:24 mdbooth My guess is swap was simply handled the same way.
14:14:55 mdbooth Although to be clear, I would remove ephemeral filesystems too. We're already not aging them.
14:17:13 mriedem i don't see where ephemerals are in here
14:18:00 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fix ComputeNode ovo compatibility code https://review.openstack.org/631199
14:18:23 mdbooth ext3 vs ext4 fs creation comparison: http://paste.openstack.org/show/743462/
14:19:05 mdbooth mriedem: I had to check closely, but I'm confident we don't age ephemeral backing files.
14:19:19 mdbooth reasonably confident :) That code is very hard to read.
14:20:10 mdbooth I believe ext3 was the default, might still be, and ext3 fs creation can be slow.
14:20:33 mriedem yeah i remember a ML thread from robert collins several years ago about changing the default to ext4
14:20:41 mdbooth I suspect that's how this feature came about. But note that we still have to do this for other imagebackends anyway, and ext4 fs creation is really fast anyway.
14:22:27 mriedem looks like ext4 is the default now
14:22:30 openstackgerrit Merged openstack/nova master: Add missing ws seperator between words https://review.openstack.org/632606
14:22:58 mriedem https://github.com/openstack/nova/blob/master/nova/privsep/fs.py#L258
14:26:33 kashyap mdbooth: (Our of curiosity, I ran your same test on an NVMe: http://paste.openstack.org/show/743468/)
14:27:16 mdbooth kashyap: That's consistent, although I doubt either of our tests were actually hitting disk.
14:27:27 mdbooth kashyap: Likely due to RAM/CPU speed differences.
14:28:17 kashyap Oh, indeed. I always forget that point
14:28:42 kashyap (I recall that `fio` is the better way to do benchmarks)
14:29:03 kashyap (Reference: http://blog.vmsplice.net/2017/11/common-disk-benchmarking-mistakes.html)
14:33:23 kashyap mriedem: Completely unrelated, posted the initial investigation from last night here: https://bugs.launchpad.net/nova/+bug/1813192/comments/1
14:33:25 openstack Launchpad bug 1813192 in OpenStack Compute (nova) "libvirt: instance delete fails with "Cannot destroy instance, operation time out: libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)" in bionic nodes (libvirt 4.0.0, qemu 2.11)" [Medium,Confirmed]
14:33:44 kashyap I did some more digging this morning for an hour, will write it up today.
14:34:37 kashyap The unsolved problem for me here is: how to get tracebacks ("thread all apply bt") from the hung QEMU (or libvirtd) from the CI hosts.
14:35:13 kashyap The only current awful way for that is to abuse 'rootwrap' to elevate privileges and then upload a debug build of libvirt/QEMU.
14:35:30 kashyap Then you can reproduce the problem, because "Heisenbug".
14:38:59 mriedem :/
14:39:22 mriedem it's only 3 hits in 7 days in the gate so it's not burning us down like some of the other known bugs in the gate
14:44:15 kashyap Yeah, it's not. But I'd like to track down these gnarly bugs. Because if you recall, we had the same problem with the fated "live snapshot" bug
14:44:23 kashyap ... that made us reap the whirlind.
14:45:02 kashyap (Typo earlier: "Then you can" --> "Then you can't")
14:45:26 kashyap mriedem: Mind pointing me to the "known bugs" in the gate?
14:46:09 mriedem kashyap: http://status.openstack.org/elastic-recheck/
14:46:17 mriedem http://status.openstack.org/elastic-recheck/#1813198
14:46:18 kashyap Yep, remembered the URL
14:46:28 mriedem http://status.openstack.org/elastic-recheck/#1813147
14:46:35 mriedem those are 2 big new ones
14:47:10 kashyap Thanks
14:53:42 openstackgerrit Ralf Haferkamp proposed openstack/nova master: compute: reset instance events on reload https://review.openstack.org/633220
14:56:40 mriedem melwitt: do you expect to start on https://blueprints.launchpad.net/nova/+spec/count-quota-usage-from-placement soon? given you're out next week and we've got 6 weeks left i'm a bit worried about that one
15:02:42 melwitt mriedem: yeah, I hope to have something WIP uploaded by late today
15:30:34 openstackgerrit sean mooney proposed openstack/nova master: Libvirt: do not set mac when unplugging macvtap vf https://review.openstack.org/624842
15:30:35 openstackgerrit sean mooney proposed openstack/nova master: Add free for claimed, allocated devices https://review.openstack.org/616120
15:30:35 openstackgerrit sean mooney proposed openstack/nova master: Allow per-port modification of vnic_type and profile https://review.openstack.org/607365
15:30:36 openstackgerrit sean mooney proposed openstack/nova master: Add get_instance_pci_request_from_vif https://review.openstack.org/619929
15:30:36 openstackgerrit sean mooney proposed openstack/nova master: SR-IOV Live migration indirect port support https://review.openstack.org/620115
15:30:37 openstackgerrit sean mooney proposed openstack/nova master: libvirt: auto detach/attach sriov ports on migration https://review.openstack.org/629589
15:35:37 openstackgerrit Jan Gutter proposed openstack/nova master: Convert vrouter legacy plugging to os-vif https://review.openstack.org/571325
15:35:37 openstackgerrit Jan Gutter proposed openstack/nova master: Add support for vrouter HW datapath offloads https://review.openstack.org/572082
15:58:32 openstackgerrit Matt Riedemann proposed openstack/nova master: Raise InstanceFaultRollback for UnableToMigrateToSelf from _prep_resize https://review.openstack.org/633227
16:00:15 openstackgerrit Matt Riedemann proposed openstack/nova master: Change InstanceFaultRollback handling in _error_out_instance_on_exception https://review.openstack.org/633212
16:00:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Raise InstanceFaultRollback for UnableToMigrateToSelf from _prep_resize https://review.openstack.org/633227
16:04:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Change InstanceFaultRollback handling in _error_out_instance_on_exception https://review.openstack.org/633212
16:04:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Raise InstanceFaultRollback for UnableToMigrateToSelf from _prep_resize https://review.openstack.org/633227
16:11:15 melwitt mriedem: I got confused by your comments on https://bugs.launchpad.net/nova/+bug/1811235 is it that cold migrate to same host is only valid for the vmware driver? and compute manager should be kicking it out unilaterally?
16:11:16 openstack Launchpad bug 1811235 in OpenStack Compute (nova) "instance's vm-state becomes error when cold-migrate instance to same host failed" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
16:11:30 mriedem stephenfin: i'll likely need you to run it up the flagpole with danpb about why libvirt.sysinfo_serial was never guest-specific
16:11:39 mriedem i don't have the history on that and it wasn't found in the original changes
16:11:57 stephenfin mriedem: Can do
16:12:02 melwitt s/should/shouldn't/
16:12:30 mriedem the compute manager doesn't kick cold migrate to the same host out unilateraly, the api is
16:12:38 mriedem with that 2.56 change to specify host on cold migrate
16:12:47 mriedem that's a different issue from the bug though, which was reported against ocata
16:12:52 mriedem before 2.56 existed
16:13:41 mriedem i believe migrate_to_same_host exists for the vmware driver b/c you can have a single nova-compute service in your openstack deployment which is actually managing a vcenter cluster with 1000 ESXi hosts in it
16:13:47 mriedem so your only choice to cold migrate is to the same host
16:13:50 melwitt he says he thinks it exists in ocata too from reading the code. right? I'm having trouble parsing this entire bug :(
16:14:07 mriedem my patches above are working on fixing the bug he's hitting which exists in ocata
16:14:09 melwitt comment #2
16:14:09 mriedem 2.56 does not
16:14:34 mriedem 2.56 was queens
16:14:55 mriedem comment 2 is saying 2 things
16:14:58 mriedem first is the bug i'm fixing above
16:15:03 mriedem second is the 2.56 issue i think
16:15:06 mriedem which is a different bug imo
16:15:14 melwitt yeah, I think he's not saying he's reporting it for ocata. "First, why the latest version of nova not supports migrating instance to same host event through some drivers support that function..Second, if there is any problem when user migrates instance without migration destination host but there is just one compute node."
16:15:57 melwitt eh, I dunno, I think I'm just confused by the wording of these comments
16:18:33 mriedem i left a comment to try and answer the questions from comment 2
16:20:11 melwitt thanks. I think I see now, he's saying why go to error status when it "fails" because migrating to self doesn't really do anything
16:21:07 melwitt ok that comment makes it clear. good
16:21:13 mriedem correct, but the rejection in the API is not the same as the bug in the compute that sets the instance to error status
16:21:31 mriedem i've got fixes posted above for the latter, not the former
16:21:54 melwitt gotcha. I've got it starred for review
16:29:30 mriedem melwitt: if https://blueprints.launchpad.net/openstack/?searchtext=conf-max-attach-volumes is ready for review again you want to queue it up in runways?
16:29:46 mriedem jangutter's can probably go in the open slot as well
16:30:10 melwitt mriedem: sure. I don't know why I didn't think of doing that
16:34:51 jangutter melwitt, mriedem: thanks!
16:35:41 mriedem pretty easy fix here for someone https://review.openstack.org/#/c/624959/
16:35:46 mriedem needs to be backported to queens as well
16:39:49 melwitt jangutter: the bp dependencies show that vrouter-hw-offloads also depends on the generic-os-vif-offloads too. should I add that patch link in there too for the runway review?

Earlier   Later