Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-04
16:00:53 mdbooth Multi-attach doesn't even make the problem particularly worse.
16:01:35 ildikov mdbooth: well, I got it in referring to multi-attach, but I guess the path doesn't matter once you got where you wanted... :)
16:01:54 efried jaypipes mriedem cdent alex_xu Draft: http://paste.openstack.org/show/638080/ -- anything missing/incorrect/silly?
16:02:26 cdent efried: will look in a mo, thanks for doing that
16:02:36 mdbooth ildikov: Hehe, I hear you :)
16:04:39 mriedem mdbooth: is it just me or is libvirt.images_type, use_cow_images and force_raw_images set of options totally confusing?
16:04:40 stephenfin alex_xu: Done (https://review.openstack.org/#/c/530284/(
16:18:28 efried mriedem Log processing & coloring is handled in the openstack-infra/os-loganalyze project (but I think you knew that). Is there something in particular you're having trouble finding in there?
16:19:23 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Deprecate 'network_manager' https://review.openstack.org/530923
16:19:33 mriedem efried: what in infra actually calls os-loganalyze to format the logs
16:20:16 openstackgerrit Stephen Finucane proposed openstack/nova master: conf: Use new-style choice values https://review.openstack.org/530924
16:20:40 efried mriedem Ah - it's an apache plugin thingy. The files aren't actually modified - they get twiddled on the fly when you do your http request.
16:21:18 efried mriedem So you just need to muck with the filters to make sure they're being run on the files you're interested in.
16:22:21 efried mriedem If you have access to a log server, you can play by fiddling with the os-loganalyze source in place, restarting the apache server, and then reloading your browser.
16:27:51 mriedem i don't
16:44:52 cdent efried: information seems accurate at the detail level, but feels like it needs some kind of executive summary or something, a kind of "here's what we're trying to accomplish" which is then followed by the "and this is how it is being done" (which is what you've already got)
16:45:45 efried cdent Okay. I felt like it was already getting kinda long, but... tough :)
16:46:21 cdent I think length is useful in this case because there's been not enough in the way of spec, so this kind of stands in for that
16:50:17 clarkb mriedem: the test framework for it runs without apache
16:52:23 openstackgerrit Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094
16:53:53 hrw https://marcin.juszkiewicz.com.pl/2018/01/04/today-i-was-fighting-with-nova-no-idea-who-won/
16:53:57 hrw kashyap: ^^
16:54:15 kashyap hrw: On a call, will read :-)
16:54:36 hrw kashyap: thx ;)
16:54:43 kashyap Damn, I already clicked
16:58:49 hrw kashyap: :D
16:58:50 kashyap hrw: It looks fine; there's an extra tab in there
16:59:00 kashyap "It" == the review
16:59:31 rybridges Hey guys, can anyone confirm for sure whether or not ephemeral GB is part of a VMs snapshot?
17:00:07 hrw kashyap: can you mark it in comment? I am unable to find it locally
17:00:19 kashyap hrw: Done.
17:00:22 kashyap It's extra tab
17:00:30 kashyap In the rel note file
17:00:55 hrw rught
17:01:04 hrw 4 spaces to be exact ;d
17:01:09 hrw that's why I did not ofund
17:01:37 openstackgerrit Marcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64 https://review.openstack.org/530965
17:01:50 hrw done
17:03:43 efried cdent (jaypipes) Howzat: http://paste.openstack.org/raw/638137/
17:21:12 mnaser is stable/pike ci broken?
17:21:16 mriedem yes
17:21:40 mriedem https://review.openstack.org/#/c/531058/
17:21:55 lyarwood also https://review.openstack.org/#/c/531046/
17:23:02 mnaser ok, i guess i'll apply the patch locally till it lands in stable/pike (https://review.openstack.org/#/c/529384/)
17:24:26 edleafe mriedem: so for the migration bug: would the fix be to just log that there were no orig_alloc found, and not raise the exception?
17:24:51 mriedem edleafe: well, there are really 2 fixes,
17:24:58 edleafe Or should I also check if the migration is the orig_alloc (as in retries)
17:25:05 mriedem 1. If using the CachingScheduler, there won't be allocations and we need to just log something and ignore it
17:25:20 mnaser also if any stable cores for nova are around, this is pretty useful - https://review.openstack.org/#/c/529385/
17:25:49 mriedem 2. If we're rescheduling (we should know this via filter_properties 'retry' entry), then we need to modify how we swap allocations to only change the allocation for the instance record, and leave the migration allocation on the source node untouched
17:26:31 mriedem edleafe: so i'm thinking 2 separate patches
17:27:20 mriedem edleafe: i could probably wip up a simple regression test for the caching scheduler one and fix in the same patch, which could go below yours
17:27:30 mriedem since resize is just busted with caching scheduler regardless of reschedule
17:27:43 edleafe mriedem: on #2, not following. The source allocs will be the migration, and the dest allocs won't be done because we haven't picked a target host yet
17:28:13 mriedem edleafe: on #2 the problem is when we reschedule right?
17:28:17 edleafe yeah
17:28:33 edleafe it's checking that the instance is allocated against the source
17:28:35 mriedem at the point of the first reschedule, the source node allocs are on the migration record and the failed first chosen host allocs are on the instance
17:28:41 mriedem right, i'm saying,
17:28:47 edleafe on a retry, the allocs on the source will be the migration
17:28:49 mriedem we need to modify the logic that assumes the source node allocs are on the instance,
17:28:55 mriedem if we know we're doing a reschedule,
17:29:01 mriedem and we can determine that based on the filter_properties 'retry' entry
17:29:19 mriedem so on a retry, we don't do anything with the source,
17:29:22 mriedem we just update the dest
17:29:36 mriedem update the instance allocation from failed dest 1 to next dest 2
17:29:39 edleafe mriedem: what did you mean then by "only change the allocation for the instance record"?
17:30:06 mriedem on a reschedule, we move the instance allocation from the failed dest host to the next alternative host
17:30:12 mriedem and don't touch the migration allocation on the source host
17:30:18 edleafe the instance won't be allocated after a fail. Those should be rolled back, no?
17:30:49 edleafe no, the next host isn't selected until after this part of the code is run
17:31:05 edleafe This is happening in _preallocate_migration
17:32:05 mriedem where do we rollback allocations for the instance on a fail?
17:32:08 edleafe When we eventually select a host (either through select_destinations (now) or alternate host (when that last patch merges), that's when we allocate the instance to the target
17:32:40 edleafe mriedem: in compute. Let me look
17:33:36 mriedem delete_allocation_for_failed_resize in prep_resize?
17:33:52 mriedem oh i see, the call to _revert_allocation
17:34:14 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4095
17:34:46 edleafe damn, you're faster than I am :)
17:35:08 mriedem so if the compute reverts the allocations before casting back to conductor to reschedule,
17:35:14 mriedem why is conductor failing on the reschedule
17:35:15 mriedem ?
17:35:34 edleafe Because it is confirming that the instance has allocations on the source
17:35:55 mriedem yeah, which should be the case if we got here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4000
17:35:55 edleafe But at that point on a retry, it's the migration that has allocations on the source
17:36:00 mriedem not if we did ^
17:36:38 mriedem is that failing?
17:37:37 edleafe dunno - all I know is that the instance didn't have allocations on the source. Let me check that out.
17:43:58 mriedem i've created https://bugs.launchpad.net/nova/+bug/1741307 to deal with the caching scheduler + resize stuff
17:43:59 openstack Launchpad bug 1741307 in OpenStack Compute (nova) "Resize always fails when using the CachingScheduler" [High,Triaged] - Assigned to Matt Riedemann (mriedem)
17:45:12 mriedem kashyap: fyi that i updated the devstack patch to test multiattach to set CONF.libvirt.images_type=raw and CONF.use_cow_images=False to see if that makes a difference
17:48:57 openstackgerrit Merged openstack/nova master: Revert "Modify _poll_shelved_instances periodic task call _shelve_offload_instance()" https://review.openstack.org/530284
18:19:19 mriedem wow, if i'm reading his results correctly, Kevin_Zheng got a 90% improvement in listing instances with details using an ip filter after we proxy to neutron first
18:19:25 mriedem sdague: ^
18:19:40 mriedem 2000 instances in nova and 2K ports in neutron
18:21:04 mriedem told him to send the details to the dev list
18:22:27 openstackgerrit Marcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64 https://review.openstack.org/530965
18:27:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Add regression test for resizing failing when using CachingScheduler https://review.openstack.org/531211
18:27:18 mriedem edleafe: ^ functional regression test for resize + caching scheduler failure
18:29:05 edleafe mriedem: thx

Earlier   Later