| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-04 | |||
| 15:56:35 | mriedem | another thing i can try | |
| 15:57:03 | ildikov | mdbooth: the service_uuid field was added here: https://review.openstack.org/#/c/519025/ | |
| 15:57:41 | mdbooth | ildikov: Just found it. I think it's orthogonal to multi-attach, tbh. | |
| 15:57:51 | mriedem | kashyap: although it's a bit confusing, | |
| 15:57:57 | mriedem | we dump the disk config before trying to attach the device | |
| 15:58:00 | mriedem | http://paste.openstack.org/show/638081/ | |
| 15:58:00 | mriedem | and it says type="raw" | |
| 15:58:51 | mdbooth | ildikov: Still a good idea. I could also remove my NFS locking stuff in Nova if we had that, although my NFS locking is finer grained but significantly more complex. | |
| 15:59:09 | ildikov | mdbooth: it was added along with a shared_targets field so we can use a lock in case the target exported by the back end is shared among volumes/attachments | |
| 15:59:35 | mdbooth | ildikov: I get it. It's a good idea, I just don't see the relationship to multi-attach. | |
| 15:59:51 | kashyap | mriedem: (Aside - we both wrote almost same comment 4 mins apart) | |
| 15:59:54 | ildikov | mdbooth: it is supposed to help to solve our detach problems | |
| 15:59:59 | kashyap | mriedem: Looking at your paste | |
| 16:00:00 | mdbooth | We hit this with or without multi-attach. | |
| 16:00:17 | ildikov | mdbooth: as if the target is shared and gets removed with the first attachment then the remaining attachments are screwed | |
| 16:00:30 | mdbooth | ildikov: Right, but you can can do that without multi-attach. | |
| 16:00:52 | ildikov | mdbooth: and the Nova patches using it are dependencies to multi-attach as it might be a problem otherwise as well | |
| 16:00:53 | mdbooth | Multi-attach doesn't even make the problem particularly worse. | |
| 16:01:35 | ildikov | mdbooth: well, I got it in referring to multi-attach, but I guess the path doesn't matter once you got where you wanted... :) | |
| 16:01:54 | efried | jaypipes mriedem cdent alex_xu Draft: http://paste.openstack.org/show/638080/ -- anything missing/incorrect/silly? | |
| 16:02:26 | cdent | efried: will look in a mo, thanks for doing that | |
| 16:02:36 | mdbooth | ildikov: Hehe, I hear you :) | |
| 16:04:39 | mriedem | mdbooth: is it just me or is libvirt.images_type, use_cow_images and force_raw_images set of options totally confusing? | |
| 16:04:40 | stephenfin | alex_xu: Done (https://review.openstack.org/#/c/530284/( | |
| 16:18:28 | efried | mriedem Log processing & coloring is handled in the openstack-infra/os-loganalyze project (but I think you knew that). Is there something in particular you're having trouble finding in there? | |
| 16:19:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Deprecate 'network_manager' https://review.openstack.org/530923 | |
| 16:19:33 | mriedem | efried: what in infra actually calls os-loganalyze to format the logs | |
| 16:20:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Use new-style choice values https://review.openstack.org/530924 | |
| 16:20:40 | efried | mriedem Ah - it's an apache plugin thingy. The files aren't actually modified - they get twiddled on the fly when you do your http request. | |
| 16:21:18 | efried | mriedem So you just need to muck with the filters to make sure they're being run on the files you're interested in. | |
| 16:22:21 | efried | mriedem If you have access to a log server, you can play by fiddling with the os-loganalyze source in place, restarting the apache server, and then reloading your browser. | |
| 16:27:51 | mriedem | i don't | |
| 16:44:52 | cdent | efried: information seems accurate at the detail level, but feels like it needs some kind of executive summary or something, a kind of "here's what we're trying to accomplish" which is then followed by the "and this is how it is being done" (which is what you've already got) | |
| 16:45:45 | efried | cdent Okay. I felt like it was already getting kinda long, but... tough :) | |
| 16:46:21 | cdent | I think length is useful in this case because there's been not enough in the way of spec, so this kind of stands in for that | |
| 16:50:17 | clarkb | mriedem: the test framework for it runs without apache | |
| 16:52:23 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 16:53:53 | hrw | https://marcin.juszkiewicz.com.pl/2018/01/04/today-i-was-fighting-with-nova-no-idea-who-won/ | |
| 16:53:57 | hrw | kashyap: ^^ | |
| 16:54:15 | kashyap | hrw: On a call, will read :-) | |
| 16:54:36 | hrw | kashyap: thx ;) | |
| 16:54:43 | kashyap | Damn, I already clicked | |
| 16:58:49 | hrw | kashyap: :D | |
| 16:58:50 | kashyap | hrw: It looks fine; there's an extra tab in there | |
| 16:59:00 | kashyap | "It" == the review | |
| 16:59:31 | rybridges | Hey guys, can anyone confirm for sure whether or not ephemeral GB is part of a VMs snapshot? | |
| 17:00:07 | hrw | kashyap: can you mark it in comment? I am unable to find it locally | |
| 17:00:19 | kashyap | hrw: Done. | |
| 17:00:22 | kashyap | It's extra tab | |
| 17:00:30 | kashyap | In the rel note file | |
| 17:00:55 | hrw | rught | |
| 17:01:04 | hrw | 4 spaces to be exact ;d | |
| 17:01:09 | hrw | that's why I did not ofund | |
| 17:01:37 | openstackgerrit | Marcin Juszkiewicz proposed openstack/nova master: libvirt: use 'host-passthrough' as default on AArch64 https://review.openstack.org/530965 | |
| 17:01:50 | hrw | done | |
| 17:03:43 | efried | cdent (jaypipes) Howzat: http://paste.openstack.org/raw/638137/ | |
| 17:21:12 | mnaser | is stable/pike ci broken? | |
| 17:21:16 | mriedem | yes | |
| 17:21:40 | mriedem | https://review.openstack.org/#/c/531058/ | |
| 17:21:55 | lyarwood | also https://review.openstack.org/#/c/531046/ | |
| 17:23:02 | mnaser | ok, i guess i'll apply the patch locally till it lands in stable/pike (https://review.openstack.org/#/c/529384/) | |
| 17:24:26 | edleafe | mriedem: so for the migration bug: would the fix be to just log that there were no orig_alloc found, and not raise the exception? | |
| 17:24:51 | mriedem | edleafe: well, there are really 2 fixes, | |
| 17:24:58 | edleafe | Or should I also check if the migration is the orig_alloc (as in retries) | |
| 17:25:05 | mriedem | 1. If using the CachingScheduler, there won't be allocations and we need to just log something and ignore it | |
| 17:25:20 | mnaser | also if any stable cores for nova are around, this is pretty useful - https://review.openstack.org/#/c/529385/ | |
| 17:25:49 | mriedem | 2. If we're rescheduling (we should know this via filter_properties 'retry' entry), then we need to modify how we swap allocations to only change the allocation for the instance record, and leave the migration allocation on the source node untouched | |
| 17:26:31 | mriedem | edleafe: so i'm thinking 2 separate patches | |
| 17:27:20 | mriedem | edleafe: i could probably wip up a simple regression test for the caching scheduler one and fix in the same patch, which could go below yours | |
| 17:27:30 | mriedem | since resize is just busted with caching scheduler regardless of reschedule | |
| 17:27:43 | edleafe | mriedem: on #2, not following. The source allocs will be the migration, and the dest allocs won't be done because we haven't picked a target host yet | |
| 17:28:13 | mriedem | edleafe: on #2 the problem is when we reschedule right? | |
| 17:28:17 | edleafe | yeah | |
| 17:28:33 | edleafe | it's checking that the instance is allocated against the source | |
| 17:28:35 | mriedem | at the point of the first reschedule, the source node allocs are on the migration record and the failed first chosen host allocs are on the instance | |
| 17:28:41 | mriedem | right, i'm saying, | |
| 17:28:47 | edleafe | on a retry, the allocs on the source will be the migration | |
| 17:28:49 | mriedem | we need to modify the logic that assumes the source node allocs are on the instance, | |
| 17:28:55 | mriedem | if we know we're doing a reschedule, | |
| 17:29:01 | mriedem | and we can determine that based on the filter_properties 'retry' entry | |
| 17:29:19 | mriedem | so on a retry, we don't do anything with the source, | |
| 17:29:22 | mriedem | we just update the dest | |
| 17:29:36 | mriedem | update the instance allocation from failed dest 1 to next dest 2 | |
| 17:29:39 | edleafe | mriedem: what did you mean then by "only change the allocation for the instance record"? | |
| 17:30:06 | mriedem | on a reschedule, we move the instance allocation from the failed dest host to the next alternative host | |
| 17:30:12 | mriedem | and don't touch the migration allocation on the source host | |
| 17:30:18 | edleafe | the instance won't be allocated after a fail. Those should be rolled back, no? | |
| 17:30:49 | edleafe | no, the next host isn't selected until after this part of the code is run | |
| 17:31:05 | edleafe | This is happening in _preallocate_migration | |
| 17:32:05 | mriedem | where do we rollback allocations for the instance on a fail? | |
| 17:32:08 | edleafe | When we eventually select a host (either through select_destinations (now) or alternate host (when that last patch merges), that's when we allocate the instance to the target | |
| 17:32:40 | edleafe | mriedem: in compute. Let me look | |
| 17:33:36 | mriedem | delete_allocation_for_failed_resize in prep_resize? | |
| 17:33:52 | mriedem | oh i see, the call to _revert_allocation | |
| 17:34:14 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4095 | |
| 17:34:46 | edleafe | damn, you're faster than I am :) | |
| 17:35:08 | mriedem | so if the compute reverts the allocations before casting back to conductor to reschedule, | |
| 17:35:14 | mriedem | why is conductor failing on the reschedule | |
| 17:35:15 | mriedem | ? | |
| 17:35:34 | edleafe | Because it is confirming that the instance has allocations on the source | |