| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-11 | |||
| 13:44:25 | sahid | mriedem: i'm a bit under pressure with the specs i have to work on and my downstream work... I need to build an on a SRIOV to test what you have indicated on the review then I will work on the metadata thing | |
| 13:44:30 | sahid | but yes it's my plan | |
| 13:44:59 | mriedem | sahid: ok thanks | |
| 13:45:05 | jichen | mriedem: ok , I don't have too much background on that stuff, anyway, I followed their patch and updated nova accordingly | |
| 13:45:21 | jichen | hopefully anyone can take a look | |
| 13:46:04 | mriedem | jichen: https://review.openstack.org/#/c/559367/ is merged | |
| 13:46:08 | mriedem | so it seems we don't need this in nova? | |
| 13:47:31 | jichen | mriedem , no , I guess that's the reason caused my issue http://logs.openstack.org/87/523387/32/check/requirements-check/408e28c/job-output.txt.gz, Doug replied in ML and say that's a correct error ... | |
| 13:48:10 | jichen | so I assume we need some changes but not too much experience here | |
| 13:48:11 | mriedem | ok that's a different change https://review.openstack.org/#/c/523387/ | |
| 13:48:33 | mriedem | ok i can try to sort this out, i know it's late for you | |
| 13:48:46 | jichen | ok, thank you for your help | |
| 13:50:11 | openstackgerrit | Merged openstack/nova master: libvirt: handle DiskNotFound during update_available_resource https://review.openstack.org/553067 | |
| 13:50:26 | openstackgerrit | Merged openstack/nova master: Update ImageMetaProp object to expose traits https://review.openstack.org/557795 | |
| 13:51:00 | jaypipes | bhagyashris: did you need something? | |
| 13:52:26 | efried | would be after limits, the granulars would be after member_of... IMO it's better if it's in the most sensible logical order for users of the latest microversion. | |
| 13:52:26 | efried | mriedem: I noticed your reference to https://wiki.openstack.org/wiki/NovaAPIRef#Parameter_Verification in a patch. I have to say I don't agree that the ordering (required=>optional=>by-microversion) should be strictly followed where it doesn't make sense. For example, here https://review.openstack.org/#/c/517757/7/placement-api-ref/source/allocation_candidates.inc it makes more logical sense to order them this way. If I | |
| 13:53:51 | bhagyashris | jaypipes: Jus want to inform few thing regarding the shared resource provider that we have discussed earlier, I have done some testing and analysis. | |
| 13:54:10 | bhagyashris | jaypipes: s/Jus/just | |
| 13:54:59 | mriedem | efried: you'll have to debate that with sdague | |
| 13:55:17 | mriedem | i assume microversions come last in order because we are trying to write api ref docs for the lowest common denominator | |
| 13:55:32 | efried | mriedem: Maybe placement is different. | |
| 13:55:39 | mriedem | placement is always different | |
| 13:56:06 | efried | mriedem: I'm not necessarily suggesting action (like an edit of the guidelines). Just something I noticed and wanted to get off my chest. | |
| 13:56:14 | efried | Thanks for listening :) | |
| 13:57:48 | bhagyashris | jaypipes: I have tested the scenario and here are the observation http://paste.openstack.org/show/718936/ | |
| 13:58:58 | efried | bhagyashris: Woot! That looks just like it should! | |
| 13:59:50 | bhagyashris | efried, jaypipes: Based on the current analysis, it appears that compute node shouldn't report it's own resources in inventory if it's configured to use it from shared resource providers but currently, compute node doesn't have any mechanism to report selective resources in inventory. | |
| 13:59:50 | jaypipes | efried: indeed, after commenting out the "compute node sets DISK_GB inventory item automatically" code. | |
| 14:00:20 | jaypipes | bhagyashris: correct. the original idea was to use the aggregates information in the ProviderTree to determine whether to report "local disk" or not. | |
| 14:00:41 | openstackgerrit | Steve Noyes proposed openstack/nova master: Enables MySQL Cluster Support for Nova https://review.openstack.org/446643 | |
| 14:00:42 | jaypipes | bhagyashris: efried can share some information with you about the aggregate tracking in the ProviderTree object | |
| 14:01:00 | efried | jaypipes: You're saying that if we find a sharing provider for a given RC, we should *remove* that RC from the compute node's inventory?? | |
| 14:01:19 | jaypipes | efried: I'm saying that was my original idea, yeah. | |
| 14:01:32 | bhagyashris | efried, jaypipes: There is a TODO note here : https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L212-L216 that the first allocation request object is use to claim resources | |
| 14:01:36 | jaypipes | (that's why I originally added the aggregate tracking in the reportclient to begin with) | |
| 14:02:27 | efried | bhagyashris: jaypipes: Do we have any evidence that the virt driver actually *used* the shared storage to make the disk? | |
| 14:02:31 | jaypipes | bhagyashris: yes, but that's more about the use case where we have both shared and local disk and can't decide which to consume from. what efried and I are discussing is how to signal to the compute node not to report local disk if it sees a sharing provider of disk. | |
| 14:02:34 | efried | or has that piece not been hooked up yet? | |
| 14:02:39 | bhagyashris | efried, jaypipes: yeah that is the best idea and that will not need to make any change at sch side | |
| 14:02:53 | jaypipes | efried: well, that's an entirely different question :) | |
| 14:03:01 | efried | But an important one, you'll agree. | |
| 14:03:08 | jaypipes | efried: and I see no reason why it wouldn't. | |
| 14:03:20 | efried | Doesn't do us much good to have the alloc come from the right place if the storage still comes from the wrong place. | |
| 14:03:20 | bhagyashris | jaypipes: yeah totally agree with you point | |
| 14:03:29 | jaypipes | efried: it's the same reason why pinning of CPU resources still works even though VCPU is tracked in placement. | |
| 14:03:37 | jaypipes | efried: in other words, assignment != allocation | |
| 14:04:30 | efried | jaypipes: So you're saying that by virtue of whatever existing code recognized the flavor request as wanting the storage to come from shared... is still working. | |
| 14:04:43 | efried | I don't know what that mechanism is, but if that's the case... fine. | |
| 14:04:43 | jaypipes | efried: don't confuse a request for 10TB of disk with block device configuration information. | |
| 14:05:08 | efried | ah, the bdm somehow identifies the source of the disk space? | |
| 14:05:51 | jaypipes | efried: no. nothing really knows whether the source of the disk space is shared. it's just that /var/lib/instances is on mounted shared storage. | |
| 14:06:14 | jaypipes | efried: that's pretty much the reason why shared storage resource accounting has been so wonky. | |
| 14:06:18 | efried | ohh. | |
| 14:06:26 | efried | So in this setup, there really *is* no local storage. | |
| 14:06:53 | jaypipes | efried: there totally *could* be. no way to know other than manually commenting out that code right now. | |
| 14:06:59 | openstackgerrit | Merged openstack/nova master: Update wording in @safe_connect placement warnings https://review.openstack.org/560039 | |
| 14:07:50 | bhagyashris | jaypipes: yeah that way I have tested | |
| 14:07:56 | jaypipes | efried: for the record, we used to determine if "the node was on shared storage" by literally attempting to write a file from two different compute hosts to a single location and checking if each node could read it. | |
| 14:08:13 | jaypipes | efried: IIRC, this was during live migration setup. | |
| 14:08:15 | efried | jaypipes: So here's where the virt driver would need to recognize whether the storage is shared or not, and if so, create* the sharing provider and put the DISK_GB inventory there instead of the compute node RP. | |
| 14:08:16 | efried | (*create, co-create, make sure it exists, etc.) | |
| 14:08:23 | jaypipes | efried: not sure if we still do that, but that's how it used to be. | |
| 14:08:49 | jaypipes | efried: no, the idea was never to have a compute node create the sharing provider. | |
| 14:08:58 | efried | don't focus on that bit. | |
| 14:09:04 | jaypipes | efried: the admin would do that, since they know they are setting up shared storage for a set of nodes. | |
| 14:09:27 | efried | well, okay; I'm also thinking about the PowerVM case, where we *do* want the driver to co-manage creation of the sharing provider. | |
| 14:10:09 | efried | because that would also be a non-hacky way to get rid of the compute node RP's DISK_GB inventory :) | |
| 14:10:45 | efried | Because we can't just blindly remove it (outside of the virt driver) based on the presence of an aggregated provider of the same RC. | |
| 14:11:06 | jaypipes | efried: I don't understand that last sentence. | |
| 14:11:11 | efried | We may legitimately have both shared and local at the same time. We may legitimately want the virt driver to do that management, etc. | |
| 14:11:50 | jaypipes | efried: why would you want the virt driver on a single compute node managing disk resources for multiple compute nodes? | |
| 14:11:52 | efried | jaypipes: I'm just saying if we're going to fiddle with the compute RP's DISK_GB inventory based on existence of sharing DISK_GB providers, that logic needs to be fully owned by the virt driver. We can't assume things outside of that scope. | |
| 14:12:46 | jaypipes | efried: sure. the process of fiddling with any resources that are owned by that compute node (and its children) should of course be controlled by the virt driver. | |
| 14:13:02 | efried | jaypipes: In the PowerVM case, it's because we can; and because we can, it means we don't need any outside agent (human or code) manitaining that sharing RP. | |
| 14:13:04 | jaypipes | efried: I don't agree that a single compute node should control inventory on shared disk providers, though. | |
| 14:13:24 | jaypipes | efried: then powervm can do that I guess. | |
| 14:13:32 | jaypipes | efried: obviously libvirt won't :) | |
| 14:13:35 | efried | It won't be a single compute node; it'll all the compute nodes attached to that sharing provider. | |
| 14:13:44 | efried | co-managing the SSP. | |
| 14:13:58 | efried | ...because they already co-manage the SSP (outside of the auspices of openstack) | |
| 14:13:58 | jaypipes | efried: right, because it's a clustered hypervisor manager. | |
| 14:14:04 | efried | yeah, that. | |
| 14:14:18 | jaypipes | in any case... | |
| 14:14:27 | efried | So anyway, I'm saying I'm on board with the idea if we state it like this: | |
| 14:15:28 | efried | Because libvirt (somehow) knows that the storage it's looking at is shared, libvirt's impl of update_provider_tree shall remove the DISK_GB inventory from the compute node RP. | |
| 14:15:51 | efried | Perhaps the (somehow) is signalled by the operator having created the sharing provider and associated it via aggregate with the compute node. | |
| 14:16:12 | efried | libvirt's u_p_t will look for that and, if found, nix the compute RP's DISK_GB inventory. | |
| 14:16:18 | efried | Does that gel with your thinking? | |
| 14:16:44 | efried | zero part of this involves nova code outside of the libvirt driver. | |
| 14:17:41 | jaypipes | efried: that is precisely my thinking, yes. | |
| 14:18:17 | efried | jaypipes: Cool beans. bhagyashris, are you on the hook for proposing that code? I can help you understand how to make it work. | |
| 14:19:12 | bhagyashris | efried: ok means the CN RP DISK_GB inventory will be remove if the shared resource provider is configured | |
| 14:19:55 | efried | bhagyashris: Yes. The part I was unclear on at first was that that would be done within the libvirt compute driver's implementation of update_provider_tree. | |
| 14:20:05 | efried | ...which I bet hasn't been proposed yet. | |
| 14:21:57 | openstack | Launchpad bug 1762876 in OpenStack Compute (nova) "test_resize_with_reschedule_then_live_migrate intermittently failing; migration is not yet complete" [High,Confirmed] - Assigned to Matt Riedemann (mriedem) | |
| 14:21:57 | mriedem | fyi if you see test_resize_with_reschedule_then_live_migrate failing in functional tests, it's https://bugs.launchpad.net/nova/+bug/1762876 | |
| 14:21:58 | mriedem | i'm working on a fix | |
| 14:22:07 | mriedem | apparently the instance status goes to ACTIVE before the migration is completed | |
| 14:24:14 | bhagyashris | efried: ok. so for this is there any bp or bug reported? Actually i am not fully aware about update_provider_tree part but i will start to understand that part and will get to know | |