| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-17 | |||
| 10:18:08 | tojuvone | I mean spec in Ocata | |
| 10:18:42 | tojuvone | all that start, stop, state was put to external service where there would only be a link | |
| 10:19:10 | tojuvone | that surely is not so good when looking scheduler should read it from there | |
| 10:19:14 | lyarwood | mdbooth: well given where we are in the cycle now I guess I'm fine with going after the bigger fix but I don't see the harm in also resolving this edge case in the short term | |
| 10:19:33 | mdbooth | lyarwood: The harm I see is that it's obtuse. | |
| 10:19:46 | mdbooth | I think if we add that there, it will never be removed. | |
| 10:20:28 | lyarwood | mdbooth: and if we fix the underlying issue that shouldn't be a problem right? | |
| 10:20:35 | mdbooth | lyarwood: Right. | |
| 10:20:49 | mdbooth | That fix is going to work for shops who use almost entirely BFV. | |
| 10:21:03 | mdbooth | Because it basically disables DiskFilter. | |
| 10:21:11 | mdbooth | But... they might as well just disable DiskFilter. | |
| 10:21:18 | tojuvone | Well, if I get to write enough to spec about the case in admin, user and host monitoring perspective. Let's see where it goes | |
| 10:21:38 | johnthetubaguy | tojuvone: well, we really want to see if just the URL is enough for the required use cases, and write that up in a spec to show that it is | |
| 10:21:59 | lyarwood | mdbooth: right but to hit this you would need to run a mix of bfv and local disks | |
| 10:22:19 | mdbooth | I think you'll hit this if you run 100% bfv | |
| 10:22:19 | lyarwood | mdbooth: for the reported host values to go negative, AFAICT at least | |
| 10:22:38 | lyarwood | ah, it's just on the flavors? | |
| 10:22:44 | mdbooth | Yeah | |
| 10:22:46 | tojuvone | johnthetubaguy, yes | |
| 10:23:18 | johnthetubaguy | tojuvone: basically, a very common request is for the scheduler to not place new VMs where maintenance is about to occur, and prefer unaffected places | |
| 10:23:27 | mdbooth | lyarwood: Did you see the code I highlighted in resourcemanager? | |
| 10:23:47 | lyarwood | mdbooth: yeah, just going back through scrollback now, I missed your convo with bauzas about this yesterday | |
| 10:23:48 | mdbooth | Not 100% sure, but at least moderately sure that's the source of the error. | |
| 10:23:53 | johnthetubaguy | tojuvone: or just prefer things that now are no longer impacted by the maintenance, I guess | |
| 10:24:08 | lyarwood | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1049-L1053 | |
| 10:24:11 | johnthetubaguy | tojuvone: lets discuss making sure we have enough information to make that all work | |
| 10:24:18 | mdbooth | resource tracker, even. Yeah | |
| 10:24:19 | bauzas | mdbooth: sorry, yesterday's evening was pretty busy and I had no time replying to you | |
| 10:24:31 | mdbooth | bauzas: NP. I was just trying to short cut :) | |
| 10:24:33 | bauzas | mdbooth: what's the gist of the problem ? | |
| 10:24:51 | mdbooth | bauzas: I think it's the code lyarwood just pasted above | |
| 10:25:13 | tojuvone | johnthetubaguy, yes, the scheduling I had earlier as it was mostly wanted by ops | |
| 10:25:17 | mdbooth | bauzas: The issue, I think, is that it doesn't take account of bdms. | |
| 10:25:34 | mdbooth | So, although the flavor has root_gb of X, if the instance is BFV it actually doesn't use that | |
| 10:25:41 | mdbooth | Same with ephemeral | |
| 10:25:49 | johnthetubaguy | tojuvone: we need to keep the use cases that are required like that one | |
| 10:25:54 | mdbooth | And it doesn't account for swap, or kernel/ramdisk | |
| 10:26:23 | bauzas | mdbooth: right, that's a known issue that we're trying to solve by using shared resource providers | |
| 10:26:33 | mdbooth | bauzas: That's a different problem | |
| 10:26:42 | mdbooth | This has nothing to do with shared storage | |
| 10:27:01 | bauzas | mdbooth: it has to do about volumes, right? | |
| 10:27:19 | mdbooth | bauzas: In this case it has to do with block device mappings | |
| 10:27:45 | bauzas | mdbooth: lemme rephrase then | |
| 10:27:49 | mdbooth | IIUC, that code is what's calculating an instance's resource usage to update whatever | |
| 10:27:59 | mdbooth | However, it's incorrectly calculating an instance's resource usage | |
| 10:28:16 | bauzas | mdbooth: the problem you state is that if you have an attached volume that maps your instance, the flavor disk information is not accurate, right? | |
| 10:28:16 | mdbooth | If the thing it's updated is aware of shared storage, it'll still update it incorrectly | |
| 10:28:34 | mdbooth | The flavor disk information doesn't necessarily apply to the instance | |
| 10:28:51 | mdbooth | So although the instance is booted with flavor X which has root_gb=20 | |
| 10:29:08 | bauzas | so, mdbooth I said : | |
| 10:29:10 | bauzas | (11:28:16) bauzas: mdbooth: the problem you state is that if you have an attached volume that maps your instance, the flavor disk information is not accurate, right? | |
| 10:29:11 | mdbooth | It may actually be using any amount of root disk between 0 and 20 | |
| 10:29:21 | tojuvone | johnthetubaguy, Thanks, I'll work that out. | |
| 10:29:56 | bauzas | mdbooth: it may be using less than 20, but in which case ? | |
| 10:29:59 | mdbooth | bauzas: block device mappings also do volumes, so yeah I guess that's an aspect | |
| 10:30:11 | bauzas | ah-ah I see | |
| 10:30:16 | bauzas | I see your point about BDMs | |
| 10:30:28 | bauzas | the fact that we don't know the real size ? | |
| 10:30:31 | mdbooth | Yes | |
| 10:30:33 | johnthetubaguy | tojuvone: yeah, the aim here is to cover your use cases in a way that fits with long term direction of the APIs etc | |
| 10:30:57 | johnthetubaguy | tojuvone: so keep all the use cases and problems that need to be solved, and concentrate on getting a good user story | |
| 10:31:17 | johnthetubaguy | tojuvone: the negotiation here is on how we implement it, and where the hard stuff goes | |
| 10:32:06 | bauzas | mdbooth: well, the fact is that it depends on what's behind the BDM, right? | |
| 10:32:40 | bauzas | mdbooth: but whatever it is, it's still capping at the maximum size given by the flavor ? | |
| 10:32:51 | tojuvone | johnthetubaguy, Yes and there is different sides. Generic "host state" to be outside with notif about change makes it again easier to catch that in monitoring hosts | |
| 10:32:57 | mdbooth | bauzas: Yes, flavor is a maximum. | |
| 10:33:37 | mdbooth | The issue reported here is that if your instances are predominantly BFV, then RM thinks you ran out of disk long before you actually did. | |
| 10:34:02 | mdbooth | In fact, RM thinks you have negative disk space | |
| 10:34:06 | bauzas | mdbooth: but those instances could max out up to that size | |
| 10:34:21 | johnthetubaguy | tojuvone: yeah, lets get that admin view and user view pinned down nicely | |
| 10:34:40 | mdbooth | bauzas: Interesting idea. Not without rescheduling, I think. | |
| 10:34:56 | mdbooth | bauzas: I think you'd have to resize. | |
| 10:35:08 | bauzas | mdbooth: if my BDM source type is image | |
| 10:35:19 | mdbooth | Even then, can you change the non-volume bdms? | |
| 10:36:00 | lyarwood | don't we have periodic jobs already checking the sizes of local disks etc on each of the hosts? I assumed we were already reporting this somewhere for quotas etc? | |
| 10:36:13 | bauzas | the size on the disk can grow for a non-volume, right? | |
| 10:36:26 | mdbooth | bauzas: I can't think of anything in the libvirt driver which allows resizing/adding ephemeral disks, except for the resize operation | |
| 10:36:30 | mdbooth | bauzas: No | |
| 10:36:41 | mdbooth | This isn't overcommit. | |
| 10:36:55 | bauzas | I'm puzzled honestly | |
| 10:36:56 | mdbooth | This is, the user specified a BDM with a smaller disk size. | |
| 10:37:07 | mdbooth | Or in the case of BFV, no disk at all. | |
| 10:37:34 | bauzas | okay, let's admit a flavor of 0 | |
| 10:40:53 | bauzas | mdbooth: tbh, I'm not particularly seasoned with BDMs, so reading https://docs.openstack.org/developer/nova/block_device_mapping.html | |
| 10:41:11 | mdbooth | bauzas: You're not alone :) | |
| 10:41:41 | bauzas | mdbooth: I mean, I trust you when you describe it could be suboptimal | |
| 10:42:03 | bauzas | mdbooth: I'm just tryint to get more picture to know whether it's by design | |
| 10:42:40 | bauzas | mdbooth: lemme explain | |
| 10:42:50 | mdbooth | bauzas: So I'm not that familiar with the resource manager code, which was probably pretty apparent yesterday | |
| 10:43:15 | bauzas | mdbooth: that's how community works :) | |
| 10:43:17 | mdbooth | I have a vague understanding of what happens, but I'm not entirely sure which bits do what, or even what all the bits are. So I did some digging. | |
| 10:44:07 | mdbooth | I *think* that code above is what RM ultimately uses to update the scheduler's understanding of host resource usage, based on scheduling a specific instance. | |
| 10:44:34 | mdbooth | Which I *think* means that, if it's wrong, something up the line will ultimately add up a bunch of wrong numbers. | |
| 10:54:03 | openstackgerrit | jichenjc proposed openstack/nova master: add zvm migration object https://review.openstack.org/414090 | |
| 10:58:00 | BobBall | mriedem: Many thanks for your help + shepherding on the XAPI live migration fix :) I'm really glad it made Ocata. Now it's up to us to make sure it doesn't happen again. | |
| 11:11:05 | openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova-specs master: Live Migration of Rescued Instances https://review.openstack.org/347161 | |
| 11:18:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: encryptors: Switch to os-brick encryptor classes https://review.openstack.org/391597 | |
| 11:22:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 11:22:32 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 11:22:32 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |