Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-02-17
10:23:48 mdbooth Not 100% sure, but at least moderately sure that's the source of the error.
10:23:53 johnthetubaguy tojuvone: or just prefer things that now are no longer impacted by the maintenance, I guess
10:24:08 lyarwood https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1049-L1053
10:24:11 johnthetubaguy tojuvone: lets discuss making sure we have enough information to make that all work
10:24:18 mdbooth resource tracker, even. Yeah
10:24:19 bauzas mdbooth: sorry, yesterday's evening was pretty busy and I had no time replying to you
10:24:31 mdbooth bauzas: NP. I was just trying to short cut :)
10:24:33 bauzas mdbooth: what's the gist of the problem ?
10:24:51 mdbooth bauzas: I think it's the code lyarwood just pasted above
10:25:13 tojuvone johnthetubaguy, yes, the scheduling I had earlier as it was mostly wanted by ops
10:25:17 mdbooth bauzas: The issue, I think, is that it doesn't take account of bdms.
10:25:34 mdbooth So, although the flavor has root_gb of X, if the instance is BFV it actually doesn't use that
10:25:41 mdbooth Same with ephemeral
10:25:49 johnthetubaguy tojuvone: we need to keep the use cases that are required like that one
10:25:54 mdbooth And it doesn't account for swap, or kernel/ramdisk
10:26:23 bauzas mdbooth: right, that's a known issue that we're trying to solve by using shared resource providers
10:26:33 mdbooth bauzas: That's a different problem
10:26:42 mdbooth This has nothing to do with shared storage
10:27:01 bauzas mdbooth: it has to do about volumes, right?
10:27:19 mdbooth bauzas: In this case it has to do with block device mappings
10:27:45 bauzas mdbooth: lemme rephrase then
10:27:49 mdbooth IIUC, that code is what's calculating an instance's resource usage to update whatever
10:27:59 mdbooth However, it's incorrectly calculating an instance's resource usage
10:28:16 mdbooth If the thing it's updated is aware of shared storage, it'll still update it incorrectly
10:28:16 bauzas mdbooth: the problem you state is that if you have an attached volume that maps your instance, the flavor disk information is not accurate, right?
10:28:34 mdbooth The flavor disk information doesn't necessarily apply to the instance
10:28:51 mdbooth So although the instance is booted with flavor X which has root_gb=20
10:29:08 bauzas so, mdbooth I said :
10:29:10 bauzas (11:28:16) bauzas: mdbooth: the problem you state is that if you have an attached volume that maps your instance, the flavor disk information is not accurate, right?
10:29:11 mdbooth It may actually be using any amount of root disk between 0 and 20
10:29:21 tojuvone johnthetubaguy, Thanks, I'll work that out.
10:29:56 bauzas mdbooth: it may be using less than 20, but in which case ?
10:29:59 mdbooth bauzas: block device mappings also do volumes, so yeah I guess that's an aspect
10:30:11 bauzas ah-ah I see
10:30:16 bauzas I see your point about BDMs
10:30:28 bauzas the fact that we don't know the real size ?
10:30:31 mdbooth Yes
10:30:33 johnthetubaguy tojuvone: yeah, the aim here is to cover your use cases in a way that fits with long term direction of the APIs etc
10:30:57 johnthetubaguy tojuvone: so keep all the use cases and problems that need to be solved, and concentrate on getting a good user story
10:31:17 johnthetubaguy tojuvone: the negotiation here is on how we implement it, and where the hard stuff goes
10:32:06 bauzas mdbooth: well, the fact is that it depends on what's behind the BDM, right?
10:32:40 bauzas mdbooth: but whatever it is, it's still capping at the maximum size given by the flavor ?
10:32:51 tojuvone johnthetubaguy, Yes and there is different sides. Generic "host state" to be outside with notif about change makes it again easier to catch that in monitoring hosts
10:32:57 mdbooth bauzas: Yes, flavor is a maximum.
10:33:37 mdbooth The issue reported here is that if your instances are predominantly BFV, then RM thinks you ran out of disk long before you actually did.
10:34:02 mdbooth In fact, RM thinks you have negative disk space
10:34:06 bauzas mdbooth: but those instances could max out up to that size
10:34:21 johnthetubaguy tojuvone: yeah, lets get that admin view and user view pinned down nicely
10:34:40 mdbooth bauzas: Interesting idea. Not without rescheduling, I think.
10:34:56 mdbooth bauzas: I think you'd have to resize.
10:35:08 bauzas mdbooth: if my BDM source type is image
10:35:19 mdbooth Even then, can you change the non-volume bdms?
10:36:00 lyarwood don't we have periodic jobs already checking the sizes of local disks etc on each of the hosts? I assumed we were already reporting this somewhere for quotas etc?
10:36:13 bauzas the size on the disk can grow for a non-volume, right?
10:36:26 mdbooth bauzas: I can't think of anything in the libvirt driver which allows resizing/adding ephemeral disks, except for the resize operation
10:36:30 mdbooth bauzas: No
10:36:41 mdbooth This isn't overcommit.
10:36:55 bauzas I'm puzzled honestly
10:36:56 mdbooth This is, the user specified a BDM with a smaller disk size.
10:37:07 mdbooth Or in the case of BFV, no disk at all.
10:37:34 bauzas okay, let's admit a flavor of 0
10:40:53 bauzas mdbooth: tbh, I'm not particularly seasoned with BDMs, so reading https://docs.openstack.org/developer/nova/block_device_mapping.html
10:41:11 mdbooth bauzas: You're not alone :)
10:41:41 bauzas mdbooth: I mean, I trust you when you describe it could be suboptimal
10:42:03 bauzas mdbooth: I'm just tryint to get more picture to know whether it's by design
10:42:40 bauzas mdbooth: lemme explain
10:42:50 mdbooth bauzas: So I'm not that familiar with the resource manager code, which was probably pretty apparent yesterday
10:43:15 bauzas mdbooth: that's how community works :)
10:43:17 mdbooth I have a vague understanding of what happens, but I'm not entirely sure which bits do what, or even what all the bits are. So I did some digging.
10:44:07 mdbooth I *think* that code above is what RM ultimately uses to update the scheduler's understanding of host resource usage, based on scheduling a specific instance.
10:44:34 mdbooth Which I *think* means that, if it's wrong, something up the line will ultimately add up a bunch of wrong numbers.
10:54:03 openstackgerrit jichenjc proposed openstack/nova master: add zvm migration object https://review.openstack.org/414090
10:58:00 BobBall mriedem: Many thanks for your help + shepherding on the XAPI live migration fix :) I'm really glad it made Ocata. Now it's up to us to make sure it doesn't happen again.
11:11:05 openstackgerrit Sivasathurappan Radhakrishnan proposed openstack/nova-specs master: Live Migration of Rescued Instances https://review.openstack.org/347161
11:18:51 openstackgerrit Lee Yarwood proposed openstack/nova master: encryptors: Switch to os-brick encryptor classes https://review.openstack.org/391597
11:22:31 openstackgerrit Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622
11:22:32 openstackgerrit Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100
11:22:32 openstackgerrit Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082
11:22:34 openstackgerrit Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941
11:22:34 openstackgerrit Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021
11:22:36 openstackgerrit Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520
11:22:36 openstackgerrit Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387
11:36:55 openstackgerrit Silvan Kaiser proposed openstack/nova master: Removes getfattr from Quobyte Nova driver https://review.openstack.org/428646
11:41:35 mdbooth bauzas: Hey, what does scheduler/client/report.py do? That's also not quite right.
11:41:58 bauzas mdbooth: it's both updating the DB and also calling the placement API
11:42:13 openstackgerrit Artom Lifshitz proposed openstack/nova-specs master: Virtual guest device role tagging https://review.openstack.org/435397
11:42:38 mdbooth bauzas: Is the behaviour of DiskFilter provided by anything else, btw?
11:42:47 mdbooth Can we just deprecate it in favour of X?
11:59:00 openstackgerrit Sean Dague proposed openstack/nova master: Verify project_id when quotas are checked https://review.openstack.org/435010
12:01:43 lyarwood sdague: morning, just looking at https://review.openstack.org/#/c/433825/ - would https://review.openstack.org/435406 - disabling gate-tempest-dsvm-full-devstack-plugin-ceph-ubuntu-trusty for stable/mitaka help here?
12:02:14 lyarwood or should we skip this with another devstack rc file?
12:02:34 lyarwood s/skip this/skip the test/g
12:03:04 sdague lyarwood: yeh, we could make it non voting on mitaka to get that other fix in
12:03:42 sdague lyarwood: is anyone working on the ceph issue?
12:04:21 lyarwood sdague: I was about to add the fact that it's non-voting to the ptg etherpad, I don't think anyone is working on it tbh
12:06:37 bauzas mdbooth: so, the DiskFilter is now legacy
12:06:53 bauzas mdbooth: given we now use the Placement API for verifying the disk usage
12:07:08 bauzas we still use it for the CachingScheduler
12:07:28 mdbooth bauzas: So it's worth fixing ResourceTracker?
12:08:32 bauzas mdbooth: well, the RT is still used for providing the resources

Earlier   Later