Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-24
15:03:44 sean-k-mooney ya we do
15:04:23 sean-k-mooney disk_allocation_ratio=1.0,disk_available_least=8,free_disk_gb=10,f
15:04:33 dansmith we're only asking placement for DISK_GB=1 allocation so I don't think we're getting a bad number from glance or anything
15:06:15 sean-k-mooney what do our flavor look like
15:06:27 sean-k-mooney actully no never mind
15:06:30 sean-k-mooney this is not bfv
15:06:51 sean-k-mooney the flavor should be either 1 or 2GB per instance i think
15:07:20 dansmith and it seems like 1 since we're asking for that size allocation
15:09:44 sean-k-mooney ya its based on teh image size https://github.com/openstack/devstack/blob/2ecd1823850ae0e00ad0ecebbbceb312be60ccf4/lib/tempest#L204-L206
15:09:53 sean-k-mooney so for cirros image it will be 1g
15:10:04 dansmith sudo ceph -c /etc/ceph/ceph.conf osd pool create vms 8 8
15:10:10 dansmith that's 8G for the vms pool
15:10:18 dansmith I dunno where we're getting 10G
15:10:26 sean-k-mooney i dont think that is the size
15:10:43 sean-k-mooney i think that is the buckest to share it in
15:10:45 sean-k-mooney let me check
15:11:15 dansmith hmm, okay it seems like size
15:11:35 sean-k-mooney i think its the placment groups but its been a while
15:11:41 dansmith okay yeah, maybe you're right
15:12:39 sean-k-mooney ceph osd pool create <pool-name> <pg-num> <pgp-num> [replicated] \
15:12:41 sean-k-mooney [crush-ruleset-name] [expected-num-objects]
15:12:52 sean-k-mooney so ya its not the size
15:13:04 dansmith yeah
15:13:22 dansmith I still dunno where we're getting 10G,
15:13:34 sean-k-mooney same
15:14:27 dansmith because CEPH_LOOPBACK_DISK_SIZE=24G
15:14:57 sean-k-mooney so it should be 8 https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/settings#L17
15:15:01 sean-k-mooney by default
15:15:12 dansmith it's overridden in our job somewhere
15:15:16 dansmith you can see in the devstacklog
15:15:24 sean-k-mooney CEPH_LOOPBACK_DISK_SIZE is
15:15:31 sean-k-mooney is VOLUME_BACKING_FILE_SIZE
15:15:34 dansmith ues
15:15:36 dansmith both are
15:15:47 sean-k-mooney ok cool
15:16:12 dansmith VOLUME_BACKING_FILE_SIZE=24G
15:16:18 dansmith and the df shows 24G on /var/lib/ceph
15:17:48 sean-k-mooney ah yes it does
15:17:55 dansmith we run "ceph df" to get the DISK_GB we report,
15:17:56 dansmith and don't really do much to it,
15:18:08 dansmith so it really seems like we're being told 10G
15:19:52 dansmith lyarwood: do you know anything about what ceph df may be telling us about total pool size that differs from the backing store's size?
15:20:34 lyarwood dansmith: nope, AFAIK it just reports the size of the images_rbd_pool
15:21:16 dansmith seems straightforward :)
15:21:38 lyarwood https://github.com/openstack/nova/blob/master/nova/virt/libvirt/storage/rbd_utils.py#L374-L382 - ah well melwitt has a handy comment here that might help
15:21:50 lyarwood gah highlights are a new lines off but you get the point
15:21:58 dansmith oh I read that,
15:22:01 dansmith but didn't grok until now
15:22:10 dansmith so replication makes the thing looks smaller I guess?
15:22:37 dansmith seems weird to go from 24G to 10G, as that's not an even factor
15:23:15 dansmith er, no wait
15:23:24 dansmith that's for max_avail, which is "free" not total right?
15:24:30 lyarwood right sorry and you're seeing 10 reported as the total capacity right?
15:24:34 dansmith correct
15:24:40 lyarwood kk sorry then that isn't it
15:26:43 dansmith I guess one thing we could do is increase the ceph backing size to 36G and see if DISK_GB goes up
15:26:45 bauzas can someone tell me what the fuck is ? http://paste.openstack.org/show/796292/
15:26:57 bauzas tl;dr: ssh: connect to host review.openstack.org port 29418: Network is unreachable
15:27:11 bauzas have I missed a memo ?
15:27:25 melwitt there's a openstackstatus above ^ said there will be a short outage
15:29:00 openstackgerrit Sylvain Bauza proposed openstack/nova-specs master: WIP: Offline Reshape tool spec https://review.opendev.org/742908
15:29:05 bauzas yay, it worked
15:29:09 bauzas melwitt: thanks
15:29:46 bauzas calling it a day
15:32:15 melwitt dansmith: MAX_AVAIL should be total actually, just taking number of replicas into account. if you only have 1 replica (default NUM_REPLICAS=1) then MAX_AVAIL should match whatever total says in 'ceph df'
15:32:49 dansmith melwitt: you're reporting free as max_avail though in that thing aren't you?
15:32:59 dansmith or does MAX_AVAIL != max_avail ?
15:32:59 melwitt but if you've set NUM_REPLICAS=2 when you deployed a devstack, then since the devstack ceph plugin creates 2 OSDs on the same HDD in that case, it would be 2x the real disk
15:33:07 melwitt no MAX_AVAIL is a ceph thing
15:33:30 melwitt (if you're referring to what is written about ceph df in rbd_utils.py)
15:33:32 dansmith you mean half the disk I assume
15:33:34 dansmith yeah
15:33:57 melwitt no like the old behavior used to report 20G if you had a 10G disk, of you had created 2 OSDs that point at the same HDD
15:34:02 dansmith so maybe (24 - overhead) / 2 == 10 or something
15:34:22 melwitt you're using NUM_REPLICAS=1 right? you didn't set it in the job
15:34:31 melwitt if so, there shouldn't be a difference
15:34:38 dansmith I'm not setting it, but let me look if it's getting set
15:35:06 melwitt I doubt it, I've never seen it set in CI before. I had to set it locally to do the testing for that MAX_AVAIL change
15:35:26 dansmith yeah I don't even see that variable anywhere
15:35:41 dansmith is that a devstack-plugin-ceph thing?
15:35:45 melwitt yeah sec
15:36:02 lyarwood dansmith: https://docs.ceph.com/docs/jewel/rados/operations/pools/#create-a-pool ; sudo ceph -c /etc/ceph/ceph.conf osd pool create vms 8 8 ; that doesn't mean create a 8GB pool
15:36:09 melwitt bah sorry it's CEPH_REPLICAS https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L109
15:36:15 dansmith lyarwood: yeah we established that :)
15:36:22 lyarwood ah sorry wasn't watching irc
15:36:30 dansmith lyarwood: somewhere in the plugin I saw a comment that made it sound like that was size
15:37:16 melwitt 10G honestly I would have thought is just the cloud image's disk size, no?
15:37:20 dansmith melwitt: yeah 1
15:37:23 melwitt or do we probably use something larger in CI
15:37:26 dansmith melwitt: no, said above, it's 24G
15:37:37 dansmith melwitt: https://zuul.opendev.org/t/openstack/build/13d8a055ff1b4be0b627205f4d51d50f/log/controller/logs/df.txt
15:37:46 dansmith and it's overridden to 24G in the devstack log
15:37:50 lyarwood that's total for the three different pools
15:37:59 lyarwood vms images and volumes?
15:38:04 melwitt oh I see
15:38:14 dansmith lyarwood: and are the pools set to something specific for size? that's what we're trying to find and can't :)
15:38:38 dansmith lyarwood: the way it looks now I'd assume it just reports that they're all 24G in size, with various amounts free like zfs does for filesystems on a pool,
15:38:42 dansmith but I'm just guessing
15:38:52 dansmith I'm stacking a ceph devstack so I can poke but right now all I have is logs
15:39:28 dansmith if total decreases as we use space, then we're not really reporting the right thing to placement

Earlier   Later