Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-26
21:11:16 dansmith 22
21:11:16 dansmith echo $((`cat ceph | grep vms | awk '{print $5}' | egrep -o '[0-9]+'` / 1024))
21:11:45 dansmith oh json format is nice
21:11:51 dansmith is jq on the worker nodes? I bet it is
21:12:03 mriedem 25482182656,
21:12:03 mriedem $ sudo ceph df --format=json | python -m json.tool | grep total_avail_bytes | awk 'END{print $2}'
21:12:03 mriedem except the comma
21:12:22 dansmith hold up
21:13:03 mriedem 25481973760
21:13:03 mriedem $ sudo ceph df --format=json | python -m json.tool | grep total_avail_bytes | awk 'END{print $2}' | egrep -o '[0-9]+'
21:13:03 mriedem haha
21:13:35 dansmith ceph df --format=json |
21:13:43 dansmith ceph df --format=json | jq .total_bytes
21:13:49 dansmith er
21:13:57 dansmith ceph df --format=json | jq stats.total_bytes
21:13:59 dansmith I think
21:14:14 mriedem don't have jq
21:14:28 dansmith ceph df --format=json | jq .stats.total_bytes
21:14:30 dansmith ^ works
21:14:37 dansmith on the worker nodes?
21:14:40 dansmith that sucks
21:14:43 mriedem 25757220864
21:14:43 mriedem $ sudo ceph df --format=json | jq .stats.total_bytes
21:14:47 mriedem i mean, we can install it...
21:14:58 mriedem $ sudo ceph df --format=json | python -m json.tool | grep total_avail_bytes | egrep -o '[0-9]+'
21:14:58 mriedem but this also works
21:14:59 mriedem 25481973760
21:15:12 dansmith yup
21:15:22 dansmith convert to GB of course
21:16:41 mriedem $ sudo ceph df --format=json-pretty | grep total_avail_bytes | egrep -o '[0-9]+'
21:16:41 mriedem yup; even shorter with json-pretty
21:16:42 mriedem 25481887744
21:27:01 mriedem oh do we have to pull from the vms pool specifically?
21:27:02 mriedem i suppose huh
21:28:29 dansmith I assume it's all the same space since it's the same size
21:28:35 melwitt rc todo etherpad https://etherpad.openstack.org/p/nova-rocky-release-candidate-todo
21:28:37 dansmith and you're only going to update it once (and it's just inventory)
21:28:42 dansmith so, probably meh
21:31:19 mriedem the max available is the same for the 3 pools
21:32:06 mriedem /dev/loop0 24G 228M 24G 1% /var/lib/ceph
21:32:06 mriedem oh i see
21:32:06 dansmith yeah, I dunno what that's showing really
21:32:18 dansmith could be that if you allocate some from the images you'll lose max on the vms
21:32:22 dansmith like ZFS in a pool
21:32:27 mriedem yeah so it's the same 24GB block shared across 3 pools?
21:32:31 dansmith yeah I'm sure
21:33:02 dansmith hard to say what the right thing to do is, but for testing just saying it's 24G is probably fine
21:33:35 mriedem was trying to think what we should put for reserved
21:33:37 mriedem or just leave that 0
21:33:43 dansmith it won't be right regardless,
21:33:58 dansmith other than maybe reserving enough for whatever images and snaps we do in a regular tempest run
21:34:02 dansmith but still, it's just a gamble
21:35:19 mriedem yeah so i can really just use CEPH_LOOPBACK_DISK_SIZE in devstack, and don't need to parse ceph df output then..
21:36:33 dansmith hah
21:38:29 mriedem but this has been fun
22:03:49 openstack Launchpad bug 1783896 in placement-osc-plugin "openstack resource provider aggregate set <name> should be <uuid>" [Medium,Triaged]
22:03:49 mriedem easy bug https://bugs.launchpad.net/placement-osc-plugin/+bug/1783896
22:05:15 melwitt mriedem: this patch for the metadata API bp, it looks good to me but I wasn't 100% sure the vmware CI fail on it is legit https://review.openstack.org/580742 forgot to mention it this morning
22:05:46 melwitt the other third party CI fails on it look unrelated
22:06:00 mriedem can't really dig into that atm
22:06:05 mriedem already shouldn't be working anymore
22:06:14 melwitt ok
22:11:04 melwitt mriedem: so we're Depends-On https://review.openstack.org/566813 and https://review.openstack.org/564452 for r-3, is there anything else that you know of?
22:11:35 mriedem dansmith: melwitt: https://review.openstack.org/586363
22:11:55 melwitt oh yeah, guh.
22:12:03 mriedem those are the 2 i'm watching
22:12:19 mriedem no, https://review.openstack.org/#/c/586363/ is the ceph job using a shared provider in placement for disk
22:12:22 mriedem efried: ^
22:13:01 openstackgerrit OpenStack Release Bot proposed openstack/python-novaclient stable/rocky: Update .gitreview for stable/rocky https://review.openstack.org/586364
22:13:03 openstackgerrit OpenStack Release Bot proposed openstack/python-novaclient stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/586365
22:13:05 openstackgerrit OpenStack Release Bot proposed openstack/python-novaclient master: Update reno for stable/rocky https://review.openstack.org/586366
22:13:39 melwitt mriedem: oh, coincidence, ok. but you reminded me, we need your test to determine whether or not to reno a warning not to use the shared aggregate thing. does that have to be done before my EOD or just before RC1?
22:14:16 mriedem we don't have to warn about that in a reno
22:14:20 mriedem it's totally undocumented
22:14:25 mriedem we don't need to say "yup this still doesn't work"
22:14:51 melwitt ok, thanks
22:25:37 dansmith we have it in specs,
22:25:43 dansmith and before this, setting it would do nothing
22:25:50 dansmith people do read and try stuff that are in specs
22:26:05 dansmith so I think we should just patch it out of the driver if we're not going to mention it personally (assuming it's broken like we expect)
22:26:07 dansmith just MHO
22:26:37 dansmith if we start claiming in stein that it works, people will try it in older releases too
22:30:45 melwitt mriedem_afk: fyi nova r-3 release patch https://review.openstack.org/586368
22:31:38 dansmith first hit: https://docs.openstack.org/nova/latest/reference/update-provider-tree.html
22:33:39 openstack Launchpad bug 1771728 in OpenStack Compute (nova) "libvirt: Shared Resource Provider (RP) DISK_GB is NOT taken into account if it's configured with Compute Node RPs" [Undecided,Fix released] - Assigned to Bhagyashri Shewale (bhagyashri-shewale)
22:33:39 melwitt ok ... thinking. the change was done as a bug fix, https://bugs.launchpad.net/nova/+bug/1771728 which was found via someone setting MISC_SHARES_VIA_AGGREGATE
22:35:02 melwitt if we patch the bug fix out, then it goes back to setting it does nothing. but that still seems weird because people think they can use it (because of this doc, possibly)
22:36:53 dansmith yeah, so someone thought they could use it and reported it did nothing,
22:36:59 melwitt I dunno, maybe just a doc update to this page saying MISC_SHARES_VIA_AGGREGATE isn't tested in tempest?
22:37:09 melwitt like a warning?
22:37:17 dansmith but if that's the case and it actually breaks resource reporting and probably migrations...
22:37:48 dansmith if it totally doesn't do anything other than break things we should probably not do it until the rest of the feature works
22:37:55 dansmith but at least a known issue reno, IMHO
22:38:18 dansmith not tested in tempest is different than "totally broken and we know it"
22:38:20 melwitt what I mean is, was the previous state of it doing nothing actually doing nothing? or could it have been breaking things underneath?
22:38:49 dansmith no, it was just being ignored when people expected it to magically make the compute node do the shared storage accounting right
22:39:12 dansmith now, setting it will cause the compute node to not expose disk inventory,
22:39:31 dansmith and likely in a migration situation we will clobber the allocation against the shared pool,
22:39:37 dansmith which will result in instances with no disk allocation,
22:40:02 melwitt okay ... so maybe, revert the bug fix, add a warning to the doc to mention MISC_SHARES_VIA_AGGREGATE isn't usable in nova yet, and then don't allow the bug fix back in until we have tempest setup with ceph etc?

Earlier   Later