| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-26 | |||
| 19:22:33 | mriedem | resize to same host probably doesn't handle it either, not sure | |
| 19:22:34 | dansmith | yeah, I remember now.. I was thinking those were only problems we had with the doubling approach, but I remember now | |
| 19:22:39 | mriedem | anywho, | |
| 19:22:52 | mriedem | CI testing with ceph + modeling shared is a todo for the ptg etherpad most likely | |
| 19:22:56 | mriedem | i'll add it | |
| 19:23:10 | dansmith | yeah, that probably needs to be a priority for someone, | |
| 19:23:30 | dansmith | because right now if you associate a sharing provider, things are going to break in unhelpful ways | |
| 19:23:33 | mriedem | i'll assign it to my cat | |
| 19:23:55 | dansmith | melwitt: ^ | |
| 19:24:08 | mriedem | yeah - that's why i've tried to temper enthusiasm / communication that "now it's fixed" | |
| 19:24:17 | mriedem | also because i'm a debby downer | |
| 19:24:58 | dansmith | fair enough, I just saw it had merged when I was triaging a bug related to that stuff | |
| 19:25:05 | dansmith | and figured it had actually been finished | |
| 19:25:29 | melwitt | I've been summoned for ceph stuff eh? | |
| 19:25:39 | dansmith | melwitt: no, for ptl stuff | |
| 19:26:11 | melwitt | oh, thought you were referring to the ceph CI testing add | |
| 19:26:17 | dansmith | I am, | |
| 19:26:41 | dansmith | I'm saying we probably need to be making sure that the test gets updated to validate whether this created a worse hole, | |
| 19:26:50 | dansmith | and maybe decide if we want to either patch that out, | |
| 19:26:54 | dansmith | or reno a known issue or whatever | |
| 19:26:55 | purplerbot | <mriedem> i've beaten cdent over the head with a few times already :) [2018-07-26 19:18:01.246831] [n 1fky] | |
| 19:26:55 | cdent | mriedem: on [t 1fky] | |
| 19:27:07 | melwitt | dansmith: I see, okay | |
| 19:27:35 | cdent | I'm trying to make sure that people are aware, but I'm not sure anybody really reads the weekly pupdates and I'm only willing to take full responsibility for the placement side (where it's ready) | |
| 19:28:03 | cdent | but I agree we have some holes to fill before much more time passes | |
| 19:28:54 | mriedem | L65 https://etherpad.openstack.org/p/nova-ptg-stein | |
| 19:29:21 | mriedem | i will check to see if we have an existing ceph multinode job, but i know the nova-live-migration job is multinode and re-configures itself to use ceph in the 2nd part | |
| 19:29:36 | mriedem | but live migration is less interesting since we don't do claims in the compute or anything | |
| 19:30:19 | dansmith | we really just need a multinode ceph job that boots an instance for base level verification, | |
| 19:30:26 | dansmith | but yeah it should do a cold migration at least | |
| 19:30:40 | dansmith | can we just convert the ceph job to multinode? | |
| 19:31:31 | mriedem | we can do anything we want | |
| 19:31:48 | cdent | anything? | |
| 19:31:49 | mriedem | but i'd rather re-use something than add yet another job | |
| 19:32:04 | mriedem | cdent: with zuul all things are possible (tm) | |
| 19:32:05 | dansmith | mriedem: well just converting the existing one to multinode I mean | |
| 19:32:12 | mriedem | dansmith: i know | |
| 19:32:36 | cdent | chris-runs-around-naked -1 | |
| 19:33:08 | dansmith | heck, even just the ceph job as a single node with the sharing flag would be enough to shake out some of it | |
| 19:33:33 | mriedem | true, it would be like offloading the DISK_GB management to another project which isn't untrue or unlike a 2 node job | |
| 19:33:41 | mriedem | s/project/provider/ | |
| 19:33:50 | dansmith | yeah | |
| 19:33:53 | mriedem | just don't get to test the interesting move bits | |
| 19:33:58 | dansmith | well, | |
| 19:34:05 | dansmith | single node resize is an interesting bit too | |
| 19:34:50 | mriedem | btw, we should probably make the ceph job voting again at some point... | |
| 19:34:57 | mriedem | i think it's been fairly healthy for awhile again | |
| 19:35:09 | melwitt | I am in support of that idea | |
| 19:35:13 | dansmith | just a DNM patch to set the flag and see how bad it's broken is probably important to decide if we need a "don't set this, it'll break" reno | |
| 19:35:21 | mriedem | i can probably wip that up | |
| 19:35:33 | openstackgerrit | Merged openstack/nova master: Port binding based on events during live migration https://review.openstack.org/434870 | |
| 19:35:36 | melwitt | sounds like a good plan | |
| 19:35:38 | dansmith | dear god | |
| 19:35:41 | dansmith | something merged | |
| 19:36:00 | melwitt | thanks mriedem that would be a big help | |
| 19:37:03 | mriedem | i dont' know how to look at graphite.openstack.org anymore to see if legacy-tempest-dsvm-full-devstack-plugin-ceph has been trending with the tempest-full job | |
| 19:37:20 | mriedem | and http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 is broken | |
| 19:39:41 | melwitt | me neither. could try to regenerate it with https://github.com/sdague/failopotamus but I'm not sure whether that works with zuul or not | |
| 19:40:40 | mriedem | doesn't work anymore | |
| 19:40:51 | mriedem | the graphite stats_count structure has changed so it's by project rather than by job now | |
| 19:40:55 | mriedem | i'm asking in -infra | |
| 19:40:57 | melwitt | gotcha | |
| 19:43:04 | dansmith | I assume that we're good to keep rechecking things that are already +Wd past the end of the day, given the gate yeah? | |
| 19:45:05 | melwitt | that's been my understanding | |
| 19:45:06 | openstackgerrit | Merged openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537 | |
| 19:47:53 | mriedem | LURE&title=ceph%20vs%20tempest-full%20fail%20rates%20over%203%20months | |
| 19:47:53 | mriedem | melwitt: dansmith: http://graphite.openstack.org/render/?width=1531&height=592&_salt=1532634445.928&from=00%3A00_20180428&until=23%3A59_20180726&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack_nova.master.job.legacy-tempest-dsvm-full-devstack-plugin-ceph.FAILURE&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack_nova.master.job.tempest-full | |
| 19:48:30 | mriedem | so something big got fixed the first week of may | |
| 19:49:00 | mriedem | anyway it looks pretty stable over 3 months | |
| 19:49:22 | dansmith | so we eff it up by setting the sharing flag yeah? :) | |
| 19:50:19 | melwitt | :***( | |
| 19:50:54 | dansmith | I'm joking.. turning that on should generate a constant level of fail or success | |
| 19:51:05 | dansmith | so if it totally doesn't work, then we don't merge it | |
| 19:52:41 | melwitt | mriedem: IIRC, the fixes were https://review.openstack.org/563870 and https://review.openstack.org/563313 | |
| 19:53:35 | melwitt | that was slightly before may though | |
| 19:54:57 | melwitt | or a week before | |
| 19:55:24 | mriedem | same timeframe | |
| 19:55:29 | mriedem | the query goes to April 28 | |
| 19:57:19 | melwitt | can you paste the link, it got cutoff weirdly for me trying to use it | |
| 19:58:08 | melwitt | *pastebin | |
| 19:58:53 | mriedem | https://goo.gl/WbXoW5 | |
| 19:59:19 | mriedem | there are no multinode ceph jobs except for our usage of it in the nova-live-migration job which only runs live migration tests | |
| 19:59:50 | melwitt | thanks. it's weird it cuts off the bottom part of the graph for me | |
| 20:01:21 | melwitt | https://imgur.com/a/3Fdstt9 | |
| 20:04:10 | dansmith | that's at the 1.0 mark I think | |
| 20:04:13 | dansmith | and no fails at zero | |
| 20:04:50 | dansmith | you have to null the missing points and do connected mode to get it to look right IIRC | |
| 20:05:46 | mriedem | what's gonna be the best way to get the disk info for the provider DISK_GB inventory? df -P /root ? | |
| 20:06:09 | dansmith | there's a ceph command you can run I think | |
| 20:06:19 | mriedem | oo even better | |
| 20:06:43 | mriedem | which would match the libvirt driver's reporting so i should probably use that | |
| 20:07:09 | mriedem | whatever get_cluster_stats does | |
| 20:08:04 | mriedem | http://docs.ceph.com/docs/jewel/man/8/ceph/#status ? | |
| 20:08:59 | dansmith | I dunno, I've seen people paste output of it but I don't know exactly | |
| 20:09:13 | mriedem | i'd have to get a devstack setup with the same ceph config to play with this | |
| 20:09:21 | mriedem | but seems pretty straight forward | |
| 20:09:31 | dansmith | ceph osd df -h ? | |
| 20:10:33 | dansmith | or just ceph df maybe | |
| 20:10:46 | dansmith | http://docs.ceph.com/docs/master/rados/operations/monitoring/#checking-a-cluster-s-usage-stats | |
| 20:11:07 | mriedem | cool | |