| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-26 | |||
| 19:26:55 | cdent | mriedem: on [t 1fky] | |
| 19:26:55 | purplerbot | <mriedem> i've beaten cdent over the head with a few times already :) [2018-07-26 19:18:01.246831] [n 1fky] | |
| 19:27:07 | melwitt | dansmith: I see, okay | |
| 19:27:35 | cdent | I'm trying to make sure that people are aware, but I'm not sure anybody really reads the weekly pupdates and I'm only willing to take full responsibility for the placement side (where it's ready) | |
| 19:28:03 | cdent | but I agree we have some holes to fill before much more time passes | |
| 19:28:54 | mriedem | L65 https://etherpad.openstack.org/p/nova-ptg-stein | |
| 19:29:21 | mriedem | i will check to see if we have an existing ceph multinode job, but i know the nova-live-migration job is multinode and re-configures itself to use ceph in the 2nd part | |
| 19:29:36 | mriedem | but live migration is less interesting since we don't do claims in the compute or anything | |
| 19:30:19 | dansmith | we really just need a multinode ceph job that boots an instance for base level verification, | |
| 19:30:26 | dansmith | but yeah it should do a cold migration at least | |
| 19:30:40 | dansmith | can we just convert the ceph job to multinode? | |
| 19:31:31 | mriedem | we can do anything we want | |
| 19:31:48 | cdent | anything? | |
| 19:31:49 | mriedem | but i'd rather re-use something than add yet another job | |
| 19:32:04 | mriedem | cdent: with zuul all things are possible (tm) | |
| 19:32:05 | dansmith | mriedem: well just converting the existing one to multinode I mean | |
| 19:32:12 | mriedem | dansmith: i know | |
| 19:32:36 | cdent | chris-runs-around-naked -1 | |
| 19:33:08 | dansmith | heck, even just the ceph job as a single node with the sharing flag would be enough to shake out some of it | |
| 19:33:33 | mriedem | true, it would be like offloading the DISK_GB management to another project which isn't untrue or unlike a 2 node job | |
| 19:33:41 | mriedem | s/project/provider/ | |
| 19:33:50 | dansmith | yeah | |
| 19:33:53 | mriedem | just don't get to test the interesting move bits | |
| 19:33:58 | dansmith | well, | |
| 19:34:05 | dansmith | single node resize is an interesting bit too | |
| 19:34:50 | mriedem | btw, we should probably make the ceph job voting again at some point... | |
| 19:34:57 | mriedem | i think it's been fairly healthy for awhile again | |
| 19:35:09 | melwitt | I am in support of that idea | |
| 19:35:13 | dansmith | just a DNM patch to set the flag and see how bad it's broken is probably important to decide if we need a "don't set this, it'll break" reno | |
| 19:35:21 | mriedem | i can probably wip that up | |
| 19:35:33 | openstackgerrit | Merged openstack/nova master: Port binding based on events during live migration https://review.openstack.org/434870 | |
| 19:35:36 | melwitt | sounds like a good plan | |
| 19:35:38 | dansmith | dear god | |
| 19:35:41 | dansmith | something merged | |
| 19:36:00 | melwitt | thanks mriedem that would be a big help | |
| 19:37:03 | mriedem | i dont' know how to look at graphite.openstack.org anymore to see if legacy-tempest-dsvm-full-devstack-plugin-ceph has been trending with the tempest-full job | |
| 19:37:20 | mriedem | and http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 is broken | |
| 19:39:41 | melwitt | me neither. could try to regenerate it with https://github.com/sdague/failopotamus but I'm not sure whether that works with zuul or not | |
| 19:40:40 | mriedem | doesn't work anymore | |
| 19:40:51 | mriedem | the graphite stats_count structure has changed so it's by project rather than by job now | |
| 19:40:55 | mriedem | i'm asking in -infra | |
| 19:40:57 | melwitt | gotcha | |
| 19:43:04 | dansmith | I assume that we're good to keep rechecking things that are already +Wd past the end of the day, given the gate yeah? | |
| 19:45:05 | melwitt | that's been my understanding | |
| 19:45:06 | openstackgerrit | Merged openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537 | |
| 19:47:53 | mriedem | melwitt: dansmith: http://graphite.openstack.org/render/?width=1531&height=592&_salt=1532634445.928&from=00%3A00_20180428&until=23%3A59_20180726&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack_nova.master.job.legacy-tempest-dsvm-full-devstack-plugin-ceph.FAILURE&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack_nova.master.job.tempest-full | |
| 19:47:53 | mriedem | LURE&title=ceph%20vs%20tempest-full%20fail%20rates%20over%203%20months | |
| 19:48:30 | mriedem | so something big got fixed the first week of may | |
| 19:49:00 | mriedem | anyway it looks pretty stable over 3 months | |
| 19:49:22 | dansmith | so we eff it up by setting the sharing flag yeah? :) | |
| 19:50:19 | melwitt | :***( | |
| 19:50:54 | dansmith | I'm joking.. turning that on should generate a constant level of fail or success | |
| 19:51:05 | dansmith | so if it totally doesn't work, then we don't merge it | |
| 19:52:41 | melwitt | mriedem: IIRC, the fixes were https://review.openstack.org/563870 and https://review.openstack.org/563313 | |
| 19:53:35 | melwitt | that was slightly before may though | |
| 19:54:57 | melwitt | or a week before | |
| 19:55:24 | mriedem | same timeframe | |
| 19:55:29 | mriedem | the query goes to April 28 | |
| 19:57:19 | melwitt | can you paste the link, it got cutoff weirdly for me trying to use it | |
| 19:58:08 | melwitt | *pastebin | |
| 19:58:53 | mriedem | https://goo.gl/WbXoW5 | |
| 19:59:19 | mriedem | there are no multinode ceph jobs except for our usage of it in the nova-live-migration job which only runs live migration tests | |
| 19:59:50 | melwitt | thanks. it's weird it cuts off the bottom part of the graph for me | |
| 20:01:21 | melwitt | https://imgur.com/a/3Fdstt9 | |
| 20:04:10 | dansmith | that's at the 1.0 mark I think | |
| 20:04:13 | dansmith | and no fails at zero | |
| 20:04:50 | dansmith | you have to null the missing points and do connected mode to get it to look right IIRC | |
| 20:05:46 | mriedem | what's gonna be the best way to get the disk info for the provider DISK_GB inventory? df -P /root ? | |
| 20:06:09 | dansmith | there's a ceph command you can run I think | |
| 20:06:19 | mriedem | oo even better | |
| 20:06:43 | mriedem | which would match the libvirt driver's reporting so i should probably use that | |
| 20:07:09 | mriedem | whatever get_cluster_stats does | |
| 20:08:04 | mriedem | http://docs.ceph.com/docs/jewel/man/8/ceph/#status ? | |
| 20:08:59 | dansmith | I dunno, I've seen people paste output of it but I don't know exactly | |
| 20:09:13 | mriedem | i'd have to get a devstack setup with the same ceph config to play with this | |
| 20:09:21 | mriedem | but seems pretty straight forward | |
| 20:09:31 | dansmith | ceph osd df -h ? | |
| 20:10:33 | dansmith | or just ceph df maybe | |
| 20:10:46 | dansmith | http://docs.ceph.com/docs/master/rados/operations/monitoring/#checking-a-cluster-s-usage-stats | |
| 20:11:07 | mriedem | cool | |
| 20:11:27 | openstackgerrit | melanie witt proposed openstack/os-vif stable/rocky: Add vif_plug_noop to setup.cfg packages https://review.openstack.org/586340 | |
| 20:13:12 | mriedem | well i shall start the stack while writing my monthly report... | |
| 20:14:12 | melwitt | how can I tell who is stable/core for $project? I haven't noticed a way to figure that out for libs like os-vif | |
| 20:14:19 | melwitt | figure out via gerrit | |
| 20:16:07 | mriedem | people | |
| 20:16:19 | mriedem | oh well, | |
| 20:17:26 | mriedem | i thought it was in project-config but can't find it | |
| 20:17:57 | melwitt | ok, just curious. since I keep asking, was wondering if there was a way to avoid bugging people | |
| 20:18:24 | melwitt | I assume for os-vif it's the same stable team as for nova | |
| 20:18:55 | melwitt | I need stable review for a FFE release of os-vif https://review.openstack.org/586340 | |
| 20:19:44 | mriedem | oh i found it | |
| 20:20:07 | mriedem | https://github.com/openstack-infra/project-config/blob/master/gerrit/acls/openstack/os-vif.config | |
| 20:20:39 | melwitt | cool! thanks | |
| 20:20:53 | mriedem | +2 | |
| 20:21:21 | melwitt | thanks 2x | |
| 20:23:12 | melwitt | dansmith: plz https://review.openstack.org/586340 | |
| 20:24:57 | mriedem | commence to stacking | |
| 20:36:31 | melwitt | thx dansmith | |
| 20:36:54 | dansmith | aye | |
| 20:37:49 | melwitt | mriedem: fyi https://review.openstack.org/586307 sorry I didn't get your ack before osc-placement got released | |