| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-26 | |||
| 19:34:50 | mriedem | btw, we should probably make the ceph job voting again at some point... | |
| 19:34:57 | mriedem | i think it's been fairly healthy for awhile again | |
| 19:35:09 | melwitt | I am in support of that idea | |
| 19:35:13 | dansmith | just a DNM patch to set the flag and see how bad it's broken is probably important to decide if we need a "don't set this, it'll break" reno | |
| 19:35:21 | mriedem | i can probably wip that up | |
| 19:35:33 | openstackgerrit | Merged openstack/nova master: Port binding based on events during live migration https://review.openstack.org/434870 | |
| 19:35:36 | melwitt | sounds like a good plan | |
| 19:35:38 | dansmith | dear god | |
| 19:35:41 | dansmith | something merged | |
| 19:36:00 | melwitt | thanks mriedem that would be a big help | |
| 19:37:03 | mriedem | i dont' know how to look at graphite.openstack.org anymore to see if legacy-tempest-dsvm-full-devstack-plugin-ceph has been trending with the tempest-full job | |
| 19:37:20 | mriedem | and http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 is broken | |
| 19:39:41 | melwitt | me neither. could try to regenerate it with https://github.com/sdague/failopotamus but I'm not sure whether that works with zuul or not | |
| 19:40:40 | mriedem | doesn't work anymore | |
| 19:40:51 | mriedem | the graphite stats_count structure has changed so it's by project rather than by job now | |
| 19:40:55 | mriedem | i'm asking in -infra | |
| 19:40:57 | melwitt | gotcha | |
| 19:43:04 | dansmith | I assume that we're good to keep rechecking things that are already +Wd past the end of the day, given the gate yeah? | |
| 19:45:05 | melwitt | that's been my understanding | |
| 19:45:06 | openstackgerrit | Merged openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537 | |
| 19:47:53 | mriedem | LURE&title=ceph%20vs%20tempest-full%20fail%20rates%20over%203%20months | |
| 19:47:53 | mriedem | melwitt: dansmith: http://graphite.openstack.org/render/?width=1531&height=592&_salt=1532634445.928&from=00%3A00_20180428&until=23%3A59_20180726&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack_nova.master.job.legacy-tempest-dsvm-full-devstack-plugin-ceph.FAILURE&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack_nova.master.job.tempest-full | |
| 19:48:30 | mriedem | so something big got fixed the first week of may | |
| 19:49:00 | mriedem | anyway it looks pretty stable over 3 months | |
| 19:49:22 | dansmith | so we eff it up by setting the sharing flag yeah? :) | |
| 19:50:19 | melwitt | :***( | |
| 19:50:54 | dansmith | I'm joking.. turning that on should generate a constant level of fail or success | |
| 19:51:05 | dansmith | so if it totally doesn't work, then we don't merge it | |
| 19:52:41 | melwitt | mriedem: IIRC, the fixes were https://review.openstack.org/563870 and https://review.openstack.org/563313 | |
| 19:53:35 | melwitt | that was slightly before may though | |
| 19:54:57 | melwitt | or a week before | |
| 19:55:24 | mriedem | same timeframe | |
| 19:55:29 | mriedem | the query goes to April 28 | |
| 19:57:19 | melwitt | can you paste the link, it got cutoff weirdly for me trying to use it | |
| 19:58:08 | melwitt | *pastebin | |
| 19:58:53 | mriedem | https://goo.gl/WbXoW5 | |
| 19:59:19 | mriedem | there are no multinode ceph jobs except for our usage of it in the nova-live-migration job which only runs live migration tests | |
| 19:59:50 | melwitt | thanks. it's weird it cuts off the bottom part of the graph for me | |
| 20:01:21 | melwitt | https://imgur.com/a/3Fdstt9 | |
| 20:04:10 | dansmith | that's at the 1.0 mark I think | |
| 20:04:13 | dansmith | and no fails at zero | |
| 20:04:50 | dansmith | you have to null the missing points and do connected mode to get it to look right IIRC | |
| 20:05:46 | mriedem | what's gonna be the best way to get the disk info for the provider DISK_GB inventory? df -P /root ? | |
| 20:06:09 | dansmith | there's a ceph command you can run I think | |
| 20:06:19 | mriedem | oo even better | |
| 20:06:43 | mriedem | which would match the libvirt driver's reporting so i should probably use that | |
| 20:07:09 | mriedem | whatever get_cluster_stats does | |
| 20:08:04 | mriedem | http://docs.ceph.com/docs/jewel/man/8/ceph/#status ? | |
| 20:08:59 | dansmith | I dunno, I've seen people paste output of it but I don't know exactly | |
| 20:09:13 | mriedem | i'd have to get a devstack setup with the same ceph config to play with this | |
| 20:09:21 | mriedem | but seems pretty straight forward | |
| 20:09:31 | dansmith | ceph osd df -h ? | |
| 20:10:33 | dansmith | or just ceph df maybe | |
| 20:10:46 | dansmith | http://docs.ceph.com/docs/master/rados/operations/monitoring/#checking-a-cluster-s-usage-stats | |
| 20:11:07 | mriedem | cool | |
| 20:11:27 | openstackgerrit | melanie witt proposed openstack/os-vif stable/rocky: Add vif_plug_noop to setup.cfg packages https://review.openstack.org/586340 | |
| 20:13:12 | mriedem | well i shall start the stack while writing my monthly report... | |
| 20:14:12 | melwitt | how can I tell who is stable/core for $project? I haven't noticed a way to figure that out for libs like os-vif | |
| 20:14:19 | melwitt | figure out via gerrit | |
| 20:16:07 | mriedem | people | |
| 20:16:19 | mriedem | oh well, | |
| 20:17:26 | mriedem | i thought it was in project-config but can't find it | |
| 20:17:57 | melwitt | ok, just curious. since I keep asking, was wondering if there was a way to avoid bugging people | |
| 20:18:24 | melwitt | I assume for os-vif it's the same stable team as for nova | |
| 20:18:55 | melwitt | I need stable review for a FFE release of os-vif https://review.openstack.org/586340 | |
| 20:19:44 | mriedem | oh i found it | |
| 20:20:07 | mriedem | https://github.com/openstack-infra/project-config/blob/master/gerrit/acls/openstack/os-vif.config | |
| 20:20:39 | melwitt | cool! thanks | |
| 20:20:53 | mriedem | +2 | |
| 20:21:21 | melwitt | thanks 2x | |
| 20:23:12 | melwitt | dansmith: plz https://review.openstack.org/586340 | |
| 20:24:57 | mriedem | commence to stacking | |
| 20:36:31 | melwitt | thx dansmith | |
| 20:36:54 | dansmith | aye | |
| 20:37:49 | melwitt | mriedem: fyi https://review.openstack.org/586307 sorry I didn't get your ack before osc-placement got released | |
| 20:38:08 | melwitt | I meant to | |
| 20:40:37 | mriedem | doesn't that need to branch? | |
| 20:41:21 | melwitt | are we supposed to branch manually in the release? I thought that would happen automatically | |
| 20:41:33 | mriedem | no it's a designated thing in the deliverable yaml | |
| 20:41:46 | mriedem | i.e. what was removed on the left side https://review.openstack.org/#/c/586307/1/deliverables/rocky/osc-placement.yaml | |
| 20:41:55 | mriedem | you'd need that for 1.3.0 to be where they branch stable/rocky | |
| 20:42:07 | melwitt | gdi | |
| 20:42:31 | mriedem | i don't know what "And it removes the stable/rocky branch that had to be manually deleted because of the mis-categorization of the library as a non-client." all means but that's what i've done in the past | |
| 20:43:13 | melwitt | what happened was that osc-placement has been categorized as a 'non-client library' up to now, | |
| 20:43:25 | mriedem | oh i see https://github.com/openstack/releases/commit/8dadc73479e3c303c4f5ee3975628c3e333d61cc#diff-64bede2e011f3f457420a6313880e97b | |
| 20:43:27 | mriedem | so smcginnis did it for you | |
| 20:44:11 | mriedem | i did it in queens https://github.com/openstack/releases/commit/a5a77d24664a2502c68cc5b0cba17db8373499c2#diff-b9ac789921c8d258691218b7fcbeaefb | |
| 20:44:13 | melwitt | and I had asked around in -releases about whether it should be considered a client library since it's a osc plugin, we agreed it should be a client lib and smcginnis had meant to omit it in the branch cutting that happened last week for non client libs, and that I would propose a change to the library 'type' when I released it this week | |
| 20:44:41 | melwitt | so, they ended up deleting the stable/rocky branch for us to go ahead and do the release today as a client library | |
| 20:45:11 | mriedem | yup so i guess push a patch to branch stable/rocky at 1.3.0 | |
| 20:45:24 | mriedem | have you done a python-novaclient release? | |
| 20:45:37 | mriedem | b/c same thing there | |
| 20:45:47 | melwitt | sounds like it, I'll ask in -releases. it looks like this branch thing might be part of their (release team) process | |
| 20:45:55 | melwitt | not yet, putting that together now | |
| 20:45:59 | mriedem | i've always done it myself | |
| 20:46:34 | mriedem | i never trusted smcginnis touching my sweet sweet deliverables | |
| 20:47:13 | melwitt | heh :) | |
| 20:48:37 | melwitt | yeah, that was his 'clean up after people' script that did it. I'm a terrible release proposer | |
| 20:49:54 | smcginnis | :) | |
| 20:50:24 | smcginnis | I've thought of so many responses to that. None of which I feel I should write here. | |