| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-26 | |||
| 19:16:40 | mriedem | https://review.openstack.org/#/c/560459/ | |
| 19:17:11 | mriedem | well, we should, | |
| 19:17:20 | mriedem | the allocations could go against the provider that has the DISK_GB inventory | |
| 19:17:22 | dansmith | I'm not sure that move ops need to change? | |
| 19:17:31 | mriedem | which is in the MISC_SHARES_VIA_AGGREGATE relationship | |
| 19:17:34 | dansmith | right | |
| 19:17:48 | dansmith | so I'm not sure what you're saying... | |
| 19:17:49 | mriedem | there was something i look up every time this comes up that i know is broken | |
| 19:18:01 | mriedem | i've beaten cdent over the head with a few times already :) | |
| 19:18:32 | dansmith | have we updated the ceph job to do the right thing? | |
| 19:18:41 | dansmith | that'd be a good way to either show it's broken or show that it works | |
| 19:18:53 | dansmith | and you should be able to osc-placement one-shot yourself a sharing provider during setup | |
| 19:19:00 | mriedem | nope, and that's what i suggested in one of the placement update ML threads | |
| 19:19:05 | dansmith | okay | |
| 19:19:17 | dansmith | not sure why we merged that patch without that testing then | |
| 19:19:34 | dansmith | ^ that was rhetorical :D | |
| 19:20:51 | dansmith | mriedem: you're still trying to remember what the missing thing is right? | |
| 19:21:21 | mriedem | well, this is one https://github.com/openstack/nova/blob/master/nova/conductor/tasks/migrate.py#L57 | |
| 19:21:48 | dansmith | ooh right | |
| 19:22:03 | mriedem | and _revert_allocation in compute | |
| 19:22:33 | mriedem | resize to same host probably doesn't handle it either, not sure | |
| 19:22:34 | dansmith | yeah, I remember now.. I was thinking those were only problems we had with the doubling approach, but I remember now | |
| 19:22:39 | mriedem | anywho, | |
| 19:22:52 | mriedem | CI testing with ceph + modeling shared is a todo for the ptg etherpad most likely | |
| 19:22:56 | mriedem | i'll add it | |
| 19:23:10 | dansmith | yeah, that probably needs to be a priority for someone, | |
| 19:23:30 | dansmith | because right now if you associate a sharing provider, things are going to break in unhelpful ways | |
| 19:23:33 | mriedem | i'll assign it to my cat | |
| 19:23:55 | dansmith | melwitt: ^ | |
| 19:24:08 | mriedem | yeah - that's why i've tried to temper enthusiasm / communication that "now it's fixed" | |
| 19:24:17 | mriedem | also because i'm a debby downer | |
| 19:24:58 | dansmith | fair enough, I just saw it had merged when I was triaging a bug related to that stuff | |
| 19:25:05 | dansmith | and figured it had actually been finished | |
| 19:25:29 | melwitt | I've been summoned for ceph stuff eh? | |
| 19:25:39 | dansmith | melwitt: no, for ptl stuff | |
| 19:26:11 | melwitt | oh, thought you were referring to the ceph CI testing add | |
| 19:26:17 | dansmith | I am, | |
| 19:26:41 | dansmith | I'm saying we probably need to be making sure that the test gets updated to validate whether this created a worse hole, | |
| 19:26:50 | dansmith | and maybe decide if we want to either patch that out, | |
| 19:26:54 | dansmith | or reno a known issue or whatever | |
| 19:26:55 | purplerbot | <mriedem> i've beaten cdent over the head with a few times already :) [2018-07-26 19:18:01.246831] [n 1fky] | |
| 19:26:55 | cdent | mriedem: on [t 1fky] | |
| 19:27:07 | melwitt | dansmith: I see, okay | |
| 19:27:35 | cdent | I'm trying to make sure that people are aware, but I'm not sure anybody really reads the weekly pupdates and I'm only willing to take full responsibility for the placement side (where it's ready) | |
| 19:28:03 | cdent | but I agree we have some holes to fill before much more time passes | |
| 19:28:54 | mriedem | L65 https://etherpad.openstack.org/p/nova-ptg-stein | |
| 19:29:21 | mriedem | i will check to see if we have an existing ceph multinode job, but i know the nova-live-migration job is multinode and re-configures itself to use ceph in the 2nd part | |
| 19:29:36 | mriedem | but live migration is less interesting since we don't do claims in the compute or anything | |
| 19:30:19 | dansmith | we really just need a multinode ceph job that boots an instance for base level verification, | |
| 19:30:26 | dansmith | but yeah it should do a cold migration at least | |
| 19:30:40 | dansmith | can we just convert the ceph job to multinode? | |
| 19:31:31 | mriedem | we can do anything we want | |
| 19:31:48 | cdent | anything? | |
| 19:31:49 | mriedem | but i'd rather re-use something than add yet another job | |
| 19:32:04 | mriedem | cdent: with zuul all things are possible (tm) | |
| 19:32:05 | dansmith | mriedem: well just converting the existing one to multinode I mean | |
| 19:32:12 | mriedem | dansmith: i know | |
| 19:32:36 | cdent | chris-runs-around-naked -1 | |
| 19:33:08 | dansmith | heck, even just the ceph job as a single node with the sharing flag would be enough to shake out some of it | |
| 19:33:33 | mriedem | true, it would be like offloading the DISK_GB management to another project which isn't untrue or unlike a 2 node job | |
| 19:33:41 | mriedem | s/project/provider/ | |
| 19:33:50 | dansmith | yeah | |
| 19:33:53 | mriedem | just don't get to test the interesting move bits | |
| 19:33:58 | dansmith | well, | |
| 19:34:05 | dansmith | single node resize is an interesting bit too | |
| 19:34:50 | mriedem | btw, we should probably make the ceph job voting again at some point... | |
| 19:34:57 | mriedem | i think it's been fairly healthy for awhile again | |
| 19:35:09 | melwitt | I am in support of that idea | |
| 19:35:13 | dansmith | just a DNM patch to set the flag and see how bad it's broken is probably important to decide if we need a "don't set this, it'll break" reno | |
| 19:35:21 | mriedem | i can probably wip that up | |
| 19:35:33 | openstackgerrit | Merged openstack/nova master: Port binding based on events during live migration https://review.openstack.org/434870 | |
| 19:35:36 | melwitt | sounds like a good plan | |
| 19:35:38 | dansmith | dear god | |
| 19:35:41 | dansmith | something merged | |
| 19:36:00 | melwitt | thanks mriedem that would be a big help | |
| 19:37:03 | mriedem | i dont' know how to look at graphite.openstack.org anymore to see if legacy-tempest-dsvm-full-devstack-plugin-ceph has been trending with the tempest-full job | |
| 19:37:20 | mriedem | and http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 is broken | |
| 19:39:41 | melwitt | me neither. could try to regenerate it with https://github.com/sdague/failopotamus but I'm not sure whether that works with zuul or not | |
| 19:40:40 | mriedem | doesn't work anymore | |
| 19:40:51 | mriedem | the graphite stats_count structure has changed so it's by project rather than by job now | |
| 19:40:55 | mriedem | i'm asking in -infra | |
| 19:40:57 | melwitt | gotcha | |
| 19:43:04 | dansmith | I assume that we're good to keep rechecking things that are already +Wd past the end of the day, given the gate yeah? | |
| 19:45:05 | melwitt | that's been my understanding | |
| 19:45:06 | openstackgerrit | Merged openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537 | |
| 19:47:53 | mriedem | LURE&title=ceph%20vs%20tempest-full%20fail%20rates%20over%203%20months | |
| 19:47:53 | mriedem | melwitt: dansmith: http://graphite.openstack.org/render/?width=1531&height=592&_salt=1532634445.928&from=00%3A00_20180428&until=23%3A59_20180726&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack_nova.master.job.legacy-tempest-dsvm-full-devstack-plugin-ceph.FAILURE&target=stats_counts.zuul.tenant.openstack.pipeline.check.project.git_openstack_org.openstack_nova.master.job.tempest-full | |
| 19:48:30 | mriedem | so something big got fixed the first week of may | |
| 19:49:00 | mriedem | anyway it looks pretty stable over 3 months | |
| 19:49:22 | dansmith | so we eff it up by setting the sharing flag yeah? :) | |
| 19:50:19 | melwitt | :***( | |
| 19:50:54 | dansmith | I'm joking.. turning that on should generate a constant level of fail or success | |
| 19:51:05 | dansmith | so if it totally doesn't work, then we don't merge it | |
| 19:52:41 | melwitt | mriedem: IIRC, the fixes were https://review.openstack.org/563870 and https://review.openstack.org/563313 | |
| 19:53:35 | melwitt | that was slightly before may though | |
| 19:54:57 | melwitt | or a week before | |
| 19:55:24 | mriedem | same timeframe | |
| 19:55:29 | mriedem | the query goes to April 28 | |
| 19:57:19 | melwitt | can you paste the link, it got cutoff weirdly for me trying to use it | |
| 19:58:08 | melwitt | *pastebin | |