| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-02 | |||
| 11:53:05 | mdbooth | johnthetubaguy: More context on the db_sync thing, looks like we had a puppet job which retried on failure, which included exceeding its timeout. Seems db sync was taking too long and we ended up running 2 simultaneously. | |
| 11:53:14 | mdbooth | So perhaps robustification might be in order. | |
| 11:53:37 | mdbooth | Perhaps the migration should hold an exclusive lock while it runs. | |
| 11:53:44 | johnthetubaguy | mdbooth: it should be all done in a single transaction, for each version, I thought? | |
| 11:54:01 | johnthetubaguy | maybe we never added that stuff in db sync | |
| 11:54:27 | johnthetubaguy | maybe thats not possible... | |
| 11:54:48 | mdbooth | Does msql honour transactions for ddl? | |
| 11:55:00 | johnthetubaguy | I was just thinking the same question, I don't know | |
| 11:55:13 | johnthetubaguy | I had always assume that, but you might be right | |
| 11:55:27 | johnthetubaguy | nothing like an olso.concurrency lock to stop that being an issue, like you say | |
| 11:55:51 | mdbooth | Interwebs say no transactional ddl | |
| 11:56:50 | mdbooth | Think it would have to be a db lock, because we can't be sure where db sync is running. | |
| 11:57:05 | mdbooth | Also, think change would have to be in sqla migrations | |
| 11:58:42 | johnthetubaguy | mdbooth: true, DB lock is safer | |
| 12:10:37 | owalsh | mdbooth: DB lock might be overkill... | |
| 12:11:05 | owalsh | the first db sync succeeded IIRC | |
| 12:11:21 | mdbooth | owalsh: I don't think so. It would only have to used for exclusion of the migration itself. | |
| 12:11:44 | mdbooth | I think it could be pretty simple to implement, tbh. | |
| 12:12:02 | openstackgerrit | Derek Higgins proposed openstack/nova master: Simplify uses of assert_has_calls https://review.openstack.org/428108 | |
| 12:12:16 | derekh | johnthetubaguy: sfinucan ^^ sorry forgot to push it up yesterday | |
| 12:12:26 | mdbooth | When you initially read the current version from the migrate_version table you'd take an exclusive lock on it. | |
| 12:12:52 | mdbooth | Then you'd exclude any other migration from starting until this once completed, but other db operations can continue. | |
| 12:13:15 | sfinucan | derekh: Sweet. Cheers :) Think it can/should wait until Ocata is cut though | |
| 12:13:42 | derekh | sfinucan: sounds fine by me, just letting ye know its there | |
| 12:20:09 | owalsh | mdbooth: hmm the oslo.db tests used to lock - https://review.openstack.org/58830 | |
| 12:20:52 | mdbooth | owalsh: A good reason to use a db lock | |
| 12:21:22 | owalsh | but why protect the tests with a lock instead of the migration? | |
| 12:21:56 | mdbooth | owalsh: I think every test runs the migrations. | |
| 12:22:01 | mdbooth | Unless... optimisation. | |
| 12:22:04 | mdbooth | But mostly. | |
| 12:26:28 | owalsh | forget that, (should have looked at the diff), it's just locking the engine reset | |
| 12:27:20 | mdbooth | owalsh: I've emailed zzzeek for an opinion. cc'd you. | |
| 12:42:21 | openstack | bug 1660878 in OpenStack Compute (nova) "test_reboot_deleted_server fails with 409 "Cannot 'reboot' instance while it is in vm_state building"" [High,In progress] https://launchpad.net/bugs/1660878 - Assigned to Matt Riedemann (mriedem) | |
| 12:42:21 | lyarwood | hopefully a stupid question, does the gate rebase changes? I keep hitting Bug 1660878 in https://review.openstack.org/#/c/427267/ and wonder if I need to manually rebase it to pick up https://review.openstack.org/#/c/427782/ | |
| 12:46:45 | lyarwood | http://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e4e67b3/logs/devstacklog.txt.gz#_2017-02-02_10_50_57_699 - okay, so we merge the change, so I should have https://review.openstack.org/#/c/427782/ | |
| 12:54:24 | openstackgerrit | John Garbutt proposed openstack/nova master: Report reserved_host_disk_mb in GB not KB https://review.openstack.org/428120 | |
| 12:55:46 | johnthetubaguy | bauzas: sfinucan: a nasty placement issue the openstack ansible guys were hitting ^ | |
| 12:55:57 | johnthetubaguy | just want to make sure I am not reading that incorrectly | |
| 13:04:25 | mdbooth | lyarwood: Yep | |
| 13:05:19 | lyarwood | mdbooth: thanks, I'll wait for dansmith and mriedem to come online before I recheck https://review.openstack.org/#/c/427267/ again | |
| 13:08:11 | mdbooth | johnthetubaguy: Do you think you'll get around to https://review.openstack.org/#/c/383859/ before release? Fixes NFS, and makes doing the same for other filesystem-backed drivers trivial. See glusterfs follow-on for eg: https://review.openstack.org/#/c/423402/ | |
| 13:08:13 | gcb | sfinucan, mriedem: As I know we have been preparing for final octata release, Can we still merge python 3 unit tests patches now ? | |
| 13:09:00 | mdbooth | johnthetubaguy: So the NFS change is somewhat complex because it includes infrastructure applicable to all filesystem-backed volume backends. | |
| 13:09:16 | sfinucan | gcb: They're pretty low risk (unit test only) but I think we're only accepting critical bugfixes and doc patches at the moment | |
| 13:09:23 | mdbooth | It's pretty well tested, though. | |
| 13:09:38 | sfinucan | The Pike window will be open tomorrow though | |
| 13:10:02 | mdbooth | It's a bug. Are we passed the bugfix window? | |
| 13:10:10 | mdbooth | past | |
| 13:10:33 | mdbooth | That said, if I can get it into Pike I'll take it and propose for backport. | |
| 13:10:49 | mdbooth | I just want to land the damned thing. | |
| 13:11:13 | gcb | sfinucan: thanks, got it. The only two patches to make all unit tests work on Python 3.5 https://review.openstack.org/#/q/topic:bug/1482633 | |
| 13:11:28 | gcb | it's okay to make them merge into Pike :) | |
| 13:11:51 | sfinucan | gcb: Perfect. I'll take a look tomorrow or Monday :) Will be good to get that all closed off | |
| 13:12:21 | gcb | sfinucan: cool, thanks | |
| 13:18:21 | sfinucan | johnthetubaguy: Took a gawk. Makes sense to me | |
| 13:24:33 | jaypipes | morning folks | |
| 13:24:35 | bauzas | mriedem: FWIW, we have another related problem | |
| 13:24:57 | openstack | Launchpad bug 1659266 in OpenStack Compute (nova) "Disk allocation for instance is not good with swap" [High,Confirmed] - Assigned to Tanvir Talukder (tanvirt16) | |
| 13:24:57 | bauzas | johnthetubaguy: https://bugs.launchpad.net/nova/+bug/1659266 | |
| 13:25:10 | bauzas | but fixing that could be difficult given we already have allocations | |
| 13:25:17 | sfinucan | trinaths: Not sure if you got a response to your question or not. Have you tried building an instance with a NUMA topology but without hugepages? | |
| 13:28:10 | mriedem | gcb: sfinucan: mdbooth: we shouldn't be putting things into the gate or check queue that don't need to be, py3 unit test cleanup included | |
| 13:28:41 | sfinucan | mriedem: Morning. That's what we thought | |
| 13:28:57 | gcb | mriedem: ack | |
| 13:31:32 | mdbooth | lyarwood: I'm just going to grab coffee. Want to talk about swap_volume in a few minutes? | |
| 13:31:43 | lyarwood | mdbooth: sure | |
| 13:31:48 | mriedem | bauzas: is that related to the thing that johnthetubaguy has a patch up for? | |
| 13:32:06 | mriedem | https://review.openstack.org/#/c/428120/ | |
| 13:32:47 | sfinucan | mriedem, bauzas: That's the one, yes | |
| 13:33:04 | bauzas | mriedem: not really but the same problme | |
| 13:33:20 | bauzas | mriedem: honestly, MHO is that we should not use GBs for disk resources but rather bytes | |
| 13:33:37 | bauzas | or KB | |
| 13:33:43 | bauzas | if that's too big | |
| 13:33:54 | bauzas | but, we have lots of ceilings like the one I wrote | |
| 13:34:11 | mriedem | dansmith: melwitt: hmm, lyarwood's patch failed in the gate on the cells inst delete/create race thing http://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e4e67b3/console.html#_2017-02-02_11_48_07_755243 | |
| 13:34:28 | bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L148 | |
| 13:34:49 | mriedem | https://review.openstack.org/#/c/427782/ | |
| 13:35:07 | mriedem | which merged at 7:34pm | |
| 13:35:29 | mriedem | and lyarwood's change failed at 2017-02-02 11:48:07.755243 | |
| 13:35:51 | lyarwood | mriedem: twice, I think the previous run also hit this a few hours before that | |
| 13:36:07 | mriedem | lyarwood: well i could see it hitting before the other fix was merged, | |
| 13:36:14 | mriedem | but i'm worried if it hit after the fix was merged | |
| 13:36:37 | mriedem | i think ES is backed up too so we probably can't tell if the hits have dropped off after the fix merged | |
| 13:36:49 | mriedem | yeah we can't tell http://status.openstack.org//elastic-recheck/index.html | |
| 13:36:52 | mriedem | everything is backed up | |
| 13:37:25 | lyarwood | mriedem: http://logs.openstack.org/67/427267/2/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/cc0ed47/console.html#_2017-02-02_05_45_00_341186 was the other hit | |
| 13:39:18 | mriedem | yeah that's before the fix was merged | |
| 13:39:21 | mriedem | so not a surprise | |
| 13:39:32 | mriedem | but failing 3 hours after the merge worries me | |
| 13:39:41 | lyarwood | mriedem: ah did it land at 7:34am? | |
| 13:39:47 | mriedem | pm | |
| 13:39:51 | lyarwood | mriedem: you said pm above | |
| 13:39:58 | lyarwood | mriedem: that's 05:45am | |
| 13:40:02 | mriedem | oh yeah | |
| 13:40:12 | lyarwood | anyway | |
| 13:41:18 | nmathew- | hi all | |
| 13:41:35 | nmathew- | can anybody point to a material on flavors | |
| 13:41:38 | mriedem | lyarwood: apparently the times in the zuul log don't match up with what's in the gerrit ui | |
| 13:41:55 | mriedem | nmathew-: http://docs.openstack.org/admin-guide/compute-flavors.html | |
| 13:42:00 | mriedem | docs.o.o has a search box | |