I am a brand new user to dabo.

First, I used the runtime installer yesterday, and then removed it and did the

svn install on Windows XP.

I have installed the latest svn version of dabo on Windows XP, along with

Python2.6, PIL1.1.7, psycopg2-2.4.5,reportlab2.5, and wx-2.8-msw-unicode.

I have created a new project using dabo.quickstart.

I have started to create a new form with the classdesigner and have created a

new connection to a very large postgres database that has 376 schemas and a

number of tables within each schema.

When I test the connection, it works fine.

However, when I select the saved connection and click on next, Windows gets

stuck and times out trying to get all of the table names from the postgres

database(I'm guessing?)

Yesterday, when I was using the runtime version, it was over 15 minutes waiting

for the tables to display for choosing.

Today, it is taking even longer.

Any suggestions for speeding this up? (I am stuck with having my postgres schema

in this large database).

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/loom.20120412T202724-709@post.gmane.org

On Thursday, April 12, 2012 06:46:41 PM Cheryl wrote:

> I am a brand new user to dabo.

> First, I used the runtime installer yesterday, and then removed it and did

> the svn install on Windows XP.

> I have installed the latest svn version of dabo on Windows XP, along with

> Python2.6, PIL1.1.7, psycopg2-2.4.5,reportlab2.5, and wx-2.8-msw-unicode.

> I have created a new project using dabo.quickstart.

> I have started to create a new form with the classdesigner and have created

> a new connection to a very large postgres database that has 376 schemas and

> a number of tables within each schema.

> When I test the connection, it works fine.

> However, when I select the saved connection and click on next, Windows gets

> stuck and times out trying to get all of the table names from the postgres

> database(I'm guessing?)

> Yesterday, when I was using the runtime version, it was over 15 minutes

> waiting for the tables to display for choosing.

> Today, it is taking even longer.

> Any suggestions for speeding this up? (I am stuck with having my postgres

> schema in this large database).

I sort doubt that a speed up is possible. I believe the app is gathering all

the available table and views from all the schema. For example dealing with

only 10 tables per schema means gathering information from 3760 tables. Each

of those tables are processed for field information including name, data

types, and determining if the field is primary key. And all of the data is

stored in ram.

But you could hand code the bizobj's required some like this:

class WarehousepwtranrtBizobj(dabo.biz.dBizobj):

def afterInit(self):

self.DataStructure = (

("pkid", "I", True, "warehouse.pwtranrt", "pkid"),

("fk_transid", "I", False, "warehouse.pwtranrt", "fk_transid"),

("fk_prdctid", "I",False, "warehouse.pwtranrt", "fk_prdctid"),

("mnotes", "I", False, "warehouse.pwtranrt", "mnotes"),

("nflatfee1", 'N', False, "warehouse.pwtranrt", "nflatfee1"),

("nhrate", "N", False, "warehouse.pwtranrt", "nhrate"),

("nsrate", "N", False, "warehouse.pwtranrt", "nsrate"),

("ncycledays", "I", False, "warehouse.pwtranrt", "ncycledays"),

("ccustno", "C", False,"warehouse.pwtranrt","ccustno"),

("centerby", "C", False, "warehouse.pwtranrt", "centerby"),

("nratelevel", "I", False, "warehouse.pwtranrt", "nratelevel"),

("dnextbill", "D", False, "warehouse.pwtranrt", "dnextbill"),

("nstatus", "I", False, "warehouse.pwtranrt", "nstatus"),

("dlastinvoice", "D", False, "warehouse.pwtranrt",

"dlastinvoice"),

("dpaidthru", "D", False, "warehouse.pwtranrt", "dpaidthru"),

("dentered", "D", False, "warehouse.pwtranrt", "dentered"),

("deffective", "D", False, "warehouse.pwtranrt", "deffective"),

("cuid", "C", False, "warehouse.pwtranrt", "cuid"),

("nbillby", "I", False, "warehouse.pwtranrt", "nbillby"),

("nfactor", "N", False, "warehouse.pwtranrt", "nfactor"),

)

self.DataSource = "warehouse.pwtranrt"

self.KeyField = ""

self.addFrom("warehouse.pwtranrt")

self.addField("centerby")

self.addField("mnotes")

self.addField("nstatus")

self.addField("nsrate")

self.addField("dlastinvoice")

self.addField("dpaidthru")

self.addField("nhrate")

self.addField("fk_transid")

self.addField("nbillby")

self.addField("nflatfee1")

self.addField("nfactor")

self.addField("ccustno")

self.addField("dentered")

self.addField("nratelevel")

self.addField("ncycledays")

self.addField("cuid")

self.addField("deffective")

self.addField("fk_prdctid")

self.addField("pkid")

self.addField("dnextbill")

And then use ClassDesigner to create the screen and associate the fields with

properties of a control.

Johnf

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/5585030.mSzSHqe8ld@linux-12

On Apr 12, 2012, at 1:46 PM, Cheryl wrote:

> I have created a new project using dabo.quickstart.

> I have started to create a new form with the classdesigner and have created a

> new connection to a very large postgres database that has 376 schemas and a

> number of tables within each schema.

> When I test the connection, it works fine.

> However, when I select the saved connection and click on next, Windows gets

> stuck and times out trying to get all of the table names from the postgres

> database(I'm guessing?)

> Yesterday, when I was using the runtime version, it was over 15 minutes waiting

> for the tables to display for choosing.

Instead of using the Class Designer to define your database connection, have you tried using the CxnEditor.py tool instead? It may not be terribly different, but perhaps the interaction of the modal Class Designer wizard and the CxnEditor is making the slowness worse.

-- Ed Leafe

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/70DFE8B5-564F-42FD-B290-90440C76208B@leafe.com

On Thursday, April 12, 2012 01:32:34 PM John Fabiani wrote:

> On Thursday, April 12, 2012 06:46:41 PM Cheryl wrote:

> > I am a brand new user to dabo.

> > First, I used the runtime installer yesterday, and then removed it and

> > did the svn install on Windows XP.

> > I have installed the latest svn version of dabo on Windows XP, along

> > with

> > Python2.6, PIL1.1.7, psycopg2-2.4.5,reportlab2.5, and

> > wx-2.8-msw-unicode.

> > I have created a new project using dabo.quickstart.

> > I have started to create a new form with the classdesigner and have

> > created a new connection to a very large postgres database that has 376

> > schemas and a number of tables within each schema.

> > When I test the connection, it works fine.

> > However, when I select the saved connection and click on next, Windows

> > gets stuck and times out trying to get all of the table names from the

> > postgres database(I'm guessing?)

> > Yesterday, when I was using the runtime version, it was over 15 minutes

> > waiting for the tables to display for choosing.

> > Today, it is taking even longer.

> > Any suggestions for speeding this up? (I am stuck with having my

> > postgres

> > schema in this large database).

>

> I sort doubt that a speed up is possible. I believe the app is gathering

> all the available table and views from all the schema. For example dealing

> with only 10 tables per schema means gathering information from 3760

> tables. Each of those tables are processed for field information including

> name, data types, and determining if the field is primary key. And all of

> the data is stored in ram.

You do realize that you are in a very unique situation with 3760 schema!

I decided I'd check the ClassDesigner code to see if I was right and as I

suggested the issue is processing all the table information. Just getting the

table information is relatively quick (Postgres is always quick). But after

getting the table/views names per schema the code processes each of the

tables.

From

QuickLayoutWizard->makeConnection:

tbls = crs.getTables()

for tb in tbls:

fldDict = {}

flds = crs.getFields(tb)

for fld in flds:

fldname = fld[0]

fldInfo = fldDict[fldname] = {}

fldInfo["type"] = fld[1]

fldInfo["pk"] = fld[2]

self._dataEnv[tb] = fldDict

return True

This suggest that all the tables and the table field information is in ram.

That of course take time just to retrieve and the overhead to process is high.

Then of course how much ram do you have?

So if you have 3760 schema, each with 10 tables, each with 5 fields , each

with 4 items of data - you get 734,000 items to process. Still 15 mins seems

a little long. Maybe it would be improved if you increased your ram.

Although, my experience with windows and min of ram have been very good.

Johnf

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/1707166.5FGFeJ0c94@linux-12

On Apr 12, 2012, at 4:19 PM, John Fabiani wrote:

> for tb in tbls:

> fldDict = {}

> flds = crs.getFields(tb)

> for fld in flds:

> fldname = fld[0]

> fldInfo = fldDict[fldname] = {}

> fldInfo["type"] = fld[1]

> fldInfo["pk"] = fld[2]

> self._dataEnv[tb] = fldDict

> return True

Looking at this, I'm wondering if it could be sped up a bit using a list comp:

self._dataEnv = dict([(tb, dict([(fld[0], {"type": fld[1], "pk": fld[2]})

for fld in crs.getFields(tb)]))

for tb in crs.getTables()])

I don't have any db schemas that would generate a noticeable time difference. Cheryl, if I sent you a modified version of the wizard, would you be able to test it out to see if it improves the performance for your database?

-- Ed Leafe

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/1AFA15C4-8270-4E45-BE4C-B0B21CCD409A@leafe.com

On 4/12/12 3:09 PM, Ed Leafe wrote:

> Looking at this, I'm wondering if it could be sped up a bit using a list comp:

>

> self._dataEnv = dict([(tb, dict([(fld[0], {"type": fld[1], "pk": fld[2]})

> for fld in crs.getFields(tb)]))

> for tb in crs.getTables()])

I think that could speed it up quite a bit, but is a generator expression possible

here? Because then we only consume RAM for the current iteration which should speed

things up if the performance problem is with Python needing to continuously ask the

OS for more vram.

Paul

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/4F875478.8060207@ulmcnett.com

On Apr 12, 2012, at 5:17 PM, Paul McNett wrote:

> I think that could speed it up quite a bit, but is a generator expression possible

> here? Because then we only consume RAM for the current iteration which should speed

> things up if the performance problem is with Python needing to continuously ask the

> OS for more vram.

I wrote it that way originally, but all those parens looked even more confusing. :)

Sure, I can just change the square brackets back to parens.

-- Ed Leafe

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/671C4116-068E-4BDF-A75C-B3147F9E35E8@leafe.com

Ed Leafe <ed@...> writes:

>

> I don't have any db schemas that would generate a noticeable time difference.

Cheryl, if I sent you a

> modified version of the wizard, would you be able to test it out to see if it

improves the performance for

> your database?

>

Yes. I could try out a modified version of the wizard.

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/loom.20120413T213447-647@post.gmane.org

Would it be possible to pass in a schema name to filter the list of tables that

get returned from the database?

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/loom.20120413T215054-236@post.gmane.org

On Apr 13, 2012, at 2:53 PM, Cheryl wrote:

> Would it be possible to pass in a schema name to filter the list of tables that

> get returned from the database?

Right now, no, as many DB backends don't have the same sort of schema layout that PostgreSQL does. But it might be an idea for a future improvement.

-- Ed Leafe

_______________________________________________

Post Messages to: Dabo-users@leafe.com

Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users

Searchable Archives: http://leafe.com/archives/search/dabo-users

This message: http://leafe.com/archives/byMID/59BE4E01-DE29-4717-AEDD-7C4FA1EF4763@leafe.com