Re: Large dataset performance

On Mar 20, 10:53 pm, "jma" <junkmailav...@xxxxxxxxx> wrote:
Hello Cimode,

first let me say that you're right I am far away from being literate
in SQL.
Then you know what to do. Understand better set operations on SQL and
read books about relational model that will help you.

Introduction to database systems - CJ DATE

However, consider that I am simulating the way a client
application would be writing data to the database and this cannot
change... I would really love to read directly from a binary stream
into a db table, so that the performance would be adequate. Such a
solution would fit me very well, since my data have a persistent
pattern, for example int,double,double,double. Anyway, thanks for your
remark
Any practice or implementation should first be guided by a good grasp
of underlying principles. The chances of success in building decent
systems without knowledge is close to NULL (One *may* get lucky
sometimes).

Hope this helps...


BR

jma

Ï/Ç Cimode Ýãñáøå:

On 20 mar, 16:39, "jma" <junkmailav...@xxxxxxxxx> wrote:
<<I would like your opinion and experience on the performance of
writing
large datasets. I am writing in one loop about 3.5 million rows where
each row is an integer and 3 doubles to an Interbase db. All in all
it's about 100MB. My time measurement was 210 seconds. Is this normal?
To me it appears as a veeerryyy long time.... >>
The principle reason I see is the *Looping* algirythmics which is not
what a db does best. I suggest you learn better the power of set
operation through a better mastery of good ol SQL...Hope this helps...


.

Relevant Pages

  • Re: How to understand huge code
    ... product) I feel very difficult understanding the complete code flow. ... writing it in a semi-programming language helps reduce clutter. ... For things like SQL, I ask myself what the SQL is retrieving ...
    (comp.programming)
  • Re: Is there a way to keep the StreamWriter open?
    ... SQL that I can writing to a text file with some decient programming ... hardware and it would be able to keep up but not be faster. ... It is, in some ways, but the process writing the IIS logs is optimized ...
    (microsoft.public.dotnet.framework.aspnet)
  • Re: NEED TO GO FASTER
    ... Writing your entire message in UPPER CASE is considered shouting, ... entirely written in upper case, then it is very had to read (especially ... SQL is a set oriented language, ... Make sure the proper indexes are in place. ...
    (comp.databases.ms-sqlserver)
  • Re: Querying a DataSet/DataTable in .NET
    ... to writing an in-memory database engine. ... >>>I think it would be nice if we could have a SQL engine available on the ... >>> display only (I could put it in business objects so I can reuse the ...
    (microsoft.public.dotnet.general)
  • Re: Monthly update scheme
    ... SQL Server does not have an UPSERT or MERGE command, ... it in SQL 2005 with set operations ... UPDATE Target FROM (source INTERSECT Target); ...
    (comp.databases)