ZPack update

the updated tools/source can be downloaded here:
http://bgb-sys.sourceforge.net/zpack_0_1.zip

as before, comments would be appriciated.
there may yet be more updates in the future.


partly, a lot has been going on recently and I have not been able to put
much time into this.

of note, a few additions have been made to the format, in particular:

multiple forks. though presently unused in the implementation, these may be
used later for adding additional metadata to files (for example, names
longer than 32 chars, or possible os/implementation specific metadata).

for very small files, packing the data directly into the directory entries
(this is currently limited to about 20 bytes). in this way, very small (and
typically uncompressible) files can be stored absent needing to locate space
elsewhere in the image, and saving an entry in the spans tables.

directory indexing. this is considered presently as a possible/optional
feature. in this case, an avl tree may be built for specific directories and
used during lookup operations. as this format is intended for use in
persistent stores, there is a high probability of single directories with a
very large number of entries (several M files or more), so in this case,
building an indexing structure and storing it to disk may make sense.

another possibility includes claiming that the directory is indexed, but not
storing the index structure itself. this will indicate instead to build the
index structure at load time, which may make the most sense for directories
containing a few k files or less...

note, I am thinking of implementing avl trees here as I doubt that there
will be sufficient file counts or performance requirements to demand
implementing a b-tree (in these cases, it would likely make more sense to
implement a custom format anyways).

likewise, b-trees are less likely to play well with the (byte oriented)
nature of the format (most implementations of b-trees I have seen have been
block-oriented). putting a block structure on top of a byte-oriented format
imo seems rather inelegant...


another thing is that of also the possibility of storing the spans tree to
disk, which may be helpful in the possible case of large images (> 100k or
1M files). at present, my test images involve a small enough number of files
(5 to 10k) that storing this index is unlikely to be all that helpful, but
it does remain as a possibility...

I guess the limit is how many files to expect in an image, and how to expect
them to be distributed. assuming < 100k may well be sufficient.


.

Relevant Pages

  • Re: Standards question on edited fields
    ... > Who said I was storing it? ... and it seems you would not present such a format to ... it's not a sloppy design at all - and there is a valid reason. ... Judson McClendon ...
    (comp.lang.cobol)
  • Re: 2D height map to 3D model?
    ... (you're storing 12 doubles for almost every data-point). ... If you have a format that allows for triangle/quadrilateral strips, ... can make the rendering far more efficient using them. ...
    (comp.lang.python)
  • Re: Increment number based on list box
    ... I say 'number' because the format you're using is not numerical. ... So what you have to be doing is storing the number and then formating it ... You can have a table to store just the one value and keep incrementing it. ... Is there a macro that can accomplish this ...
    (microsoft.public.access.forms)
  • Re: [9fans] Scaleable mail repositories.
    ... It's easy to write file servers, but that does not mean that it's the ... it's easier to store the mail in that format. ... the code necessary to handle your storage and index structure becomes ... rather than maintaining a separate index file or files, ...
    (comp.os.plan9)
  • Re: Basic Date question
    ... Leaving it uninitialized is ... >You might also considering storing date in the numerical date format. ... I thought about that but we're going to use Crystal Reports and SQL ... Reporting Services and those aren't going to like that format. ...
    (microsoft.public.dotnet.languages.vb)