Re: Big volumes, small files

On Apr 18, 9:48 am, the.ak...@xxxxxxxxx wrote:
Hello,

I'd like to plan a new storage solution for a system currently in
production.

The system's storage is based on code which writes many files to the
file system, with overall storage needs currently around 40TB and
expected to reach hundreds of TBs. The average file size of the system
is ~100K, which translates to ~500 million files today, and billions
of files in the future. This storage is accessed over NFS by a rack of
40 Linux blades, and is mostly read-only (99% of the activity is
reads). While I realize calling this sub-optimal system design is
probably an understatement, the design of the system is beyond my
control and isn't likely to change in the near future.

The system's current storage is based on 4 VxFS filesystems, created
on SVM meta-devices each ~10TB in size. A 2-node Sun Cluster serves
the filesystems, 2 filesystems per node. Each of the filesystems
undergoes growfs as more storage is made available. We're looking for
an alternative solution, in an attempt to improve performance and
ability to recover from disasters (fsck on 2^42 files isn't practical,
and I'm getting pretty worried due to this fact - even the smallest
filesystem inconsistency will leave me lots of useless bits).

Question is - can someone with experience with large filesystems and
many small-files share his stories? Is it practical to base such a
solution on a few (8) large volumes, each with single large filesystem
in it?

Many thanks in advance for any advice,
- Yaniv

The best bet, would e to go to a NAS appliance, IE EMC or NetApp.
There NAS devices can handle this load better than any veritas
solution. The NSX model from EMC will let you go to 32 TB per file
system per data mover. It also allows for backups via snap shots. Do
not use ATA storage, try to use low cost fiber channel drives. They
have a higher run rate then standard ATA. If you used a DMX3 and a
NSX you would be able to handle 2 to 3 years worth of growth within a
two unit environment.

.

Relevant Pages

  • Advanced storage management ( suggestion )
    ... I see many people interested in designing of new filesystems for ... This layer should provide low level storage services to simplify the ... development of filesystems, DBMS, LDAP servers, or any other ... applications that require high performance storage. ...
    (Linux-Kernel)
  • Re: Big volumes, small files
    ... The system's storage is based on code which writes many files to the ... The system's current storage is based on 4 VxFS filesystems, ... may be a workload which ZFS could handle well (I don't know what level ... still has to map blocks to files, which is the long part of a backup. ...
    (comp.arch.storage)
  • Re: Big volumes, small files
    ... I'd like to plan a new storage solution for a system currently in ... The system's storage is based on code which writes many files to the ... The system's current storage is based on 4 VxFS filesystems, ... Though I have no direct experience with it, my impression is that this may be a workload which ZFS could handle well. ...
    (comp.arch.storage)
  • Re: Big volumes, small files
    ... The system's storage is based on code which writes many files to the ... The system's current storage is based on 4 VxFS filesystems, ... Backups are going to be painful, ... FlashBackup will take the entire image as a backup and not spend eons ...
    (comp.arch.storage)