Re: The 9997th file specific DOSFS problem

Thanks a lot Peper! That makes a lot of sense. Now, I am thinking how
to hack the code to verify your theory and eventually find a solution.
As I don't
have the complete source code and not sure if the source code is the
same version
as the binary we are using, one easy way seems to just overwrite the
value of

pFatDesc->fatGroupSize

i.e., use dosFsVolDescGet(volumeName) to get the pointer to the volumn
descriptor
and overwrite the value of fatGroupSize.

We do have small files as well as large ones stored. Assuming the
search time
penalty is not very significant, we can probably increase the
fatGroupSize
to a big value. Alternatively, can we also set it to a very small
value, such that
it it will take small hit performance periodically but not very bit
hit?

Here is the dump of the dosFsShow:

volume descriptor ptr (pVolDesc): 0x42816c0
cache block I/O descriptor ptr (cbio): 0x42817c0
auto disk check on mount: NOT ENABLED
max # of simultaneously open files: 42
file descriptors in use: 2
# of different files in use: 2
# of descriptors for deleted files: 0
# of obsolete descriptors: 0

current volume configuration:
- volume label: test ; (in boot sector: test)
- volume Id: 0x69747261
- total number of sectors: 286,749,487
- bytes per sector: 512
- # of sectors per cluster: 64
- # of reserved sectors: 32
- FAT entry size: FAT32
- # of sectors per FAT copy: 35,004
- # of FAT table copies: 2
- # of hidden sectors: 0
- first cluster is in sector # 70,040
- Update last access date for open-read-close = FALSE
- directory structure: VFAT
- root dir start cluster: 2

FAT handler information:
------------------------
- allocation group size: 448 clusters
- free space on volume: 115,074,957,396 bytes

Other than increasing the value of pFatDesc->fatGroupSize, is there a
not
very complicated way of improving the searching algorithm and avoid
the
performance hit all together? Currently we do not delete any files
from the
device to avoid fragmentation and it's not of a concern if we leave
some
empty sectors here and there.

Yuejin
On Jul 12, 5:06 pm, peter.mit...@xxxxxxxxx wrote:
y...@xxxxxx wrote:
It can handle the size without problem. We are able to connect to
devices up to 1TB in size.

A bit more info: if the disk is unmounted and then remounted and there
are more than ~10000
files on there already, this problem will happen right away. It
happens when writing the very first
file, and sometime on the second file as well.

Also, if a file is deleted or rewriten, the problem will happen after
that,

because 30 seconds is a long time on a 1GHz PPC machine, it mush be
doing some very
CPU intensive work (searching?) in addition to disk I/O. This is shown
by the very busy CPU
and I/O during the 30 seconds of time.

 You mentioned that the last allocated/searched/accessed sector is
cached. I wonder if this
variable is reset under the above scenarios and it is then trying to
search the whole FAT?

Some kind sole sent me pieces of the dosFs source code. In dosFsLib.c,
I see the definition

int        fatClugFac = 10000;     /* cluster allocation group size factor */

I wonder is this ha anything to do with the problem.

On Jul 11, 7:49�am, peter.mit...@xxxxxxxxx wrote:

AHA! <fatClugFac>.  Thank you kind soul.

I think I have a potential solution.

Short answer :  Change <fatClugFac> to a larger value.  If you change
it to X, then the performance penalty will occur about every X files.
See the long answer for more details.

Long answer.
Whoever that kind soul was, has delivered the final piece to the
puzzle.  Here is what I believe is happening.  To reduce both
fragmentation and head seek penalties, DosFS tries allocate more
clusters than are immediately needed.  Unused clusters are freed when
the file is closed.  You can find out what this value is by calling
dosFsShow() and checking the displayed item called something
"allocation group size".  I'm pretty sure that command exists on 5.5..
This value is essentially the number of FAT entries divided by
<fatClugFac>.  And since VxWorks tracks the last allocated cluster, I
think this will give us periodic behaviour with slow-downs every
<fatClugFac> files (for the most part) as the search for the next free
cluster will require the search to "wrap around".

Playing with some numbers ...
512 GB disk --> ~2^24 million FAT entries (~16 million)
fat allocation group size = 2^24 / <fatClugFac> = ~1600 FAT entries.
1600 FAT entries @ 32 kB cluster size = ~52 MB

If your typical file size is 4 MB, if you set <fatClugFac> to about
(130000) on a 512 GB disk, then you should see a significant slow down
around the 130000th file.  If you did not delete any files, this
should mean that your disk will be almost completely full.

Peter- Hide quoted text -

- Show quoted text -

.

Relevant Pages

  • Error (sysFixBootSector:359) lseek failed
    ... Volume Parameters: FAT type: FAT16, sectors per cluster 4 ... FAT copies, 0 clusters, 200 sectors per FAT ...
    (comp.os.vxworks)
  • Re: How many sectors does a FAT1 and FAT2 copies contain
    ... > it to work as windows format utility works.I am trying to format ... > happens if I allocate 6 sectors to each FAT copy.Please let me know. ... enough FAT entries to reference every cluster on the media, ...
    (comp.programming)
  • Re: CoolPix 950, FAT16 problem with large card
    ... Each cluster is 64 sectors. ... I reformatted the CF card in the camera, powered down, moved the card ... I then used Win98 to reformat the filesystem. ...
    (rec.photo.digital)
  • Re: FAT32 or NTFS?
    ... >>of a hugely bloated FAT and large cluster sizes, ... resilience --- As far as FAT partitions go, don't forget to wave goodbye to ... technologies imbedded in the file system ensure that data is only written on ... Which brings me to this - In an NTFS file ...
    (microsoft.public.windowsxp.general)
  • Re: Resizing FAT partitions
    ... full cluster, and are thus part of the volume, but unused. ... determining the max amount of space to partition on a given drive. ... each type of FAT with a fixed table used for floppies. ... not be able to copy to such small drives. ...
    (microsoft.public.win32.programmer.kernel)