Re: Help wanted - problems with heap

On 2007-12-02 07:56:11, tyrecius13@xxxxxxxxx wrote:

On Dec 2, 12:54 am, andrewdoull wrote:

I suspect that I'm either running into one of three things:

1. Memory pressure, because of limited heap size. But step 2 should have
identified that by failures to allocate memory. I'm not sure if any major OS
still has significant heap limits and what those are.
2. Heap fragmentation. Unangband allocates 'a lot of memory' in one go (up to
130K per init file) during the initialisation phase of the program where it
parses external data files and creates binary structures from them, then
determines what the correct size for the initialised data structure should be
once it has read in the data, then frees the larger chunks. This undoubtedly
causes problems with heap fragmentation. Again, I'd expect memory allocation
failures, which should results in quitting, if this is the case.
3. Heap corruption. This is the most likely.

Any pointers as to work out what is happening would be welcome.

Ah. Pointers are the problem. :-)

Here are a few things. First, make sure that you were had all the
options that valgrind allows for memory checking turned on. You might
also try similar tools that do the same thing. Some are more clever/
full-proof than others. Electric-fence tends to be popular. Also, make
sure that you don't fall outside of the tolerances of the tool. For
instance, valgrind assumes that it can replace the default malloc/free
with its own through the dynamic linker. If you are statically
linking, it doesn't help.

Firstly, thanks for all the advice. I've got some more questions for you...

Would it pick up any errors if I didn't have the above set corrrectly for
Valgrind? e.g. I was able to pick up some out by one errors, as I previously
mentioned.

Second, tools like valgrind etc. do run-time checks which means that
if the bug does not get tickled on a particular run, then they don't
tell you about the problem. If you can't see the bug happening while
running this kind of tool, then you can't say anything about the
nature of the bug.

I experienced the crash while running Valgrind - however, I'm not sure if the
resulting output was able to tell me anything. e.g. I not sure if everything
cleaned itself up properly including the stdout file handle. I'm actually
running Valgrind telnetting into someone else's box and compiling there (just to
complicate things) as my primary development platform is OS/X at the moment.
Angband is cross-platform luckily. I'll try to reproduce the fault again to see
what state I end up in.

Third, it may be stack corruption. This might cause a later crash in
free() if the pointer that you are deleting was overwritten with a
weird value.

If I had stack corruption, would I be able to backtrace in gdb? I'm assuming
not, or is it possible to get a corrupt stack with all the frame pointers still
fine.

Fourth, unless you have a callstack from the crash, it may not be
happening where you think it is happening. Buffers are not flushed on
a crash so you can have situations where the last thing actually sent
to the screen or a log is far away from the actual problem.

I've always had a callstack with this, which pretty much isolated the problem.

I find that the best way to deal with these problems as a systemic
issue is to assign ownership to pointers. Pointer x has ownership and
pointer y is just using it, so you need to make sure that y is gone
before freeing x. That kind of reasoning. Make sure that you are
especially documenting interfaces that use pointers. When a pointer is
passed through an interface, it should always be clear whether the
interface expects to own that object from then on and delete it at its
own pace, or whether it is just borrowing it. If an interface is just
borrowing it, make sure that the interface is cleaned up before
destroying the object.

Angband doesn't do much creation and destruction of objects in game (e.g. no
dynamic linked lists etc.). The heap is primarily used to hold large dynamic
structures such as maps, and info files (information about the game parsed at
start-up). Most of these are just large 1d or 2d arrays, that are allocated
once, near startup, and destroyed once, usually at shut-down.

This cuts down on the complexity of memory management, which obviously you want
to avoid in a program written in C.

Andrew
--
The Roflwtfzomgbbq Quylthulg summons L33t Paladins -more-
"Apple: Celebrating the poisoning of Alan Turing since 1977."
ASCII Dreams: http://roguelikedeveloper.blogspot.com
Unangband: http://unangband.blogspot.com
.

Relevant Pages

  • Re: Struct inside class
    ... The compiler figures computes the ... including anything derived from "Object": Heap pointer ... memory leaks and why the .NET good garbage collector is required. ... If GC_ALLOCATE can't allocate the requested ...
    (microsoft.public.dotnet.framework)
  • Re: run-time vs compile-time
    ... > offset related to some location (like stack base) somewhere. ... > offset from heap to pi. ... When you allocate an int on the heap, it is allocated at address 1. ... application has a given amount of memory it can use as it wishes. ...
    (alt.comp.lang.learn.c-cpp)
  • Re: run-time vs compile-time
    ... > offset related to some location (like stack base) somewhere. ... > offset from heap to pi. ... When you allocate an int on the heap, it is allocated at address 1. ... application has a given amount of memory it can use as it wishes. ...
    (comp.lang.cpp)
  • Re: Huge pages and small pages. . .
    ... >> what looks like contiguous memory and away you go. ... you need to have them cached in the TLB; if the TLB runs out of ... > low end of the heap, until someone figures out a way to tell the system ... When you allocate memory, the kernel just marks a promised ...
    (Linux-Kernel)
  • Re: Help wanted - problems with heap
    ... Memory pressure, because of limited heap size. ... Any pointers as to work out what is happening would be welcome. ... interface expects to own that object from then on and delete it at its ...
    (rec.games.roguelike.development)