Re: Good Dungeon Mapping e-Tool?

On Wed, 11 Jun 2008 23:18:20 +0200, Mart van de Wege
<mvdwege.usenet@xxxxxxxxxx> wrote:

Kyle Wilson <kyle@xxxxxxxxxxxxx> writes:

On Wed, 11 Jun 2008 21:25:51 +0200, Mart van de Wege
<mvdwege.usenet@xxxxxxxxxx> wrote:

Kyle Wilson <kyle@xxxxxxxxxxxxx> writes:

Just adding a few bits to Keith's posts further down:

Ok, I've been pretty much entirely on Win32 for the last ten years or
so. I know where most of the bodies are buried and where there be
dragons in NT lineage systems. My memories of the Unix side of things
are blurrier, but I do recall real issues with modularity in
particular (signal handling being a big one, if you had multiple
modules loaded into a single process and more than one needed SIGIO,
you were pretty much screwed unless you build a dispatcher yourself).

Depending on your kernel/C library combination, this *may* still be an
issue. Relatively recent Linux kernels and GNU libc have gotten most
of the issues out, and signals are now threadsafe.

Dunno about other combinations these days though.

Not so much a thread safety issue that I was thinking of as a
modularity issue. One example is async I/O.

They go together on *nix, ass the standard model is still heavily
biased towards multiprocessing, with each process more or less
independently handling its chores without much inter-process
communication.

No wonder *nix shines in situations where the workload is close to
this model, like e.g. webserving.

Ah, much of the code in the system I work on is very tightly coupled,
multithreaded code. Pretty much heavy mathematical data processing
pipelined across multiple thread stages. It sounds like doing
concurrent programming in the Unix world would require a very
different approach from what I'm used to.

On old Solaris (last time I mucked with it) you'd set a signal handler
for SIGIO and the runtime library would return the address of the last
installed handler for SIGIO. Given two libraries that both want to
handle that signal (linked dynamically or statically to a program),
the second one can either chain to the first one's handler (and the
first one can never unregister for this SIGIO without some special
arrangement with the other library or it can discard the first one's
handler and thus library #1 stops getting signals with no notice that
its handler has been removed). If you are writing the main program
and the libraries then you can enforce behavior that makes this work
by mandating that library code not call signal(...) and instead call
MySignal(...) and write you own code to manage and dispatch multiple
signals to multiple callers.

AFAIK, signals are per-process. As long as you split off processes
instead of multithreading, you can have multiple handlers per signal.

Ok, so it sounds like I was basically right. To 'load' a complex
plug-in, you implement it as a separate executable and fork/exec to
get concurrency and a separate process context. Windows processes are
heavy weight enough that the environment is designed to make spinning
a new process relatively uncommon.

Under Linux, with its lightweight process model, this is virtually the
same as using multiple threads to handle a signal. But see below.

clone() on Linux actually creates a shared execution context that *is*
able to independently receive signals. But again, see below.

That makes sense. I can see how it could all work quite well with the
shm*() functions for shared context and multiple processes to isolate
the various resources that would otherwise cause conflicts.

In Win32 (and I think this is mostly a 'later implementations get to
learn from the mistakes of earlier ones' issue) if you want to do
asynchronous I/O you use an OVERLAPPED struct that contains a kernel
mode provided by your code that is signaled only when the I/O object
associated with that OVERLAPPED object has activity. Two libraries
can perform such operations with no knowledge of each other.

The Posix APIs were remarkably well designed for their era. The
versions that were still standard in Solaris six or seven years ago (I
know this is a long time) hadn't really changed to accommodate ld.so
dynamic libraries and threading very well. Is there a new version of
the Posix API standard out within the last few years? I haven't heard
of anything, but I don't follow Unix/Linux programming all that
closely these days.

Posix hasn't changed much. If you're not afraid to write less portable
code, you can use various enhancements in the Linux kernel to go
beyond this model. Async I/O is one thing that is implemented in the
current 2.6 kernel series.

Dunno about how good the C library support is. I have not delved into
the glibc doc in ages. And as for other *nixes, I haven't seen a
Solaris box up close in two years, so I don't know the state of the
technology there.

I haven't written straight C code in years. My impression is that g++
is a pretty capable C++ compiler though.

Yeah, I realize that SunOS and Solaris are old and crufty. I expect
that most of the interesting new work has been going on in the Linux
community (between the open source advantage and the fact that the
other Unixes out there are aimed at *very* high reliability
environments, Linux makes sense as the focus of innovation).

I think that this was mostly a result of the Posix APIs being defined
when pretty much everything (including the kernel) was statically
linked into a single image and generally written by one team. I
assume that much of this has changed in recent Linux given that ld.so
images are supported and I believe that there are lots of programs
that run over there that dynamically load and bind to modules written
by third parties. I think the old-style answer to this was
'fork-exec' the new code and talk to it over a pipe or Unix domain
socket.

I think you're mixing up dynamic loading and threading here.

Nope. Before ld.so support was added to Unix (and I know that this
was in the dark ages) if you wanted to add a 'plug-in' to a program,
you'd start the plug in (which had to be an executable) with its stdin
and stdout chained to a pair of file descriptors in your program (I
seem to remember that there was even an API call that set this up) and
then communicate with the sub-process through the two pipes. This
provided both concurrency and modularity and I believe that a number
of early compilers ran their various passes this way. Once you've got
true dynamic libraries, you can bypass this sort of mechanism and
simply load your libraries in-process and query for function entry
points as convenient.

Or, you can fork just before you start executing a function. Yes,
that's the same old model, except that you don't need to go down to
actually starting a new executable. You can fork() everything, and the
kernel will build a new process.

Yup, I'm familiar with fork without exec and copy-on-write memory
semantics for these sorts of things. I just haven't done much more
than read about 'em.

On Linux, this may actually spawn a subthread in your main executable
image, but that's due to the way Linux has merged threading and
multiprocessing.

With COW for the shared read/write stuff and a new process descriptor,
I'm not sure there is a significant difference between the two. You
clone some kernel mode structure for the new process, tweak the
scheduler and mark the writ able pages as COW.

By default though, the only thing shared is file descriptors (and some
other administrative stuf), which you can of course chain using
pipes. If you want to do more inter-task communication, you're still
stuck using an IPC mechanism like pipes, SHM, or domain sockets. And
of course, if you use dlopen() to open a library, you can pass that
info to a forked child process.

Ah, ok so dlopen() handles attaching to an ld.so shared library at
runtime.

Or, once again, you use the lower level APIs in the kernel to set up
multiprocessing with shared data, basically going the multithread way
a la Windows. The end result is more what you're used to, but the way
to get there is still (partly) through the old-fashioned model.

And the above is not portable of course, with many Unix greybeards
pouring scorn and derision on anyone using it as being a 'Linux
weenie'.

I expect that there is much more Linux out there than the next three
Unixes combined.

You are correct that most forms of concurrent programming in Posix are
done by ye olde fork-and-exec, aka splitting off a whole process and
communicating through some form of IPC (pipes, sockets and shared
memory being most popular).

Threading has become more popular, as it is more lightweight, and the
threading APIs have matured and are now supported by most vendors (as
I mentioned above, Linux and GNU libc do).

Sure, I would suspect that everyone is still running pthread...

If that is PosixThreads, then yes.

I suspect that it is. Last time I looked into threading in a Unix
environment, the API seems to be called PThreads (which I suspect does
indeed stand for PosixThreads). I read about it, but since the code
was being ported from embedded Unix to Win32 I didn't need to write
any code using PThreads.

Even the Linux-native threading has moved that way, and then added
some of its own flavour via clone() and co.

However, both programming models have absolutely no problems
dynamically loading submodules.

Sure, I haven't done any significant Unix coding in over five years. I
expect that there is a Posix equivalent to the Win32 LoadLibrary(...)
and GetProcAddress(...) API calls that load a file as a dynamic
library at runtime and locate and return the entry point for a named
function in that module.

Of course. Ye olde dlopen() will do just fine.

I figured that there had to be something like that. The only time
I've even used dynamic libraries on Unix was when load-time linking to
the shareable C and C++ runtimes.
--

Kyle Wilson
email: kylewilson@xxxxxxxxxxxxx
.

Relevant Pages

  • Re: Good Dungeon Mapping e-Tool?
    ... and signals are now threadsafe. ... Given two libraries that both want to ... Under Linux, with its lightweight process model, this is virtually the ... I think you're mixing up dynamic loading and threading here. ...
    (rec.games.frp.dnd)
  • Re: Significant Pure Assembler Application In MASM ?
    ... When I was thinking of a literal "port" of RosAsm for LuxAsm, I was quite daunted by this!!! ... I was seeing a lot of APIs that, I'm not sure what they do, but I don't think there's a Linux equivalent. ... I was contemplating *avoiding* libraries for LuxAsm, ... If GTK is a "replacement od the Win Api", ...
    (alt.lang.asm)
  • Re: Multiprocessing, shared memory vs. pickled copies
    ... arrays to multiple processors. ... MKL, ACML), and fast vector math libraries. ... most Python programmers' instinct seems to be to use multiple ... not a traditional three-layer network. ...
    (comp.lang.python)
  • [opensuse] Re: Manual dist upgrade procedure(s) & tools/progs/aids?
    ... and the older software needs the old libraries. ... 88 SuSE Linux 9.3 ... information on disk is stored in groupings. ... name to an inode, you can delete the first name and the information ...
    (SuSE)
  • Re: The Lib debate
    ... > I guess you only know Linux from hearing it in the newspapers commercials paid ... It is not a problem for source-level libraries at all. ... I am talking about assembler ... if you use precompiled libraries written on HLL? ...
    (alt.lang.asm)