Re: Multiple IMAP connections to same folder

Mark Crispin wrote:
I don't think that you fully understand the problem. Suppose the file opening, operations, and closing are done by a subroutine library that your application calls. From the point of view of the application, the subroutine library is a black box; it doesn't necessarily know what the subroutine library does. This characteristic of fcntl() requires that the application know what the subroutine library does. 
Ok, I see the implications here much better now.

It gets worse. Thanks to links, files can be accessed through multiple names. So, an application may not know that fileA and fileB are really the same file. So the application may have to keep a table of all open files with their device and inode numbers, and prior to opening any file, check to see if it's open by that name.

Here's what typically happens in an IMAP server. The client has a mailbox open. Since it's open, it has a shared lock open on the mailbox on the server. Now, the client uploads a message; the upload operation supplies the name of the destination mailbox and there is no requirement that it be the same mailbox that is open. So it opens another descriptor append the message to the destination, keeping the first descriptor for the open mailbox.

But suppose it is the same mailbox. It now has two descriptors open on the same file, and then when the upload finishes the second descriptor (which never locked, because it was just appending) is closed. That kills the lock on the first descriptor. That in turn allows some other session to open and lock the mailbox, and maybe do things like expunge it even though the first session still has it open...oh dear, oh my. 
Now it's all making sense.

The way I put it, having combination locks and electronic safeties on guns sounds like a real good idea...until you're in a gunfight and your gun won't go bang when you want it to... 
An example which I both clearly understand and agree with.

Any sort of database would use locks.

Most of the traditional mail applications were very primitive and never attempted to provide multiple simultaneous access, much less multiple simultaneous read/write access. Instead, they just prevented mail delivery (e.g., from sendmail) and a single mail reader from stepping on each other; and if a user ran two mail readers at a time, he "deserved" having his mail corrupted.

[In fact, I very much remember being told in the late 1980s that multiple simultaneous access to mail was "impossible" on UNIX and that I shouldn't even think of such a bizarre concept.]
That at least explains to some extent why we find ourselves in this predicament.

And if this is as serious as you seem to indicate, I wonder why there isn't more of an uproar from the Linux application developers? 

Some programmers have squawked about it!

Other programmers have given up on having good lock support in UNIX and UNIX like systems, and consequently resort to other means to accomplish the synchronization that they need.

For example, the oft-touted Maildir format is often claimed not to use locks at all; and its proponents claim this to be a benefit. Actually it does; but these locks are kernel-internal locks inside the filesystem. Maildir application code doesn't do any locking itself.
I now understand why so many people have talked about Maildir as being "safer" -- because locking support is woefully inadequate on certain platforms and Maildir to some extent sidesteps the problem. In this context, solutions such as Maildir seem to address the symptoms of the disease rather than to treat it. That's not meant as a criticism of Maildir as much as an observation of the current state of affairs.

Windows does have good lock support. I wouldn't be surprised if the evil empire uses this as a selling point for Windows. Traditional operating systems, such as OS/360, TOPS-10/20, VAX/VMS, Unisys, etc. (pretty much anything that ever had COBOL applications!) also had good lock support. It's pretty likely that your paycheck is still printed by some ancient operating system.
Wow I find this all rather fascinating if disappointing. Fortunately, I use FreeBSD for the most part so this doesn't affect me directly. However, since a programmer would likely have portability in mind, it indirectly affects any UNIX/UNIX-like OS as you have already stated. Sadly, if Linux is missing the boat on this then we all will most likely be dealing with the consequences one way or another as programmers are forced to come up with their own solutions to the problem.

Thanks so much for taking the time to explain all this. I can't be the only layman to have wondered and suffered his/her curiosity in silence.

Aaron Carlow
.

Relevant Pages

  • Re: Multiple IMAP connections to same folder
    ... it has a shared lock open on the mailbox on the server. ... So it opens another descriptor append the message to the destination, keeping the first descriptor for the open mailbox. ...
    (comp.mail.imap)
  • Union mounts and file locks (was Re: [PATCH 14/39] union-mount: Union mounts documentation)
    ... opens a file read-only and gets a lock on it, ... only happens before a file descriptor is open. ... fcntlon a read-only file descriptor as the first lock ...
    (Linux-Kernel)
  • Re: Tech says the problem is XP, is it?
    ... It is possible for software to "lock" the drive, disabling the button, ... there's no way for it to become locked again until windows is up. ... >>tray will not open. ... it opens some times others not unless I do the fix. ...
    (microsoft.public.windowsxp.general)
  • Re: Administrative access to users sub folders?
    ... trucking outfit and I set up their network, locked things down, but both you ... I created a new mailbox for him was that he, of course, had to be a member ... created the ICUM_EXH account or what have you and he knows that he has to ... >> result is that we he opens another users inbox, or caldendar, for ...
    (microsoft.public.exchange2000.information.store)
  • Re: Rename file in VB 6 if not in use
    ... If by 'in use by others' you mean it's out to an application that opens ... files without maintaining a lock on them (e.g. notepad) then you have no way ... but not write to it or rename or move it. ...
    (microsoft.public.vb.general.discussion)