dovecot/original-hg/dovecot-1.2: src/lib-storage/index/maildir/maildir-sync.c annotate

annotate src/lib-storage/index/maildir/maildir-sync.c @ 3085:73944f902165 HEAD

don't allocate memory from data stack uselessly.

author	Timo Sirainen <tss@iki.fi>
date	Mon, 10 Jan 2005 18:14:41 +0200
parents	61c8d205d887
children	391a1064872e

rev	line source
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	1 /* Copyright (C) 2004 Timo Sirainen */
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	2
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	3 /*
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	4 Here's a description of how we handle Maildir synchronization and
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	5 it's problems:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	6
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	7 We want to be as efficient as we can. The most efficient way to
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	8 check if changes have occured is to stat() the new/ and cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	9 directories and uidlist file - if their mtimes haven't changed,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	10 there's no changes and we don't need to do anything.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	11
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	12 Problem 1: Multiple changes can happen within a single second -
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	13 nothing guarantees that once we synced it, someone else didn't just
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	14 then make a modification. Such modifications wouldn't get noticed
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	15 until a new modification occured later.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	16
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	17 Problem 2: Syncing cur/ directory is much more costly than syncing
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	18 new/. Moving mails from new/ to cur/ will always change mtime of
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	19 cur/ causing us to sync it as well.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	20
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	21 Problem 3: We may not be able to move mail from new/ to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	22 because we're out of quota, or simply because we're accessing a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	23 read-only mailbox.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	24
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	25
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	26 MAILDIR_SYNC_SECS
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	27 -----------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	28
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	29 Several checks below use MAILDIR_SYNC_SECS, which should be maximum
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	30 clock drift between all computers accessing the maildir (eg. via
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	31 NFS), rounded up to next second. Our default is 1 second, since
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	32 everyone should be using NTP.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	33
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	34 Note that setting it to 0 works only if there's only one computer
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	35 accessing the maildir. It's practically impossible to make two
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	36 clocks _exactly_ synchronized.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	37
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	38 It might be possible to only use file server's clock by looking at
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	39 the atime field, but I don't know how well that would actually work.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	40
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	41 cur directory
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	42 -------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	43
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	44 We have dirty_cur_time variable which is set to cur/ directory's
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	45 mtime when it's >= time() - MAILDIR_SYNC_SECS and we _think_ we have
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	46 synchronized the directory.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	47
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	48 When dirty_cur_time is non-zero, we don't synchronize the cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	49 directory until
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	50
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	51 a) cur/'s mtime changes
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	52 b) opening a mail fails with ENOENT
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	53 c) time() > dirty_cur_time + MAILDIR_SYNC_SECS
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	54
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	55 This allows us to modify the maildir multiple times without having
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	56 to sync it at every change. The sync will eventually be done to
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	57 make sure we didn't miss any external changes.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	58
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	59 The dirty_cur_time is set when:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	60
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	61 - we change message flags
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	62 - we expunge messages
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	63 - we move mail from new/ to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	64 - we sync cur/ directory and it's mtime is >= time() - MAILDIR_SYNC_SECS
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	65
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	66 It's unset when we do the final syncing, ie. when mtime is
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	67 older than time() - MAILDIR_SYNC_SECS.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	68
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	69 new directory
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	70 -------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	71
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	72 If new/'s mtime is >= time() - MAILDIR_SYNC_SECS, always synchronize
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	73 it. dirty_cur_time-like feature might save us a few syncs, but
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	74 that might break a client which saves a mail in one connection and
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	75 tries to fetch it in another one. new/ directory is almost always
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	76 empty, so syncing it should be very fast anyway. Actually this can
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	77 still happen if we sync only new/ dir while another client is also
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	78 moving mails from it to cur/ - it takes us a while to see them.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	79 That's pretty unlikely to happen however, and only way to fix it
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	80 would be to always synchronize cur/ after new/.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	81
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	82 Normally we move all mails from new/ to cur/ whenever we sync it. If
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	83 it's not possible for some reason, we mark the mail with "probably
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	84 exists in new/ directory" flag.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	85
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	86 If rename() still fails because of ENOSPC or EDQUOT, we still save
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	87 the flag changes in index with dirty-flag on. When moving the mail
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	88 to cur/ directory, or when we notice it's already moved there, we
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	89 apply the flag changes to the filename, rename it and remove the
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	90 dirty flag. If there's dirty flags, this should be tried every time
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	91 after expunge or when closing the mailbox.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	92
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	93 uidlist
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	94 -------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	95
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	96 This file contains UID <-> filename mappings. It's updated only when
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	97 new mail arrives, so it may contain filenames that have already been
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	98 deleted. Updating is done by getting uidlist.lock file, writing the
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	99 whole uidlist into it and rename()ing it over the old uidlist. This
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	100 means there's no need to lock the file for reading.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	101
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	102 Whenever uidlist is rewritten, it's mtime must be larger than the old
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	103 one's. Use utime() before rename() if needed. Note that inode checking
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	104 wouldn't have been sufficient as inode numbers can be reused.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	105
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	106 This file is usually read the first time you need to know filename for
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	107 given UID. After that it's not re-read unless new mails come that we
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	108 don't know about.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	109
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	110 broken clients
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	111 --------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	112
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	113 Originally the middle identifier in Maildir filename was specified
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	114 only as <process id>_<delivery counter>. That however created a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	115 problem with randomized PIDs which made it possible that the same
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	116 PID was reused within one second.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	117
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	118 So if within one second a mail was delivered, MUA moved it to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	119 and another mail was delivered by a new process using same PID as
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	120 the first one, we likely ended up overwriting the first mail when
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	121 the second mail was moved over it.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	122
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	123 Nowadays everyone should be giving a bit more specific identifier,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	124 for example include microseconds in it which Dovecot does.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	125
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	126 There's a simple way to prevent this from happening in some cases:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	127 Don't move the mail from new/ to cur/ if it's mtime is >= time() -
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	128 MAILDIR_SYNC_SECS. The second delivery's link() call then fails
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	129 because the file is already in new/, and it will then use a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	130 different filename. There's a few problems with this however:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	131
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	132 - it requires extra stat() call which is unneeded extra I/O
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	133 - another MUA might still move the mail to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	134 - if first file's flags are modified by either Dovecot or another
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	135 MUA, it's moved to cur/ (you _could_ just do the dirty-flagging
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	136 but that'd be ugly)
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	137
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	138 Because this is useful only for very few people and it requires
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	139 extra I/O, I decided not to implement this. It should be however
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	140 quite easy to do since we need to be able to deal with files in new/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	141 in any case.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	142
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	143 It's also possible to never accidentally overwrite a mail by using
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	144 link() + unlink() rather than rename(). This however isn't very
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	145 good idea as it introduces potential race conditions when multiple
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	146 clients are accessing the mailbox:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	147
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	148 Trying to move the same mail from new/ to cur/ at the same time:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	149
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	150 a) Client 1 uses slightly different filename than client 2,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	151 for example one sets read-flag on but the other doesn't.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	152 You have the same mail duplicated now.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	153
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	154 b) Client 3 sees the mail between Client 1's and 2's link() calls
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	155 and changes it's flag. You have the same mail duplicated now.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	156
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	157 And it gets worse when they're unlink()ing in cur/ directory:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	158
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	159 c) Client 1 changes mails's flag and client 2 changes it back
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	160 between 1's link() and unlink(). The mail is now expunged.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	161
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	162 d) If you try to deal with the duplicates by unlink()ing another
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	163 one of them, you might end up unlinking both of them.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	164
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	165 So, what should we do then if we notice a duplicate? First of all,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	166 it might not be a duplicate at all, readdir() might have just
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	167 returned it twice because it was just renamed. What we should do is
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	168 create a completely new base name for it and rename() it to that.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	169 If the call fails with ENOENT, it only means that it wasn't a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	170 duplicate after all.
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	171 */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	172
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	173 #include "lib.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	174 #include "ioloop.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	175 #include "buffer.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	176 #include "hash.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	177 #include "str.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	178 #include "maildir-storage.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	179 #include "maildir-uidlist.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	180
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	181 #include <stdio.h>
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	182 #include <stddef.h>
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	183 #include <unistd.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	184 #include <dirent.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	185 #include <sys/stat.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	186
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	187 #define MAILDIR_SYNC_SECS 1
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	188
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	189 #define MAILDIR_FILENAME_FLAG_FOUND 128
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	190
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	191 struct maildir_sync_context {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	192 struct index_mailbox *ibox;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	193 const char new_dir, cur_dir;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	194 int partial;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	195
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	196 struct maildir_uidlist_sync_ctx *uidlist_sync_ctx;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	197 struct maildir_index_sync_context *index_sync_ctx;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	198 };
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	199
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	200 struct maildir_index_sync_context {
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	201 struct index_mailbox *ibox;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	202 struct mail_index_view *view;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	203 struct mail_index_sync_ctx *sync_ctx;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	204 struct mail_index_transaction *trans;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	205
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	206 struct mail_index_sync_rec sync_rec;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	207 uint32_t seq;
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	208 int dirty_state;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	209 };
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	210
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	211 static int maildir_expunge(struct index_mailbox ibox, const char path,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	212 void *context __attr_unused__)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	213 {
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	214 if (unlink(path) == 0) {
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	215 ibox->dirty_cur_time = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	216 return 1;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	217 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	218 if (errno == ENOENT)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	219 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	220
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	221 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	222 "unlink(%s) failed: %m", path);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	223 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	224 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	225
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	226 static int maildir_sync_flags(struct index_mailbox ibox, const char path,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	227 void *context)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	228 {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	229 struct maildir_index_sync_context *ctx = context;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	230 const char *newpath;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	231 enum mail_flags flags;
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	232 const char const keywords;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	233 uint8_t flags8;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	234
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	235 ctx->dirty_state = 0;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	236
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	237 (void)maildir_filename_get_flags(path, pool_datastack_create(),
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	238 &flags, &keywords);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	239
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	240 flags8 = flags;
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	241 mail_index_sync_flags_apply(&ctx->sync_rec, &flags8);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	242
1958 4dec6a3d79fd s/custom flags/keywords/ Timo Sirainen <tss@iki.fi> parents: 1956 diff changeset	243 newpath = maildir_filename_set_flags(path, flags8, keywords);
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	244 if (rename(path, newpath) == 0) {
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	245 if ((flags8 & MAIL_INDEX_MAIL_FLAG_DIRTY) != 0)
ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	246 ctx->dirty_state = -1;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	247 ibox->dirty_cur_time = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	248 return 1;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	249 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	250 if (errno == ENOENT)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	251 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	252
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	253 if (ENOSPACE(errno) \|\| errno == EACCES) {
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	254 mail_index_update_flags(ctx->trans, ctx->seq, MODIFY_ADD,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	255 MAIL_INDEX_MAIL_FLAG_DIRTY);
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	256 ctx->dirty_state = 1;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	257 return 1;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	258 }
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	259
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	260 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	261 "rename(%s, %s) failed: %m", path, newpath);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	262 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	263 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	264
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	265 static int maildir_sync_record(struct index_mailbox *ibox,
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	266 struct maildir_index_sync_context *ctx)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	267 {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	268 struct mail_index_sync_rec *sync_rec = &ctx->sync_rec;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	269 struct mail_index_view *view = ctx->view;
2033 4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	270 uint32_t seq, seq1, seq2, uid;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	271
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	272 switch (sync_rec->type) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	273 case MAIL_INDEX_SYNC_TYPE_APPEND:
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	274 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	275 case MAIL_INDEX_SYNC_TYPE_EXPUNGE:
2033 4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	276 /* make it go through sequences to avoid looping through huge
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	277 holes in UID range */
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	278 if (mail_index_lookup_uid_range(view, sync_rec->uid1,
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	279 sync_rec->uid2,
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	280 &seq1, &seq2) < 0)
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	281 return -1;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	282
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	283 if (seq1 == 0)
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	284 break;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	285
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	286 for (seq = seq1; seq <= seq2; seq++) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	287 if (mail_index_lookup_uid(view, seq, &uid) < 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	288 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	289 if (maildir_file_do(ibox, uid, maildir_expunge,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	290 NULL) < 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	291 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	292 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	293 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	294 case MAIL_INDEX_SYNC_TYPE_FLAGS:
2033 4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	295 if (mail_index_lookup_uid_range(view, sync_rec->uid1,
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	296 sync_rec->uid2,
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	297 &seq1, &seq2) < 0)
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	298 return -1;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	299
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	300 if (seq1 == 0)
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	301 break;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	302
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	303 for (ctx->seq = seq1; ctx->seq <= seq2; ctx->seq++) {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	304 if (mail_index_lookup_uid(view, ctx->seq, &uid) < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	305 return -1;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	306 if (maildir_file_do(ibox, uid,
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	307 maildir_sync_flags, ctx) < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	308 return -1;
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	309 if (ctx->dirty_state < 0) {
ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	310 /* flag isn't dirty anymore */
ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	311 mail_index_update_flags(ctx->trans, ctx->seq,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	312 MODIFY_REMOVE,
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	313 MAIL_INDEX_MAIL_FLAG_DIRTY);
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	314 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	315 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	316 break;
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	317 case MAIL_INDEX_SYNC_TYPE_KEYWORDS:
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	318 // FIXME
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	319 break;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	320 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	321
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	322 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	323 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	324
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	325 int maildir_sync_last_commit(struct index_mailbox *ibox)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	326 {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	327 struct maildir_index_sync_context ctx;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	328 uint32_t seq;
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	329 uoff_t offset;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	330 int ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	331
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	332 if (ibox->commit_log_file_seq == 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	333 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	334
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	335 memset(&ctx, 0, sizeof(ctx));
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	336 ctx.ibox = ibox;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	337
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	338 ibox->syncing_commit = TRUE;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	339 ret = mail_index_sync_begin(ibox->index, &ctx.sync_ctx, &ctx.view,
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	340 ibox->commit_log_file_seq,
2685 52352a5859e4 When we had dirty messages, we kept resyncing the whole mailbox constantly. Timo Sirainen <tss@iki.fi> parents: 2570 diff changeset	341 ibox->commit_log_file_offset, FALSE, FALSE);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	342 if (ret > 0) {
2877 3cb483d565a6 Allow specifying to transaction if it's external. Make mail saving Timo Sirainen <tss@iki.fi> parents: 2818 diff changeset	343 ctx.trans = mail_index_transaction_begin(ctx.view, FALSE, TRUE);
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	344
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	345 while ((ret = mail_index_sync_next(ctx.sync_ctx,
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	346 &ctx.sync_rec)) > 0) {
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	347 if (maildir_sync_record(ibox, &ctx) < 0) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	348 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	349 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	350 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	351 }
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	352 if (mail_index_transaction_commit(ctx.trans, &seq, &offset) < 0)
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	353 ret = -1;
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	354 if (mail_index_sync_commit(ctx.sync_ctx) < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	355 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	356 }
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	357 ibox->syncing_commit = FALSE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	358
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	359 if (ret == 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	360 ibox->commit_log_file_seq = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	361 ibox->commit_log_file_offset = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	362 } else {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	363 mail_storage_set_index_error(ibox);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	364 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	365 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	366 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	367
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	368 static struct maildir_sync_context *
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	369 maildir_sync_context_new(struct index_mailbox *ibox)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	370 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	371 struct maildir_sync_context *ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	372
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	373 ctx = t_new(struct maildir_sync_context, 1);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	374 ctx->ibox = ibox;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	375 ctx->new_dir = t_strconcat(ibox->path, "/new", NULL);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	376 ctx->cur_dir = t_strconcat(ibox->path, "/cur", NULL);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	377 return ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	378 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	379
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	380 static void maildir_sync_deinit(struct maildir_sync_context *ctx)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	381 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	382 if (ctx->uidlist_sync_ctx != NULL)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	383 (void)maildir_uidlist_sync_deinit(ctx->uidlist_sync_ctx);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	384 if (ctx->index_sync_ctx != NULL)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	385 maildir_sync_index_abort(ctx->index_sync_ctx);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	386 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	387
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	388 static int maildir_fix_duplicate(struct index_mailbox ibox, const char dir,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	389 const char *old_fname)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	390 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	391 const char new_fname, old_path, *new_path;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	392 int ret = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	393
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	394 t_push();
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	395
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	396 old_path = t_strconcat(dir, "/", old_fname, NULL);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	397 new_fname = maildir_generate_tmp_filename(&ioloop_timeval);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	398 new_path = t_strconcat(ibox->path, "/new/", new_fname, NULL);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	399
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	400 if (rename(old_path, new_path) == 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	401 i_warning("Fixed duplicate in %s: %s -> %s",
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	402 ibox->path, old_fname, new_fname);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	403 } else if (errno != ENOENT) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	404 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	405 "rename(%s, %s) failed: %m", old_path, new_path);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	406 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	407 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	408 t_pop();
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	409
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	410 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	411 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	412
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	413 static int maildir_scan_dir(struct maildir_sync_context *ctx, int new_dir)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	414 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	415 struct mail_storage *storage = ctx->ibox->box.storage;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	416 const char *dir;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	417 DIR *dirp;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	418 string_t src, dest;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	419 struct dirent *dp;
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	420 enum maildir_uidlist_rec_flag flags;
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	421 int move_new, ret = 1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	422
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	423 dir = new_dir ? ctx->new_dir : ctx->cur_dir;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	424 dirp = opendir(dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	425 if (dirp == NULL) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	426 mail_storage_set_critical(storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	427 "opendir(%s) failed: %m", dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	428 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	429 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	430
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	431 t_push();
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	432 src = t_str_new(1024);
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	433 dest = t_str_new(1024);
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	434
2039 f0925b2271e1 Added pop3_mails_keep_recent option. Fixed recent assert crash. Timo Sirainen <tss@iki.fi> parents: 2038 diff changeset	435 move_new = new_dir && !mailbox_is_readonly(&ctx->ibox->box) &&
f0925b2271e1 Added pop3_mails_keep_recent option. Fixed recent assert crash. Timo Sirainen <tss@iki.fi> parents: 2038 diff changeset	436 !ctx->ibox->keep_recent;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	437 while ((dp = readdir(dirp)) != NULL) {
2258 087a43e29492 No maildir filename checking after all. Timo Sirainen <tss@iki.fi> parents: 2256 diff changeset	438 if (dp->d_name[0] == '.')
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	439 continue;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	440
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	441 ret = maildir_uidlist_sync_next_pre(ctx->uidlist_sync_ctx,
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	442 dp->d_name);
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	443 if (ret == 0) {
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	444 /* new file and we couldn't lock uidlist, check this
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	445 later in next sync. */
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	446 if (new_dir)
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	447 ctx->ibox->last_new_mtime = 0;
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	448 else
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	449 ctx->ibox->dirty_cur_time = ioloop_time;
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	450 continue;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	451 }
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	452 if (ret < 0)
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	453 break;
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	454
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	455 flags = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	456 if (move_new) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	457 str_truncate(src, 0);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	458 str_truncate(dest, 0);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	459 str_printfa(src, "%s/%s", ctx->new_dir, dp->d_name);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	460 str_printfa(dest, "%s/%s", ctx->cur_dir, dp->d_name);
1970 0f4c4a043be1 When moving mails to cur/, we must add the ":2," part to filename. Timo Sirainen <tss@iki.fi> parents: 1962 diff changeset	461 if (strchr(dp->d_name, ':') == NULL)
0f4c4a043be1 When moving mails to cur/, we must add the ":2," part to filename. Timo Sirainen <tss@iki.fi> parents: 1962 diff changeset	462 str_append(dest, ":2,");
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	463 if (rename(str_c(src), str_c(dest)) == 0) {
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	464 /* we moved it - it's \Recent for us */
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	465 ctx->ibox->dirty_cur_time = ioloop_time;
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	466 flags \|= MAILDIR_UIDLIST_REC_FLAG_MOVED \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	467 MAILDIR_UIDLIST_REC_FLAG_RECENT;
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	468 } else if (ENOTFOUND(errno)) {
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	469 /* someone else moved it already */
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	470 flags \|= MAILDIR_UIDLIST_REC_FLAG_MOVED;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	471 } else if (ENOSPACE(errno)) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	472 /* not enough disk space, leave here */
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	473 flags \|= MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	474 MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	475 move_new = FALSE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	476 } else {
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	477 flags \|= MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	478 MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	479 mail_storage_set_critical(storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	480 "rename(%s, %s) failed: %m",
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	481 str_c(src), str_c(dest));
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	482 }
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	483 } else if (new_dir) {
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	484 flags \|= MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	485 MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	486 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	487
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	488 ret = maildir_uidlist_sync_next(ctx->uidlist_sync_ctx,
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	489 dp->d_name, flags);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	490 if (ret <= 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	491 if (ret < 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	492 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	493
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	494 /* possibly duplicate - try fixing it */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	495 if (maildir_fix_duplicate(ctx->ibox,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	496 dir, dp->d_name) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	497 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	498 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	499 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	500 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	501 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	502
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	503 if (closedir(dirp) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	504 mail_storage_set_critical(storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	505 "closedir(%s) failed: %m", dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	506 }
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	507
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	508 t_pop();
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	509 return ret < 0 ? -1 : 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	510 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	511
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	512 static int maildir_sync_quick_check(struct maildir_sync_context *ctx,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	513 int new_changed_r, int cur_changed_r)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	514 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	515 struct index_mailbox *ibox = ctx->ibox;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	516 struct stat st;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	517 time_t new_mtime, cur_mtime;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	518
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	519 new_changed_r = cur_changed_r = FALSE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	520
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	521 if (stat(ctx->new_dir, &st) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	522 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	523 "stat(%s) failed: %m", ctx->new_dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	524 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	525 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	526 new_mtime = st.st_mtime;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	527
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	528 if (stat(ctx->cur_dir, &st) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	529 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	530 "stat(%s) failed: %m", ctx->cur_dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	531 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	532 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	533 cur_mtime = st.st_mtime;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	534
2893 fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	535 /* cur stamp is kept in index, we don't have to sync if
fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	536 someone else has done it and updated the index. */
fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	537 ibox->last_cur_mtime = mail_index_get_header(ibox->view)->sync_stamp;
fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	538 if (ibox->dirty_cur_time == 0 && cur_mtime != ibox->last_cur_mtime) {
fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	539 /* check if the index has been updated.. */
2936 57f975a9b63b Renamed mail_index_refresh() to mail_index_reopen_if_needed(). Added public Timo Sirainen <tss@iki.fi> parents: 2893 diff changeset	540 if (mail_index_refresh(ibox->index) < 0) {
2064 5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	541 mail_storage_set_index_error(ibox);
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	542 return -1;
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	543 }
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	544
2936 57f975a9b63b Renamed mail_index_refresh() to mail_index_reopen_if_needed(). Added public Timo Sirainen <tss@iki.fi> parents: 2893 diff changeset	545 ibox->last_cur_mtime =
57f975a9b63b Renamed mail_index_refresh() to mail_index_reopen_if_needed(). Added public Timo Sirainen <tss@iki.fi> parents: 2893 diff changeset	546 mail_index_get_header(ibox->view)->sync_stamp;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	547 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	548
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	549 if (new_mtime != ibox->last_new_mtime \|\|
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	550 new_mtime >= ibox->last_new_sync_time - MAILDIR_SYNC_SECS) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	551 *new_changed_r = TRUE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	552 ibox->last_new_mtime = new_mtime;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	553 ibox->last_new_sync_time = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	554 }
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	555
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	556 if (cur_mtime != ibox->last_cur_mtime \|\|
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1958 diff changeset	557 (ibox->dirty_cur_time != 0 &&
4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1958 diff changeset	558 ioloop_time - ibox->dirty_cur_time > MAILDIR_SYNC_SECS)) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	559 /* cur/ changed, or delayed cur/ check */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	560 *cur_changed_r = TRUE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	561 ibox->last_cur_mtime = cur_mtime;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	562
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	563 ibox->dirty_cur_time =
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	564 cur_mtime >= ioloop_time - MAILDIR_SYNC_SECS ?
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	565 cur_mtime : 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	566 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	567
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	568 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	569 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	570
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	571 struct maildir_index_sync_context *
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	572 maildir_sync_index_begin(struct index_mailbox *ibox)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	573 {
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	574 struct maildir_index_sync_context *sync_ctx;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	575
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	576 sync_ctx = i_new(struct maildir_index_sync_context, 1);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	577 sync_ctx->ibox = ibox;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	578
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	579 if (mail_index_sync_begin(ibox->index, &sync_ctx->sync_ctx,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	580 &sync_ctx->view, (uint32_t)-1, (uoff_t)-1,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	581 FALSE, FALSE) <= 0) {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	582 mail_storage_set_index_error(ibox);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	583 return NULL;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	584 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	585 return sync_ctx;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	586 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	587
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	588 void maildir_sync_index_abort(struct maildir_index_sync_context *sync_ctx)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	589 {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	590 mail_index_sync_rollback(sync_ctx->sync_ctx);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	591 i_free(sync_ctx);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	592 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	593
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	594 int maildir_sync_index_finish(struct maildir_index_sync_context *sync_ctx,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	595 int partial)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	596 {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	597 struct index_mailbox *ibox = sync_ctx->ibox;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	598 struct mail_index_view *view = sync_ctx->view;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	599 struct maildir_uidlist_iter_ctx *iter;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	600 struct mail_index_transaction *trans;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	601 const struct mail_index_header *hdr;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	602 const struct mail_index_record *rec;
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	603 pool_t keyword_pool;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	604 uint32_t seq, uid;
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	605 enum maildir_uidlist_rec_flag uflags;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	606 const char *filename;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	607 enum mail_flags flags;
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	608 const char const keywords;
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	609 uint32_t uid_validity, next_uid;
1917 68938dccbc45 Forced locking to be right with mprotect()ing index file. Support for Timo Sirainen <tss@iki.fi> parents: 1915 diff changeset	610 int ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	611
2892 62d53b49110d Changed mail_index_get_header() to return the header as return value because Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	612 hdr = mail_index_get_header(view);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	613 uid_validity = maildir_uidlist_get_uid_validity(ibox->uidlist);
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	614 if (uid_validity != hdr->uid_validity &&
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	615 uid_validity != 0 && hdr->uid_validity != 0) {
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	616 /* uidvalidity changed and mailbox isn't being initialized,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	617 index must be rebuilt */
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	618 mail_storage_set_critical(ibox->box.storage,
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	619 "Maildir %s sync: UIDVALIDITY changed (%u -> %u)",
3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	620 ibox->path, hdr->uid_validity, uid_validity);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	621 mail_index_mark_corrupted(ibox->index);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	622 maildir_sync_index_abort(sync_ctx);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	623 return -1;
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	624 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	625
2877 3cb483d565a6 Allow specifying to transaction if it's external. Make mail saving Timo Sirainen <tss@iki.fi> parents: 2818 diff changeset	626 trans = mail_index_transaction_begin(view, FALSE, TRUE);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	627 sync_ctx->trans = trans;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	628
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	629 keyword_pool = pool_alloconly_create("maildir keywords", 128);
73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	630
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	631 seq = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	632 iter = maildir_uidlist_iter_init(ibox->uidlist);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	633 while (maildir_uidlist_iter_next(iter, &uid, &uflags, &filename)) {
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	634 p_clear(keyword_pool);
73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	635 maildir_filename_get_flags(filename, keyword_pool,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	636 &flags, &keywords);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	637
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	638 if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RECENT) != 0 &&
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	639 (uflags & MAILDIR_UIDLIST_REC_FLAG_NEW_DIR) != 0 &&
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	640 (uflags & MAILDIR_UIDLIST_REC_FLAG_MOVED) == 0) {
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	641 /* mail is recent for next session as well */
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	642 flags \|= MAIL_RECENT;
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	643 }
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	644
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	645 __again:
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	646 seq++;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	647
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	648 if (seq > hdr->messages_count) {
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	649 if (uid < hdr->next_uid) {
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	650 /* most likely a race condition: we read the
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	651 maildir, then someone else expunged messages
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	652 and committed changes to index. so, this
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	653 message shouldn't actually exist. mark it
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	654 racy and check in next sync.
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	655
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	656 the difference between this and the later
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	657 check is that this one happens when messages
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	658 are expunged from the end */
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	659 if ((uflags &
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	660 MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	661 /* partial syncing */
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	662 continue;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	663 }
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	664 if ((uflags &
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	665 MAILDIR_UIDLIST_REC_FLAG_RACING) != 0) {
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	666 mail_storage_set_critical(
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	667 ibox->box.storage,
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	668 "Maildir %s sync: "
3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	669 "UID < next_uid "
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	670 "(%u < %u, file = %s)",
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	671 ibox->path, uid, hdr->next_uid,
3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	672 filename);
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	673 mail_index_mark_corrupted(ibox->index);
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	674 ret = -1;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	675 break;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	676 }
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	677 ibox->dirty_cur_time = ioloop_time;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	678 maildir_uidlist_add_flags(ibox->uidlist,
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	679 filename,
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	680 MAILDIR_UIDLIST_REC_FLAG_RACING);
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	681
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	682 seq--;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	683 continue;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	684 }
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	685
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	686 mail_index_append(trans, uid, &seq);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	687 mail_index_update_flags(trans, seq, MODIFY_REPLACE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	688 flags);
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	689 // FIXME: set keywords
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	690 continue;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	691 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	692
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	693 if (mail_index_lookup(view, seq, &rec) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	694 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	695 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	696 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	697
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	698 if (rec->uid < uid) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	699 /* expunged */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	700 mail_index_expunge(trans, seq);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	701 goto __again;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	702 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	703
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	704 if (rec->uid > uid) {
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	705 /* most likely a race condition: we read the
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	706 maildir, then someone else expunged messages and
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	707 committed changes to index. so, this message
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	708 shouldn't actually exist. mark it racy and check
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	709 in next sync. */
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	710 if ((uflags &
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	711 MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	712 /* partial syncing */
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	713 seq--;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	714 continue;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	715 }
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	716 if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RACING) != 0) {
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	717 mail_storage_set_critical(ibox->box.storage,
2570 372d4b90c076 cleanup Timo Sirainen <tss@iki.fi> parents: 2533 diff changeset	718 "Maildir %s sync: "
372d4b90c076 cleanup Timo Sirainen <tss@iki.fi> parents: 2533 diff changeset	719 "UID inserted in the middle of mailbox "
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	720 "(%u > %u, file = %s)",
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	721 ibox->path, rec->uid, uid, filename);
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	722 mail_index_mark_corrupted(ibox->index);
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	723 ret = -1;
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	724 break;
2064 5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	725 }
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	726
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	727 ibox->dirty_cur_time = ioloop_time;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	728 maildir_uidlist_add_flags(ibox->uidlist, filename,
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	729 MAILDIR_UIDLIST_REC_FLAG_RACING);
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	730
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	731 seq--;
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	732 continue;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	733 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	734
2320 8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	735 if ((rec->flags & MAIL_RECENT) != 0) {
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	736 index_mailbox_set_recent(ibox, seq);
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	737 if (ibox->keep_recent) {
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	738 flags \|= MAIL_RECENT;
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	739 } else {
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	740 mail_index_update_flags(trans, seq,
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	741 MODIFY_REMOVE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	742 MAIL_RECENT);
2320 8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	743 }
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	744 }
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	745
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	746 if ((uflags & MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	747 /* partial syncing */
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	748 continue;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	749 }
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	750
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	751 if ((rec->flags & MAIL_INDEX_MAIL_FLAG_DIRTY) != 0) {
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	752 /* we haven't been able to update maildir with this
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	753 record's flag changes. don't sync them. */
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	754 continue;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	755 }
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	756
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	757 if (((uint8_t)flags & ~MAIL_RECENT) !=
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	758 (rec->flags & (MAIL_FLAGS_MASK^MAIL_RECENT))) {
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	759 /* FIXME: this is wrong if there's pending changes in
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	760 transaction log already. it gets fixed in next sync
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	761 however.. */
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	762 mail_index_update_flags(trans, seq, MODIFY_REPLACE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	763 flags);
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	764 } else if ((flags & MAIL_RECENT) == 0 &&
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	765 (rec->flags & MAIL_RECENT) != 0) {
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	766 /* just remove recent flag */
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	767 mail_index_update_flags(trans, seq, MODIFY_REMOVE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	768 MAIL_RECENT);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	769 }
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	770 // FIXME: update keywords
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	771 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	772 maildir_uidlist_iter_deinit(iter);
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	773 pool_unref(keyword_pool);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	774
2533 89911d2a9126 Don't break if saving messages to maildir without index files. Timo Sirainen <tss@iki.fi> parents: 2393 diff changeset	775 if (!partial) {
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	776 /* expunge the rest */
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	777 for (seq++; seq <= hdr->messages_count; seq++)
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	778 mail_index_expunge(trans, seq);
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	779 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	780
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	781 /* now, sync the index */
2239 1057a5dc0ca7 assertfix Timo Sirainen <tss@iki.fi> parents: 2228 diff changeset	782 ibox->syncing_commit = TRUE;
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	783 while ((ret = mail_index_sync_next(sync_ctx->sync_ctx,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	784 &sync_ctx->sync_rec)) > 0) {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	785 if (maildir_sync_record(ibox, sync_ctx) < 0) {
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	786 ret = -1;
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	787 break;
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	788 }
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	789 }
2239 1057a5dc0ca7 assertfix Timo Sirainen <tss@iki.fi> parents: 2228 diff changeset	790 ibox->syncing_commit = FALSE;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	791
2064 5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	792 if (ibox->dirty_cur_time == 0 &&
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	793 ibox->last_cur_mtime != (time_t)hdr->sync_stamp) {
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	794 uint32_t sync_stamp = ibox->last_cur_mtime;
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	795
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	796 mail_index_update_header(trans,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	797 offsetof(struct mail_index_header, sync_stamp),
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	798 &sync_stamp, sizeof(sync_stamp));
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	799 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	800
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	801 if (hdr->uid_validity == 0) {
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	802 /* get the initial uidvalidity */
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	803 if (maildir_uidlist_update(ibox->uidlist) < 0)
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	804 ret = -1;
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	805 uid_validity = maildir_uidlist_get_uid_validity(ibox->uidlist);
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	806 if (uid_validity == 0) {
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	807 uid_validity = ioloop_time;
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	808 maildir_uidlist_set_uid_validity(ibox->uidlist,
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	809 uid_validity);
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	810 }
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	811 } else if (uid_validity == 0) {
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	812 maildir_uidlist_set_uid_validity(ibox->uidlist,
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	813 hdr->uid_validity);
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	814 }
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	815
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	816 if (uid_validity != hdr->uid_validity && uid_validity != 0) {
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	817 mail_index_update_header(trans,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	818 offsetof(struct mail_index_header, uid_validity),
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	819 &uid_validity, sizeof(uid_validity));
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	820 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	821
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	822 next_uid = maildir_uidlist_get_next_uid(ibox->uidlist);
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	823 if (next_uid != 0 && hdr->next_uid != next_uid) {
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	824 mail_index_update_header(trans,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	825 offsetof(struct mail_index_header, next_uid),
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	826 &next_uid, sizeof(next_uid));
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	827 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	828
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	829 if (ret < 0) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	830 mail_index_transaction_rollback(trans);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	831 mail_index_sync_rollback(sync_ctx->sync_ctx);
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	832 } else {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	833 uint32_t seq;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	834 uoff_t offset;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	835
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	836 if (mail_index_transaction_commit(trans, &seq, &offset) < 0)
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	837 ret = -1;
2009 2e77ee652e2e When committing with no changes, don't try to sync them later. Timo Sirainen <tss@iki.fi> parents: 1988 diff changeset	838 else if (seq != 0) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	839 ibox->commit_log_file_seq = seq;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	840 ibox->commit_log_file_offset = offset;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	841 }
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	842 if (mail_index_sync_commit(sync_ctx->sync_ctx) < 0)
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	843 ret = -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	844 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	845
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	846 if (ret == 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	847 ibox->commit_log_file_seq = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	848 ibox->commit_log_file_offset = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	849 } else {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	850 mail_storage_set_index_error(ibox);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	851 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	852
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	853 i_free(sync_ctx);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	854 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	855 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	856
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	857 static int maildir_sync_context(struct maildir_sync_context *ctx, int forced)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	858 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	859 int ret, new_changed, cur_changed;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	860
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	861 if (!forced) {
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	862 if (maildir_sync_quick_check(ctx, &new_changed, &cur_changed) < 0)
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	863 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	864
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	865 if (!new_changed && !cur_changed)
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	866 return 0;
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	867 } else {
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	868 new_changed = cur_changed = TRUE;
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	869 }
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	870
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	871 /*
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	872 Locking, locking, locking.. Wasn't maildir supposed to be lockless?
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	873
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	874 We can get here either as beginning a real maildir sync, or when
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	875 committing changes to maildir but a file was lost (maybe renamed).
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	876
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	877 So, we're going to need two locks. One for index and one for
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	878 uidlist. To avoid deadlocking do the index lock first.
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	879
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	880 uidlist is needed only for figuring out UIDs for newly seen files,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	881 so theoretically we wouldn't need to lock it unless there are new
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	882 files. It has a few problems though, assuming the index lock didn't
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	883 already protect it (eg. in-memory indexes):
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	884
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	885 1. Just because you see a new file which doesn't exist in uidlist
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	886 file, doesn't mean that the file really exists anymore, or that
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	887 your readdir() lists all new files. Meaning that this is possible:
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	888
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	889 A: opendir(), readdir() -> new file ...
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	890 -- new files are written to the maildir --
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	891 B: opendir(), readdir() -> new file, lock uidlist,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	892 readdir() -> another new file, rewrite uidlist, unlock
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	893 A: ... lock uidlist, readdir() -> nothing left, rewrite uidlist,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	894 unlock
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	895
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	896 The second time running A didn't see the two new files. To handle
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	897 this correctly, it must not remove the new unseen files from
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	898 uidlist. This is possible to do, but adds extra complexity.
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	899
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	900 2. If another process is rename()ing files while we are
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	901 readdir()ing, it's possible that readdir() never lists some files,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	902 causing Dovecot to assume they were expunged. In next sync they
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	903 would show up again, but client could have already been notified of
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	904 that and they would show up under new UIDs, so the damage is
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	905 already done.
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	906
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	907 Both of the problems can be avoided if we simply lock the uidlist
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	908 before syncing and keep it until sync is finished. Typically this
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	909 would happen in any case, as there is the index lock..
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	910
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	911 The second case is still a problem with external changes though,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	912 because maildir doesn't require any kind of locking. Luckily this
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	913 problem rarely happens except under high amount of modifications.
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	914 */
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	915
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	916 if (!ctx->ibox->syncing_commit) {
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	917 ctx->index_sync_ctx = maildir_sync_index_begin(ctx->ibox);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	918 if (ctx->index_sync_ctx == NULL)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	919 return -1;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	920 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	921
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	922 if ((ret = maildir_uidlist_lock(ctx->ibox->uidlist)) <= 0) {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	923 /* failure / timeout. if forced is TRUE, we could still go
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	924 forward and check only for renamed files, but is it worth
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	925 the trouble? .. */
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	926 return ret;
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	927 }
2123 e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	928
e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	929 ctx->partial = !cur_changed;
e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	930 ctx->uidlist_sync_ctx =
e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	931 maildir_uidlist_sync_init(ctx->ibox->uidlist, ctx->partial);
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	932
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	933 if (maildir_scan_dir(ctx, TRUE) < 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	934 return -1;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	935 if (cur_changed) {
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	936 if (maildir_scan_dir(ctx, FALSE) < 0)
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	937 return -1;
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	938 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	939
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	940 /* finish uidlist syncing, but keep it still locked */
2123 e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	941 maildir_uidlist_sync_finish(ctx->uidlist_sync_ctx);
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	942 if (!ctx->ibox->syncing_commit) {
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	943 if (maildir_sync_index_finish(ctx->index_sync_ctx,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	944 ctx->partial) < 0) {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	945 ctx->index_sync_ctx = NULL;
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	946 return -1;
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	947 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	948 ctx->index_sync_ctx = NULL;
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	949 }
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	950
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	951 ret = maildir_uidlist_sync_deinit(ctx->uidlist_sync_ctx);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	952 ctx->uidlist_sync_ctx = NULL;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	953
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	954 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	955 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	956
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	957 int maildir_storage_sync_force(struct index_mailbox *ibox)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	958 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	959 struct maildir_sync_context *ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	960 int ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	961
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	962 ctx = maildir_sync_context_new(ibox);
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	963 ret = maildir_sync_context(ctx, TRUE);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	964 maildir_sync_deinit(ctx);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	965 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	966 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	967
2322 aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	968 struct mailbox_sync_context *
aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	969 maildir_storage_sync_init(struct mailbox *box, enum mailbox_sync_flags flags)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	970 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	971 struct index_mailbox ibox = (struct index_mailbox )box;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	972 struct maildir_sync_context *ctx;
2322 aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	973 int ret = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	974
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	975 if ((flags & MAILBOX_SYNC_FLAG_FAST) == 0 \|\|
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	976 ibox->sync_last_check + MAILBOX_FULL_SYNC_INTERVAL <= ioloop_time) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	977 ibox->sync_last_check = ioloop_time;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	978
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	979 ctx = maildir_sync_context_new(ibox);
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	980 ret = maildir_sync_context(ctx, FALSE);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	981 maildir_sync_deinit(ctx);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	982 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	983
2322 aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	984 return index_mailbox_sync_init(box, flags, ret < 0);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	985 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-storage/index/maildir/maildir-sync.c @ 3085:73944f902165 HEAD