dovecot/original-hg/dovecot-1.2: src/lib-storage/index/maildir/maildir-sync.c annotate

annotate src/lib-storage/index/maildir/maildir-sync.c @ 3203:391a1064872e HEAD

mail_index_sync_next() now returns keyword updates (now only thing left to do is to fix maildir and mbox syncing to use them).

author	Timo Sirainen <tss@iki.fi>
date	Mon, 14 Mar 2005 22:48:25 +0200
parents	73944f902165
children	b698ae839a18

rev	line source
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	1 /* Copyright (C) 2004 Timo Sirainen */
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	2
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	3 /*
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	4 Here's a description of how we handle Maildir synchronization and
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	5 it's problems:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	6
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	7 We want to be as efficient as we can. The most efficient way to
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	8 check if changes have occured is to stat() the new/ and cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	9 directories and uidlist file - if their mtimes haven't changed,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	10 there's no changes and we don't need to do anything.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	11
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	12 Problem 1: Multiple changes can happen within a single second -
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	13 nothing guarantees that once we synced it, someone else didn't just
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	14 then make a modification. Such modifications wouldn't get noticed
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	15 until a new modification occured later.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	16
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	17 Problem 2: Syncing cur/ directory is much more costly than syncing
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	18 new/. Moving mails from new/ to cur/ will always change mtime of
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	19 cur/ causing us to sync it as well.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	20
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	21 Problem 3: We may not be able to move mail from new/ to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	22 because we're out of quota, or simply because we're accessing a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	23 read-only mailbox.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	24
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	25
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	26 MAILDIR_SYNC_SECS
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	27 -----------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	28
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	29 Several checks below use MAILDIR_SYNC_SECS, which should be maximum
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	30 clock drift between all computers accessing the maildir (eg. via
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	31 NFS), rounded up to next second. Our default is 1 second, since
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	32 everyone should be using NTP.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	33
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	34 Note that setting it to 0 works only if there's only one computer
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	35 accessing the maildir. It's practically impossible to make two
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	36 clocks _exactly_ synchronized.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	37
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	38 It might be possible to only use file server's clock by looking at
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	39 the atime field, but I don't know how well that would actually work.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	40
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	41 cur directory
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	42 -------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	43
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	44 We have dirty_cur_time variable which is set to cur/ directory's
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	45 mtime when it's >= time() - MAILDIR_SYNC_SECS and we _think_ we have
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	46 synchronized the directory.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	47
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	48 When dirty_cur_time is non-zero, we don't synchronize the cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	49 directory until
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	50
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	51 a) cur/'s mtime changes
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	52 b) opening a mail fails with ENOENT
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	53 c) time() > dirty_cur_time + MAILDIR_SYNC_SECS
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	54
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	55 This allows us to modify the maildir multiple times without having
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	56 to sync it at every change. The sync will eventually be done to
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	57 make sure we didn't miss any external changes.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	58
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	59 The dirty_cur_time is set when:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	60
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	61 - we change message flags
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	62 - we expunge messages
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	63 - we move mail from new/ to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	64 - we sync cur/ directory and it's mtime is >= time() - MAILDIR_SYNC_SECS
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	65
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	66 It's unset when we do the final syncing, ie. when mtime is
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	67 older than time() - MAILDIR_SYNC_SECS.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	68
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	69 new directory
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	70 -------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	71
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	72 If new/'s mtime is >= time() - MAILDIR_SYNC_SECS, always synchronize
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	73 it. dirty_cur_time-like feature might save us a few syncs, but
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	74 that might break a client which saves a mail in one connection and
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	75 tries to fetch it in another one. new/ directory is almost always
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	76 empty, so syncing it should be very fast anyway. Actually this can
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	77 still happen if we sync only new/ dir while another client is also
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	78 moving mails from it to cur/ - it takes us a while to see them.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	79 That's pretty unlikely to happen however, and only way to fix it
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	80 would be to always synchronize cur/ after new/.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	81
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	82 Normally we move all mails from new/ to cur/ whenever we sync it. If
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	83 it's not possible for some reason, we mark the mail with "probably
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	84 exists in new/ directory" flag.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	85
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	86 If rename() still fails because of ENOSPC or EDQUOT, we still save
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	87 the flag changes in index with dirty-flag on. When moving the mail
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	88 to cur/ directory, or when we notice it's already moved there, we
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	89 apply the flag changes to the filename, rename it and remove the
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	90 dirty flag. If there's dirty flags, this should be tried every time
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	91 after expunge or when closing the mailbox.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	92
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	93 uidlist
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	94 -------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	95
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	96 This file contains UID <-> filename mappings. It's updated only when
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	97 new mail arrives, so it may contain filenames that have already been
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	98 deleted. Updating is done by getting uidlist.lock file, writing the
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	99 whole uidlist into it and rename()ing it over the old uidlist. This
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	100 means there's no need to lock the file for reading.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	101
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	102 Whenever uidlist is rewritten, it's mtime must be larger than the old
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	103 one's. Use utime() before rename() if needed. Note that inode checking
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	104 wouldn't have been sufficient as inode numbers can be reused.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	105
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	106 This file is usually read the first time you need to know filename for
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	107 given UID. After that it's not re-read unless new mails come that we
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	108 don't know about.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	109
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	110 broken clients
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	111 --------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	112
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	113 Originally the middle identifier in Maildir filename was specified
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	114 only as <process id>_<delivery counter>. That however created a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	115 problem with randomized PIDs which made it possible that the same
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	116 PID was reused within one second.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	117
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	118 So if within one second a mail was delivered, MUA moved it to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	119 and another mail was delivered by a new process using same PID as
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	120 the first one, we likely ended up overwriting the first mail when
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	121 the second mail was moved over it.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	122
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	123 Nowadays everyone should be giving a bit more specific identifier,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	124 for example include microseconds in it which Dovecot does.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	125
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	126 There's a simple way to prevent this from happening in some cases:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	127 Don't move the mail from new/ to cur/ if it's mtime is >= time() -
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	128 MAILDIR_SYNC_SECS. The second delivery's link() call then fails
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	129 because the file is already in new/, and it will then use a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	130 different filename. There's a few problems with this however:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	131
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	132 - it requires extra stat() call which is unneeded extra I/O
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	133 - another MUA might still move the mail to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	134 - if first file's flags are modified by either Dovecot or another
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	135 MUA, it's moved to cur/ (you _could_ just do the dirty-flagging
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	136 but that'd be ugly)
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	137
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	138 Because this is useful only for very few people and it requires
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	139 extra I/O, I decided not to implement this. It should be however
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	140 quite easy to do since we need to be able to deal with files in new/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	141 in any case.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	142
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	143 It's also possible to never accidentally overwrite a mail by using
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	144 link() + unlink() rather than rename(). This however isn't very
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	145 good idea as it introduces potential race conditions when multiple
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	146 clients are accessing the mailbox:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	147
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	148 Trying to move the same mail from new/ to cur/ at the same time:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	149
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	150 a) Client 1 uses slightly different filename than client 2,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	151 for example one sets read-flag on but the other doesn't.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	152 You have the same mail duplicated now.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	153
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	154 b) Client 3 sees the mail between Client 1's and 2's link() calls
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	155 and changes it's flag. You have the same mail duplicated now.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	156
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	157 And it gets worse when they're unlink()ing in cur/ directory:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	158
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	159 c) Client 1 changes mails's flag and client 2 changes it back
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	160 between 1's link() and unlink(). The mail is now expunged.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	161
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	162 d) If you try to deal with the duplicates by unlink()ing another
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	163 one of them, you might end up unlinking both of them.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	164
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	165 So, what should we do then if we notice a duplicate? First of all,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	166 it might not be a duplicate at all, readdir() might have just
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	167 returned it twice because it was just renamed. What we should do is
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	168 create a completely new base name for it and rename() it to that.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	169 If the call fails with ENOENT, it only means that it wasn't a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	170 duplicate after all.
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	171 */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	172
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	173 #include "lib.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	174 #include "ioloop.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	175 #include "buffer.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	176 #include "hash.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	177 #include "str.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	178 #include "maildir-storage.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	179 #include "maildir-uidlist.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	180
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	181 #include <stdio.h>
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	182 #include <stddef.h>
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	183 #include <unistd.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	184 #include <dirent.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	185 #include <sys/stat.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	186
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	187 #define MAILDIR_SYNC_SECS 1
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	188
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	189 #define MAILDIR_FILENAME_FLAG_FOUND 128
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	190
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	191 struct maildir_sync_context {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	192 struct index_mailbox *ibox;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	193 const char new_dir, cur_dir;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	194 int partial;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	195
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	196 struct maildir_uidlist_sync_ctx *uidlist_sync_ctx;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	197 struct maildir_index_sync_context *index_sync_ctx;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	198 };
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	199
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	200 struct maildir_index_sync_context {
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	201 struct index_mailbox *ibox;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	202 struct mail_index_view *view;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	203 struct mail_index_sync_ctx *sync_ctx;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	204 struct mail_index_transaction *trans;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	205
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	206 struct mail_index_sync_rec sync_rec;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	207 uint32_t seq;
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	208 int dirty_state;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	209 };
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	210
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	211 static int maildir_expunge(struct index_mailbox ibox, const char path,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	212 void *context __attr_unused__)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	213 {
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	214 if (unlink(path) == 0) {
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	215 ibox->dirty_cur_time = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	216 return 1;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	217 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	218 if (errno == ENOENT)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	219 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	220
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	221 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	222 "unlink(%s) failed: %m", path);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	223 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	224 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	225
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	226 static int maildir_sync_flags(struct index_mailbox ibox, const char path,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	227 void *context)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	228 {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	229 struct maildir_index_sync_context *ctx = context;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	230 const char *newpath;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	231 enum mail_flags flags;
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	232 const char const keywords;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	233 uint8_t flags8;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	234
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	235 ctx->dirty_state = 0;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	236
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	237 (void)maildir_filename_get_flags(path, pool_datastack_create(),
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	238 &flags, &keywords);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	239
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	240 flags8 = flags;
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	241 mail_index_sync_flags_apply(&ctx->sync_rec, &flags8);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	242
1958 4dec6a3d79fd s/custom flags/keywords/ Timo Sirainen <tss@iki.fi> parents: 1956 diff changeset	243 newpath = maildir_filename_set_flags(path, flags8, keywords);
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	244 if (rename(path, newpath) == 0) {
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	245 if ((flags8 & MAIL_INDEX_MAIL_FLAG_DIRTY) != 0)
ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	246 ctx->dirty_state = -1;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	247 ibox->dirty_cur_time = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	248 return 1;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	249 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	250 if (errno == ENOENT)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	251 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	252
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	253 if (ENOSPACE(errno) \|\| errno == EACCES) {
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	254 mail_index_update_flags(ctx->trans, ctx->seq, MODIFY_ADD,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	255 MAIL_INDEX_MAIL_FLAG_DIRTY);
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	256 ctx->dirty_state = 1;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	257 return 1;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	258 }
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	259
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	260 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	261 "rename(%s, %s) failed: %m", path, newpath);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	262 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	263 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	264
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	265 static int maildir_sync_record(struct index_mailbox *ibox,
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	266 struct maildir_index_sync_context *ctx)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	267 {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	268 struct mail_index_sync_rec *sync_rec = &ctx->sync_rec;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	269 struct mail_index_view *view = ctx->view;
2033 4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	270 uint32_t seq, seq1, seq2, uid;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	271
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	272 switch (sync_rec->type) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	273 case MAIL_INDEX_SYNC_TYPE_APPEND:
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	274 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	275 case MAIL_INDEX_SYNC_TYPE_EXPUNGE:
2033 4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	276 /* make it go through sequences to avoid looping through huge
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	277 holes in UID range */
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	278 if (mail_index_lookup_uid_range(view, sync_rec->uid1,
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	279 sync_rec->uid2,
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	280 &seq1, &seq2) < 0)
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	281 return -1;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	282
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	283 if (seq1 == 0)
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	284 break;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	285
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	286 for (seq = seq1; seq <= seq2; seq++) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	287 if (mail_index_lookup_uid(view, seq, &uid) < 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	288 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	289 if (maildir_file_do(ibox, uid, maildir_expunge,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	290 NULL) < 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	291 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	292 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	293 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	294 case MAIL_INDEX_SYNC_TYPE_FLAGS:
2033 4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	295 if (mail_index_lookup_uid_range(view, sync_rec->uid1,
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	296 sync_rec->uid2,
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	297 &seq1, &seq2) < 0)
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	298 return -1;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	299
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	300 if (seq1 == 0)
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	301 break;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	302
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	303 for (ctx->seq = seq1; ctx->seq <= seq2; ctx->seq++) {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	304 if (mail_index_lookup_uid(view, ctx->seq, &uid) < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	305 return -1;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	306 if (maildir_file_do(ibox, uid,
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	307 maildir_sync_flags, ctx) < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	308 return -1;
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	309 if (ctx->dirty_state < 0) {
ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	310 /* flag isn't dirty anymore */
ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	311 mail_index_update_flags(ctx->trans, ctx->seq,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	312 MODIFY_REMOVE,
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	313 MAIL_INDEX_MAIL_FLAG_DIRTY);
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	314 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	315 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	316 break;
3203 391a1064872e mail_index_sync_next() now returns keyword updates (now only thing left to Timo Sirainen <tss@iki.fi> parents: 3085 diff changeset	317 case MAIL_INDEX_SYNC_TYPE_KEYWORD_ADD:
391a1064872e mail_index_sync_next() now returns keyword updates (now only thing left to Timo Sirainen <tss@iki.fi> parents: 3085 diff changeset	318 case MAIL_INDEX_SYNC_TYPE_KEYWORD_REMOVE:
391a1064872e mail_index_sync_next() now returns keyword updates (now only thing left to Timo Sirainen <tss@iki.fi> parents: 3085 diff changeset	319 case MAIL_INDEX_SYNC_TYPE_KEYWORD_RESET:
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	320 // FIXME
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	321 break;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	322 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	323
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	324 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	325 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	326
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	327 int maildir_sync_last_commit(struct index_mailbox *ibox)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	328 {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	329 struct maildir_index_sync_context ctx;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	330 uint32_t seq;
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	331 uoff_t offset;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	332 int ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	333
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	334 if (ibox->commit_log_file_seq == 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	335 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	336
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	337 memset(&ctx, 0, sizeof(ctx));
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	338 ctx.ibox = ibox;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	339
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	340 ibox->syncing_commit = TRUE;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	341 ret = mail_index_sync_begin(ibox->index, &ctx.sync_ctx, &ctx.view,
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	342 ibox->commit_log_file_seq,
2685 52352a5859e4 When we had dirty messages, we kept resyncing the whole mailbox constantly. Timo Sirainen <tss@iki.fi> parents: 2570 diff changeset	343 ibox->commit_log_file_offset, FALSE, FALSE);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	344 if (ret > 0) {
2877 3cb483d565a6 Allow specifying to transaction if it's external. Make mail saving Timo Sirainen <tss@iki.fi> parents: 2818 diff changeset	345 ctx.trans = mail_index_transaction_begin(ctx.view, FALSE, TRUE);
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	346
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	347 while ((ret = mail_index_sync_next(ctx.sync_ctx,
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	348 &ctx.sync_rec)) > 0) {
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	349 if (maildir_sync_record(ibox, &ctx) < 0) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	350 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	351 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	352 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	353 }
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	354 if (mail_index_transaction_commit(ctx.trans, &seq, &offset) < 0)
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	355 ret = -1;
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	356 if (mail_index_sync_commit(ctx.sync_ctx) < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	357 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	358 }
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	359 ibox->syncing_commit = FALSE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	360
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	361 if (ret == 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	362 ibox->commit_log_file_seq = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	363 ibox->commit_log_file_offset = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	364 } else {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	365 mail_storage_set_index_error(ibox);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	366 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	367 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	368 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	369
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	370 static struct maildir_sync_context *
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	371 maildir_sync_context_new(struct index_mailbox *ibox)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	372 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	373 struct maildir_sync_context *ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	374
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	375 ctx = t_new(struct maildir_sync_context, 1);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	376 ctx->ibox = ibox;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	377 ctx->new_dir = t_strconcat(ibox->path, "/new", NULL);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	378 ctx->cur_dir = t_strconcat(ibox->path, "/cur", NULL);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	379 return ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	380 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	381
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	382 static void maildir_sync_deinit(struct maildir_sync_context *ctx)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	383 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	384 if (ctx->uidlist_sync_ctx != NULL)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	385 (void)maildir_uidlist_sync_deinit(ctx->uidlist_sync_ctx);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	386 if (ctx->index_sync_ctx != NULL)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	387 maildir_sync_index_abort(ctx->index_sync_ctx);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	388 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	389
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	390 static int maildir_fix_duplicate(struct index_mailbox ibox, const char dir,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	391 const char *old_fname)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	392 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	393 const char new_fname, old_path, *new_path;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	394 int ret = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	395
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	396 t_push();
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	397
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	398 old_path = t_strconcat(dir, "/", old_fname, NULL);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	399 new_fname = maildir_generate_tmp_filename(&ioloop_timeval);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	400 new_path = t_strconcat(ibox->path, "/new/", new_fname, NULL);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	401
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	402 if (rename(old_path, new_path) == 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	403 i_warning("Fixed duplicate in %s: %s -> %s",
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	404 ibox->path, old_fname, new_fname);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	405 } else if (errno != ENOENT) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	406 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	407 "rename(%s, %s) failed: %m", old_path, new_path);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	408 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	409 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	410 t_pop();
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	411
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	412 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	413 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	414
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	415 static int maildir_scan_dir(struct maildir_sync_context *ctx, int new_dir)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	416 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	417 struct mail_storage *storage = ctx->ibox->box.storage;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	418 const char *dir;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	419 DIR *dirp;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	420 string_t src, dest;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	421 struct dirent *dp;
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	422 enum maildir_uidlist_rec_flag flags;
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	423 int move_new, ret = 1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	424
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	425 dir = new_dir ? ctx->new_dir : ctx->cur_dir;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	426 dirp = opendir(dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	427 if (dirp == NULL) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	428 mail_storage_set_critical(storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	429 "opendir(%s) failed: %m", dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	430 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	431 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	432
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	433 t_push();
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	434 src = t_str_new(1024);
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	435 dest = t_str_new(1024);
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	436
2039 f0925b2271e1 Added pop3_mails_keep_recent option. Fixed recent assert crash. Timo Sirainen <tss@iki.fi> parents: 2038 diff changeset	437 move_new = new_dir && !mailbox_is_readonly(&ctx->ibox->box) &&
f0925b2271e1 Added pop3_mails_keep_recent option. Fixed recent assert crash. Timo Sirainen <tss@iki.fi> parents: 2038 diff changeset	438 !ctx->ibox->keep_recent;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	439 while ((dp = readdir(dirp)) != NULL) {
2258 087a43e29492 No maildir filename checking after all. Timo Sirainen <tss@iki.fi> parents: 2256 diff changeset	440 if (dp->d_name[0] == '.')
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	441 continue;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	442
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	443 ret = maildir_uidlist_sync_next_pre(ctx->uidlist_sync_ctx,
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	444 dp->d_name);
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	445 if (ret == 0) {
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	446 /* new file and we couldn't lock uidlist, check this
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	447 later in next sync. */
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	448 if (new_dir)
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	449 ctx->ibox->last_new_mtime = 0;
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	450 else
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	451 ctx->ibox->dirty_cur_time = ioloop_time;
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	452 continue;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	453 }
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	454 if (ret < 0)
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	455 break;
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	456
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	457 flags = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	458 if (move_new) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	459 str_truncate(src, 0);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	460 str_truncate(dest, 0);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	461 str_printfa(src, "%s/%s", ctx->new_dir, dp->d_name);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	462 str_printfa(dest, "%s/%s", ctx->cur_dir, dp->d_name);
1970 0f4c4a043be1 When moving mails to cur/, we must add the ":2," part to filename. Timo Sirainen <tss@iki.fi> parents: 1962 diff changeset	463 if (strchr(dp->d_name, ':') == NULL)
0f4c4a043be1 When moving mails to cur/, we must add the ":2," part to filename. Timo Sirainen <tss@iki.fi> parents: 1962 diff changeset	464 str_append(dest, ":2,");
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	465 if (rename(str_c(src), str_c(dest)) == 0) {
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	466 /* we moved it - it's \Recent for us */
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	467 ctx->ibox->dirty_cur_time = ioloop_time;
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	468 flags \|= MAILDIR_UIDLIST_REC_FLAG_MOVED \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	469 MAILDIR_UIDLIST_REC_FLAG_RECENT;
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	470 } else if (ENOTFOUND(errno)) {
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	471 /* someone else moved it already */
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	472 flags \|= MAILDIR_UIDLIST_REC_FLAG_MOVED;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	473 } else if (ENOSPACE(errno)) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	474 /* not enough disk space, leave here */
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	475 flags \|= MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	476 MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	477 move_new = FALSE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	478 } else {
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	479 flags \|= MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	480 MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	481 mail_storage_set_critical(storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	482 "rename(%s, %s) failed: %m",
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	483 str_c(src), str_c(dest));
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	484 }
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	485 } else if (new_dir) {
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	486 flags \|= MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	487 MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	488 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	489
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	490 ret = maildir_uidlist_sync_next(ctx->uidlist_sync_ctx,
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	491 dp->d_name, flags);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	492 if (ret <= 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	493 if (ret < 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	494 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	495
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	496 /* possibly duplicate - try fixing it */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	497 if (maildir_fix_duplicate(ctx->ibox,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	498 dir, dp->d_name) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	499 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	500 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	501 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	502 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	503 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	504
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	505 if (closedir(dirp) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	506 mail_storage_set_critical(storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	507 "closedir(%s) failed: %m", dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	508 }
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	509
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	510 t_pop();
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	511 return ret < 0 ? -1 : 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	512 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	513
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	514 static int maildir_sync_quick_check(struct maildir_sync_context *ctx,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	515 int new_changed_r, int cur_changed_r)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	516 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	517 struct index_mailbox *ibox = ctx->ibox;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	518 struct stat st;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	519 time_t new_mtime, cur_mtime;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	520
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	521 new_changed_r = cur_changed_r = FALSE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	522
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	523 if (stat(ctx->new_dir, &st) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	524 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	525 "stat(%s) failed: %m", ctx->new_dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	526 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	527 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	528 new_mtime = st.st_mtime;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	529
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	530 if (stat(ctx->cur_dir, &st) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	531 mail_storage_set_critical(ibox->box.storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	532 "stat(%s) failed: %m", ctx->cur_dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	533 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	534 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	535 cur_mtime = st.st_mtime;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	536
2893 fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	537 /* cur stamp is kept in index, we don't have to sync if
fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	538 someone else has done it and updated the index. */
fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	539 ibox->last_cur_mtime = mail_index_get_header(ibox->view)->sync_stamp;
fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	540 if (ibox->dirty_cur_time == 0 && cur_mtime != ibox->last_cur_mtime) {
fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	541 /* check if the index has been updated.. */
2936 57f975a9b63b Renamed mail_index_refresh() to mail_index_reopen_if_needed(). Added public Timo Sirainen <tss@iki.fi> parents: 2893 diff changeset	542 if (mail_index_refresh(ibox->index) < 0) {
2064 5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	543 mail_storage_set_index_error(ibox);
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	544 return -1;
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	545 }
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	546
2936 57f975a9b63b Renamed mail_index_refresh() to mail_index_reopen_if_needed(). Added public Timo Sirainen <tss@iki.fi> parents: 2893 diff changeset	547 ibox->last_cur_mtime =
57f975a9b63b Renamed mail_index_refresh() to mail_index_reopen_if_needed(). Added public Timo Sirainen <tss@iki.fi> parents: 2893 diff changeset	548 mail_index_get_header(ibox->view)->sync_stamp;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	549 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	550
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	551 if (new_mtime != ibox->last_new_mtime \|\|
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	552 new_mtime >= ibox->last_new_sync_time - MAILDIR_SYNC_SECS) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	553 *new_changed_r = TRUE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	554 ibox->last_new_mtime = new_mtime;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	555 ibox->last_new_sync_time = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	556 }
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	557
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	558 if (cur_mtime != ibox->last_cur_mtime \|\|
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1958 diff changeset	559 (ibox->dirty_cur_time != 0 &&
4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1958 diff changeset	560 ioloop_time - ibox->dirty_cur_time > MAILDIR_SYNC_SECS)) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	561 /* cur/ changed, or delayed cur/ check */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	562 *cur_changed_r = TRUE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	563 ibox->last_cur_mtime = cur_mtime;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	564
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	565 ibox->dirty_cur_time =
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	566 cur_mtime >= ioloop_time - MAILDIR_SYNC_SECS ?
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	567 cur_mtime : 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	568 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	569
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	570 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	571 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	572
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	573 struct maildir_index_sync_context *
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	574 maildir_sync_index_begin(struct index_mailbox *ibox)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	575 {
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	576 struct maildir_index_sync_context *sync_ctx;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	577
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	578 sync_ctx = i_new(struct maildir_index_sync_context, 1);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	579 sync_ctx->ibox = ibox;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	580
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	581 if (mail_index_sync_begin(ibox->index, &sync_ctx->sync_ctx,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	582 &sync_ctx->view, (uint32_t)-1, (uoff_t)-1,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	583 FALSE, FALSE) <= 0) {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	584 mail_storage_set_index_error(ibox);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	585 return NULL;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	586 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	587 return sync_ctx;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	588 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	589
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	590 void maildir_sync_index_abort(struct maildir_index_sync_context *sync_ctx)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	591 {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	592 mail_index_sync_rollback(sync_ctx->sync_ctx);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	593 i_free(sync_ctx);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	594 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	595
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	596 int maildir_sync_index_finish(struct maildir_index_sync_context *sync_ctx,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	597 int partial)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	598 {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	599 struct index_mailbox *ibox = sync_ctx->ibox;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	600 struct mail_index_view *view = sync_ctx->view;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	601 struct maildir_uidlist_iter_ctx *iter;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	602 struct mail_index_transaction *trans;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	603 const struct mail_index_header *hdr;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	604 const struct mail_index_record *rec;
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	605 pool_t keyword_pool;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	606 uint32_t seq, uid;
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	607 enum maildir_uidlist_rec_flag uflags;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	608 const char *filename;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	609 enum mail_flags flags;
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	610 const char const keywords;
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	611 uint32_t uid_validity, next_uid;
1917 68938dccbc45 Forced locking to be right with mprotect()ing index file. Support for Timo Sirainen <tss@iki.fi> parents: 1915 diff changeset	612 int ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	613
2892 62d53b49110d Changed mail_index_get_header() to return the header as return value because Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	614 hdr = mail_index_get_header(view);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	615 uid_validity = maildir_uidlist_get_uid_validity(ibox->uidlist);
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	616 if (uid_validity != hdr->uid_validity &&
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	617 uid_validity != 0 && hdr->uid_validity != 0) {
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	618 /* uidvalidity changed and mailbox isn't being initialized,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	619 index must be rebuilt */
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	620 mail_storage_set_critical(ibox->box.storage,
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	621 "Maildir %s sync: UIDVALIDITY changed (%u -> %u)",
3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	622 ibox->path, hdr->uid_validity, uid_validity);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	623 mail_index_mark_corrupted(ibox->index);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	624 maildir_sync_index_abort(sync_ctx);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	625 return -1;
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	626 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	627
2877 3cb483d565a6 Allow specifying to transaction if it's external. Make mail saving Timo Sirainen <tss@iki.fi> parents: 2818 diff changeset	628 trans = mail_index_transaction_begin(view, FALSE, TRUE);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	629 sync_ctx->trans = trans;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	630
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	631 keyword_pool = pool_alloconly_create("maildir keywords", 128);
73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	632
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	633 seq = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	634 iter = maildir_uidlist_iter_init(ibox->uidlist);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	635 while (maildir_uidlist_iter_next(iter, &uid, &uflags, &filename)) {
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	636 p_clear(keyword_pool);
73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	637 maildir_filename_get_flags(filename, keyword_pool,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	638 &flags, &keywords);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	639
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	640 if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RECENT) != 0 &&
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	641 (uflags & MAILDIR_UIDLIST_REC_FLAG_NEW_DIR) != 0 &&
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	642 (uflags & MAILDIR_UIDLIST_REC_FLAG_MOVED) == 0) {
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	643 /* mail is recent for next session as well */
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	644 flags \|= MAIL_RECENT;
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	645 }
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	646
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	647 __again:
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	648 seq++;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	649
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	650 if (seq > hdr->messages_count) {
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	651 if (uid < hdr->next_uid) {
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	652 /* most likely a race condition: we read the
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	653 maildir, then someone else expunged messages
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	654 and committed changes to index. so, this
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	655 message shouldn't actually exist. mark it
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	656 racy and check in next sync.
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	657
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	658 the difference between this and the later
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	659 check is that this one happens when messages
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	660 are expunged from the end */
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	661 if ((uflags &
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	662 MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	663 /* partial syncing */
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	664 continue;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	665 }
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	666 if ((uflags &
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	667 MAILDIR_UIDLIST_REC_FLAG_RACING) != 0) {
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	668 mail_storage_set_critical(
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	669 ibox->box.storage,
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	670 "Maildir %s sync: "
3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	671 "UID < next_uid "
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	672 "(%u < %u, file = %s)",
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	673 ibox->path, uid, hdr->next_uid,
3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	674 filename);
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	675 mail_index_mark_corrupted(ibox->index);
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	676 ret = -1;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	677 break;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	678 }
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	679 ibox->dirty_cur_time = ioloop_time;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	680 maildir_uidlist_add_flags(ibox->uidlist,
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	681 filename,
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	682 MAILDIR_UIDLIST_REC_FLAG_RACING);
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	683
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	684 seq--;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	685 continue;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	686 }
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	687
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	688 mail_index_append(trans, uid, &seq);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	689 mail_index_update_flags(trans, seq, MODIFY_REPLACE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	690 flags);
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	691 // FIXME: set keywords
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	692 continue;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	693 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	694
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	695 if (mail_index_lookup(view, seq, &rec) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	696 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	697 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	698 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	699
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	700 if (rec->uid < uid) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	701 /* expunged */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	702 mail_index_expunge(trans, seq);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	703 goto __again;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	704 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	705
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	706 if (rec->uid > uid) {
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	707 /* most likely a race condition: we read the
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	708 maildir, then someone else expunged messages and
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	709 committed changes to index. so, this message
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	710 shouldn't actually exist. mark it racy and check
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	711 in next sync. */
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	712 if ((uflags &
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	713 MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	714 /* partial syncing */
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	715 seq--;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	716 continue;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	717 }
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	718 if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RACING) != 0) {
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	719 mail_storage_set_critical(ibox->box.storage,
2570 372d4b90c076 cleanup Timo Sirainen <tss@iki.fi> parents: 2533 diff changeset	720 "Maildir %s sync: "
372d4b90c076 cleanup Timo Sirainen <tss@iki.fi> parents: 2533 diff changeset	721 "UID inserted in the middle of mailbox "
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	722 "(%u > %u, file = %s)",
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	723 ibox->path, rec->uid, uid, filename);
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	724 mail_index_mark_corrupted(ibox->index);
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	725 ret = -1;
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	726 break;
2064 5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	727 }
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	728
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	729 ibox->dirty_cur_time = ioloop_time;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	730 maildir_uidlist_add_flags(ibox->uidlist, filename,
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	731 MAILDIR_UIDLIST_REC_FLAG_RACING);
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	732
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	733 seq--;
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	734 continue;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	735 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	736
2320 8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	737 if ((rec->flags & MAIL_RECENT) != 0) {
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	738 index_mailbox_set_recent(ibox, seq);
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	739 if (ibox->keep_recent) {
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	740 flags \|= MAIL_RECENT;
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	741 } else {
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	742 mail_index_update_flags(trans, seq,
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	743 MODIFY_REMOVE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	744 MAIL_RECENT);
2320 8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	745 }
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	746 }
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	747
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	748 if ((uflags & MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	749 /* partial syncing */
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	750 continue;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	751 }
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	752
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	753 if ((rec->flags & MAIL_INDEX_MAIL_FLAG_DIRTY) != 0) {
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	754 /* we haven't been able to update maildir with this
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	755 record's flag changes. don't sync them. */
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	756 continue;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	757 }
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	758
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	759 if (((uint8_t)flags & ~MAIL_RECENT) !=
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	760 (rec->flags & (MAIL_FLAGS_MASK^MAIL_RECENT))) {
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	761 /* FIXME: this is wrong if there's pending changes in
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	762 transaction log already. it gets fixed in next sync
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	763 however.. */
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	764 mail_index_update_flags(trans, seq, MODIFY_REPLACE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	765 flags);
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	766 } else if ((flags & MAIL_RECENT) == 0 &&
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	767 (rec->flags & MAIL_RECENT) != 0) {
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	768 /* just remove recent flag */
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	769 mail_index_update_flags(trans, seq, MODIFY_REMOVE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	770 MAIL_RECENT);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	771 }
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	772 // FIXME: update keywords
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	773 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	774 maildir_uidlist_iter_deinit(iter);
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	775 pool_unref(keyword_pool);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	776
2533 89911d2a9126 Don't break if saving messages to maildir without index files. Timo Sirainen <tss@iki.fi> parents: 2393 diff changeset	777 if (!partial) {
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	778 /* expunge the rest */
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	779 for (seq++; seq <= hdr->messages_count; seq++)
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	780 mail_index_expunge(trans, seq);
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	781 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	782
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	783 /* now, sync the index */
2239 1057a5dc0ca7 assertfix Timo Sirainen <tss@iki.fi> parents: 2228 diff changeset	784 ibox->syncing_commit = TRUE;
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	785 while ((ret = mail_index_sync_next(sync_ctx->sync_ctx,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	786 &sync_ctx->sync_rec)) > 0) {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	787 if (maildir_sync_record(ibox, sync_ctx) < 0) {
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	788 ret = -1;
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	789 break;
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	790 }
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	791 }
2239 1057a5dc0ca7 assertfix Timo Sirainen <tss@iki.fi> parents: 2228 diff changeset	792 ibox->syncing_commit = FALSE;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	793
2064 5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	794 if (ibox->dirty_cur_time == 0 &&
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	795 ibox->last_cur_mtime != (time_t)hdr->sync_stamp) {
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	796 uint32_t sync_stamp = ibox->last_cur_mtime;
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	797
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	798 mail_index_update_header(trans,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	799 offsetof(struct mail_index_header, sync_stamp),
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	800 &sync_stamp, sizeof(sync_stamp));
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	801 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	802
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	803 if (hdr->uid_validity == 0) {
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	804 /* get the initial uidvalidity */
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	805 if (maildir_uidlist_update(ibox->uidlist) < 0)
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	806 ret = -1;
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	807 uid_validity = maildir_uidlist_get_uid_validity(ibox->uidlist);
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	808 if (uid_validity == 0) {
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	809 uid_validity = ioloop_time;
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	810 maildir_uidlist_set_uid_validity(ibox->uidlist,
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	811 uid_validity);
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	812 }
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	813 } else if (uid_validity == 0) {
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	814 maildir_uidlist_set_uid_validity(ibox->uidlist,
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	815 hdr->uid_validity);
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	816 }
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	817
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	818 if (uid_validity != hdr->uid_validity && uid_validity != 0) {
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	819 mail_index_update_header(trans,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	820 offsetof(struct mail_index_header, uid_validity),
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	821 &uid_validity, sizeof(uid_validity));
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	822 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	823
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	824 next_uid = maildir_uidlist_get_next_uid(ibox->uidlist);
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	825 if (next_uid != 0 && hdr->next_uid != next_uid) {
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	826 mail_index_update_header(trans,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	827 offsetof(struct mail_index_header, next_uid),
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	828 &next_uid, sizeof(next_uid));
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	829 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	830
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	831 if (ret < 0) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	832 mail_index_transaction_rollback(trans);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	833 mail_index_sync_rollback(sync_ctx->sync_ctx);
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	834 } else {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	835 uint32_t seq;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	836 uoff_t offset;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	837
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	838 if (mail_index_transaction_commit(trans, &seq, &offset) < 0)
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	839 ret = -1;
2009 2e77ee652e2e When committing with no changes, don't try to sync them later. Timo Sirainen <tss@iki.fi> parents: 1988 diff changeset	840 else if (seq != 0) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	841 ibox->commit_log_file_seq = seq;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	842 ibox->commit_log_file_offset = offset;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	843 }
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	844 if (mail_index_sync_commit(sync_ctx->sync_ctx) < 0)
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	845 ret = -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	846 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	847
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	848 if (ret == 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	849 ibox->commit_log_file_seq = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	850 ibox->commit_log_file_offset = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	851 } else {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	852 mail_storage_set_index_error(ibox);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	853 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	854
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	855 i_free(sync_ctx);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	856 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	857 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	858
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	859 static int maildir_sync_context(struct maildir_sync_context *ctx, int forced)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	860 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	861 int ret, new_changed, cur_changed;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	862
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	863 if (!forced) {
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	864 if (maildir_sync_quick_check(ctx, &new_changed, &cur_changed) < 0)
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	865 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	866
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	867 if (!new_changed && !cur_changed)
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	868 return 0;
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	869 } else {
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	870 new_changed = cur_changed = TRUE;
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	871 }
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	872
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	873 /*
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	874 Locking, locking, locking.. Wasn't maildir supposed to be lockless?
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	875
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	876 We can get here either as beginning a real maildir sync, or when
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	877 committing changes to maildir but a file was lost (maybe renamed).
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	878
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	879 So, we're going to need two locks. One for index and one for
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	880 uidlist. To avoid deadlocking do the index lock first.
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	881
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	882 uidlist is needed only for figuring out UIDs for newly seen files,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	883 so theoretically we wouldn't need to lock it unless there are new
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	884 files. It has a few problems though, assuming the index lock didn't
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	885 already protect it (eg. in-memory indexes):
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	886
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	887 1. Just because you see a new file which doesn't exist in uidlist
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	888 file, doesn't mean that the file really exists anymore, or that
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	889 your readdir() lists all new files. Meaning that this is possible:
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	890
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	891 A: opendir(), readdir() -> new file ...
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	892 -- new files are written to the maildir --
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	893 B: opendir(), readdir() -> new file, lock uidlist,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	894 readdir() -> another new file, rewrite uidlist, unlock
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	895 A: ... lock uidlist, readdir() -> nothing left, rewrite uidlist,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	896 unlock
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	897
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	898 The second time running A didn't see the two new files. To handle
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	899 this correctly, it must not remove the new unseen files from
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	900 uidlist. This is possible to do, but adds extra complexity.
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	901
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	902 2. If another process is rename()ing files while we are
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	903 readdir()ing, it's possible that readdir() never lists some files,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	904 causing Dovecot to assume they were expunged. In next sync they
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	905 would show up again, but client could have already been notified of
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	906 that and they would show up under new UIDs, so the damage is
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	907 already done.
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	908
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	909 Both of the problems can be avoided if we simply lock the uidlist
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	910 before syncing and keep it until sync is finished. Typically this
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	911 would happen in any case, as there is the index lock..
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	912
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	913 The second case is still a problem with external changes though,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	914 because maildir doesn't require any kind of locking. Luckily this
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	915 problem rarely happens except under high amount of modifications.
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	916 */
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	917
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	918 if (!ctx->ibox->syncing_commit) {
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	919 ctx->index_sync_ctx = maildir_sync_index_begin(ctx->ibox);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	920 if (ctx->index_sync_ctx == NULL)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	921 return -1;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	922 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	923
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	924 if ((ret = maildir_uidlist_lock(ctx->ibox->uidlist)) <= 0) {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	925 /* failure / timeout. if forced is TRUE, we could still go
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	926 forward and check only for renamed files, but is it worth
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	927 the trouble? .. */
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	928 return ret;
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	929 }
2123 e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	930
e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	931 ctx->partial = !cur_changed;
e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	932 ctx->uidlist_sync_ctx =
e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	933 maildir_uidlist_sync_init(ctx->ibox->uidlist, ctx->partial);
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	934
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	935 if (maildir_scan_dir(ctx, TRUE) < 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	936 return -1;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	937 if (cur_changed) {
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	938 if (maildir_scan_dir(ctx, FALSE) < 0)
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	939 return -1;
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	940 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	941
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	942 /* finish uidlist syncing, but keep it still locked */
2123 e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	943 maildir_uidlist_sync_finish(ctx->uidlist_sync_ctx);
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	944 if (!ctx->ibox->syncing_commit) {
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	945 if (maildir_sync_index_finish(ctx->index_sync_ctx,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	946 ctx->partial) < 0) {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	947 ctx->index_sync_ctx = NULL;
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	948 return -1;
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	949 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	950 ctx->index_sync_ctx = NULL;
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	951 }
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	952
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	953 ret = maildir_uidlist_sync_deinit(ctx->uidlist_sync_ctx);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	954 ctx->uidlist_sync_ctx = NULL;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	955
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	956 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	957 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	958
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	959 int maildir_storage_sync_force(struct index_mailbox *ibox)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	960 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	961 struct maildir_sync_context *ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	962 int ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	963
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	964 ctx = maildir_sync_context_new(ibox);
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	965 ret = maildir_sync_context(ctx, TRUE);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	966 maildir_sync_deinit(ctx);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	967 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	968 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	969
2322 aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	970 struct mailbox_sync_context *
aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	971 maildir_storage_sync_init(struct mailbox *box, enum mailbox_sync_flags flags)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	972 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	973 struct index_mailbox ibox = (struct index_mailbox )box;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	974 struct maildir_sync_context *ctx;
2322 aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	975 int ret = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	976
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	977 if ((flags & MAILBOX_SYNC_FLAG_FAST) == 0 \|\|
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	978 ibox->sync_last_check + MAILBOX_FULL_SYNC_INTERVAL <= ioloop_time) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	979 ibox->sync_last_check = ioloop_time;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	980
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	981 ctx = maildir_sync_context_new(ibox);
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	982 ret = maildir_sync_context(ctx, FALSE);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	983 maildir_sync_deinit(ctx);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	984 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	985
2322 aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	986 return index_mailbox_sync_init(box, flags, ret < 0);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	987 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-storage/index/maildir/maildir-sync.c @ 3203:391a1064872e HEAD