dovecot/original-hg/dovecot-1.2: src/lib-storage/index/maildir/maildir-sync.c annotate

annotate src/lib-storage/index/maildir/maildir-sync.c @ 3446:113c888cdca1 HEAD

Merge changes from multiple index sync records into one before actually renaming maildir files.

author	Timo Sirainen <tss@iki.fi>
date	Wed, 29 Jun 2005 01:32:38 +0300
parents	3c51658d6846
children	96de02ea7482

rev	line source
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	1 /* Copyright (C) 2004 Timo Sirainen */
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	2
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	3 /*
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	4 Here's a description of how we handle Maildir synchronization and
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	5 it's problems:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	6
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	7 We want to be as efficient as we can. The most efficient way to
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	8 check if changes have occured is to stat() the new/ and cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	9 directories and uidlist file - if their mtimes haven't changed,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	10 there's no changes and we don't need to do anything.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	11
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	12 Problem 1: Multiple changes can happen within a single second -
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	13 nothing guarantees that once we synced it, someone else didn't just
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	14 then make a modification. Such modifications wouldn't get noticed
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	15 until a new modification occured later.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	16
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	17 Problem 2: Syncing cur/ directory is much more costly than syncing
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	18 new/. Moving mails from new/ to cur/ will always change mtime of
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	19 cur/ causing us to sync it as well.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	20
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	21 Problem 3: We may not be able to move mail from new/ to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	22 because we're out of quota, or simply because we're accessing a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	23 read-only mailbox.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	24
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	25
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	26 MAILDIR_SYNC_SECS
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	27 -----------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	28
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	29 Several checks below use MAILDIR_SYNC_SECS, which should be maximum
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	30 clock drift between all computers accessing the maildir (eg. via
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	31 NFS), rounded up to next second. Our default is 1 second, since
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	32 everyone should be using NTP.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	33
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	34 Note that setting it to 0 works only if there's only one computer
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	35 accessing the maildir. It's practically impossible to make two
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	36 clocks _exactly_ synchronized.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	37
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	38 It might be possible to only use file server's clock by looking at
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	39 the atime field, but I don't know how well that would actually work.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	40
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	41 cur directory
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	42 -------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	43
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	44 We have dirty_cur_time variable which is set to cur/ directory's
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	45 mtime when it's >= time() - MAILDIR_SYNC_SECS and we _think_ we have
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	46 synchronized the directory.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	47
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	48 When dirty_cur_time is non-zero, we don't synchronize the cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	49 directory until
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	50
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	51 a) cur/'s mtime changes
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	52 b) opening a mail fails with ENOENT
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	53 c) time() > dirty_cur_time + MAILDIR_SYNC_SECS
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	54
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	55 This allows us to modify the maildir multiple times without having
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	56 to sync it at every change. The sync will eventually be done to
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	57 make sure we didn't miss any external changes.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	58
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	59 The dirty_cur_time is set when:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	60
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	61 - we change message flags
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	62 - we expunge messages
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	63 - we move mail from new/ to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	64 - we sync cur/ directory and it's mtime is >= time() - MAILDIR_SYNC_SECS
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	65
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	66 It's unset when we do the final syncing, ie. when mtime is
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	67 older than time() - MAILDIR_SYNC_SECS.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	68
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	69 new directory
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	70 -------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	71
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	72 If new/'s mtime is >= time() - MAILDIR_SYNC_SECS, always synchronize
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	73 it. dirty_cur_time-like feature might save us a few syncs, but
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	74 that might break a client which saves a mail in one connection and
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	75 tries to fetch it in another one. new/ directory is almost always
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	76 empty, so syncing it should be very fast anyway. Actually this can
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	77 still happen if we sync only new/ dir while another client is also
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	78 moving mails from it to cur/ - it takes us a while to see them.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	79 That's pretty unlikely to happen however, and only way to fix it
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	80 would be to always synchronize cur/ after new/.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	81
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	82 Normally we move all mails from new/ to cur/ whenever we sync it. If
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	83 it's not possible for some reason, we mark the mail with "probably
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	84 exists in new/ directory" flag.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	85
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	86 If rename() still fails because of ENOSPC or EDQUOT, we still save
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	87 the flag changes in index with dirty-flag on. When moving the mail
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	88 to cur/ directory, or when we notice it's already moved there, we
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	89 apply the flag changes to the filename, rename it and remove the
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	90 dirty flag. If there's dirty flags, this should be tried every time
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	91 after expunge or when closing the mailbox.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	92
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	93 uidlist
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	94 -------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	95
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	96 This file contains UID <-> filename mappings. It's updated only when
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	97 new mail arrives, so it may contain filenames that have already been
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	98 deleted. Updating is done by getting uidlist.lock file, writing the
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	99 whole uidlist into it and rename()ing it over the old uidlist. This
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	100 means there's no need to lock the file for reading.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	101
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	102 Whenever uidlist is rewritten, it's mtime must be larger than the old
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	103 one's. Use utime() before rename() if needed. Note that inode checking
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	104 wouldn't have been sufficient as inode numbers can be reused.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	105
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	106 This file is usually read the first time you need to know filename for
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	107 given UID. After that it's not re-read unless new mails come that we
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	108 don't know about.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	109
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	110 broken clients
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	111 --------------
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	112
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	113 Originally the middle identifier in Maildir filename was specified
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	114 only as <process id>_<delivery counter>. That however created a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	115 problem with randomized PIDs which made it possible that the same
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	116 PID was reused within one second.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	117
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	118 So if within one second a mail was delivered, MUA moved it to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	119 and another mail was delivered by a new process using same PID as
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	120 the first one, we likely ended up overwriting the first mail when
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	121 the second mail was moved over it.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	122
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	123 Nowadays everyone should be giving a bit more specific identifier,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	124 for example include microseconds in it which Dovecot does.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	125
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	126 There's a simple way to prevent this from happening in some cases:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	127 Don't move the mail from new/ to cur/ if it's mtime is >= time() -
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	128 MAILDIR_SYNC_SECS. The second delivery's link() call then fails
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	129 because the file is already in new/, and it will then use a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	130 different filename. There's a few problems with this however:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	131
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	132 - it requires extra stat() call which is unneeded extra I/O
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	133 - another MUA might still move the mail to cur/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	134 - if first file's flags are modified by either Dovecot or another
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	135 MUA, it's moved to cur/ (you _could_ just do the dirty-flagging
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	136 but that'd be ugly)
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	137
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	138 Because this is useful only for very few people and it requires
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	139 extra I/O, I decided not to implement this. It should be however
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	140 quite easy to do since we need to be able to deal with files in new/
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	141 in any case.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	142
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	143 It's also possible to never accidentally overwrite a mail by using
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	144 link() + unlink() rather than rename(). This however isn't very
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	145 good idea as it introduces potential race conditions when multiple
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	146 clients are accessing the mailbox:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	147
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	148 Trying to move the same mail from new/ to cur/ at the same time:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	149
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	150 a) Client 1 uses slightly different filename than client 2,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	151 for example one sets read-flag on but the other doesn't.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	152 You have the same mail duplicated now.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	153
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	154 b) Client 3 sees the mail between Client 1's and 2's link() calls
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	155 and changes it's flag. You have the same mail duplicated now.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	156
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	157 And it gets worse when they're unlink()ing in cur/ directory:
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	158
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	159 c) Client 1 changes mails's flag and client 2 changes it back
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	160 between 1's link() and unlink(). The mail is now expunged.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	161
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	162 d) If you try to deal with the duplicates by unlink()ing another
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	163 one of them, you might end up unlinking both of them.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	164
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	165 So, what should we do then if we notice a duplicate? First of all,
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	166 it might not be a duplicate at all, readdir() might have just
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	167 returned it twice because it was just renamed. What we should do is
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	168 create a completely new base name for it and rename() it to that.
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	169 If the call fails with ENOENT, it only means that it wasn't a
0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	170 duplicate after all.
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	171 */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	172
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	173 #include "lib.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	174 #include "ioloop.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	175 #include "buffer.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	176 #include "hash.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	177 #include "str.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	178 #include "maildir-storage.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	179 #include "maildir-uidlist.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	180
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	181 #include <stdio.h>
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	182 #include <stddef.h>
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	183 #include <unistd.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	184 #include <dirent.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	185 #include <sys/stat.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	186
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	187 #define MAILDIR_SYNC_SECS 1
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	188
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	189 #define MAILDIR_FILENAME_FLAG_FOUND 128
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	190
3435 6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	191 /* When rename()ing many files from new/ to cur/, it's possible that next
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	192 readdir() skips some files. we don't of course wish to lose them, so we
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	193 go and rescan the new/ directory again from beginning until no files are
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	194 left. This value is just an optimization to avoid checking the directory
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	195 twice unneededly. usually only NFS is the problem case. 1 is the safest
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	196 bet here, but I guess 5 will do just fine too. */
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	197 #define MAILDIR_RENAME_RESCAN_COUNT 5
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	198
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	199 struct maildir_sync_context {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	200 struct maildir_mailbox *mbox;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	201 const char new_dir, cur_dir;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	202 int partial;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	203
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	204 struct maildir_uidlist_sync_ctx *uidlist_sync_ctx;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	205 struct maildir_index_sync_context *index_sync_ctx;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	206 };
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	207
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	208 struct maildir_index_sync_context {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	209 struct maildir_mailbox *mbox;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	210 struct mail_index_view *view;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	211 struct mail_index_sync_ctx *sync_ctx;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	212 struct mail_index_transaction *trans;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	213
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	214 array_t ARRAY_DEFINE(sync_recs, struct mail_index_sync_rec);
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	215 uint32_t seq;
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	216 int dirty_state;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	217 };
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	218
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	219 static int maildir_expunge(struct maildir_mailbox mbox, const char path,
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	220 void *context __attr_unused__)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	221 {
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	222 if (unlink(path) == 0) {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	223 mbox->dirty_cur_time = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	224 return 1;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	225 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	226 if (errno == ENOENT)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	227 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	228
3280 2c72492dfd91 Created mbox_storage and maildir_storage. Timo Sirainen <tss@iki.fi> parents: 3279 diff changeset	229 mail_storage_set_critical(STORAGE(mbox->storage),
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	230 "unlink(%s) failed: %m", path);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	231 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	232 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	233
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	234 static int maildir_sync_flags(struct maildir_mailbox mbox, const char path,
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	235 void *context)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	236 {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	237 struct maildir_index_sync_context *ctx = context;
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	238 const struct mail_index_sync_rec *recs;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	239 const char *newpath;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	240 enum mail_flags flags;
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	241 const char const keywords;
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	242 unsigned int i, count;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	243 uint8_t flags8;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	244
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	245 ctx->dirty_state = 0;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	246
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	247 (void)maildir_filename_get_flags(path, pool_datastack_create(),
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	248 &flags, &keywords);
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	249 flags8 = flags;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	250
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	251 recs = array_get_modifyable(&ctx->sync_recs, &count);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	252 for (i = 0; i < count; i++) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	253 if (recs[i].uid1 != ctx->seq)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	254 break;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	255
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	256 switch (recs[i].type) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	257 case MAIL_INDEX_SYNC_TYPE_FLAGS:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	258 mail_index_sync_flags_apply(&recs[i], &flags8);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	259 break;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	260 case MAIL_INDEX_SYNC_TYPE_KEYWORD_ADD:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	261 case MAIL_INDEX_SYNC_TYPE_KEYWORD_REMOVE:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	262 case MAIL_INDEX_SYNC_TYPE_KEYWORD_RESET:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	263 /FIXME:mail_index_sync_keywords_apply(&recs[i], &keywords);/
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	264 break;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	265 case MAIL_INDEX_SYNC_TYPE_APPEND:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	266 case MAIL_INDEX_SYNC_TYPE_EXPUNGE:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	267 i_unreached();
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	268 break;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	269 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	270 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	271
1958 4dec6a3d79fd s/custom flags/keywords/ Timo Sirainen <tss@iki.fi> parents: 1956 diff changeset	272 newpath = maildir_filename_set_flags(path, flags8, keywords);
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	273 if (rename(path, newpath) == 0) {
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	274 if ((flags8 & MAIL_INDEX_MAIL_FLAG_DIRTY) != 0)
ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	275 ctx->dirty_state = -1;
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	276 mbox->dirty_cur_time = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	277 return 1;
1955 0f0128b4af5d More syncing changes Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	278 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	279 if (errno == ENOENT)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	280 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	281
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	282 if (ENOSPACE(errno) \|\| errno == EACCES) {
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	283 mail_index_update_flags(ctx->trans, ctx->seq, MODIFY_ADD,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	284 MAIL_INDEX_MAIL_FLAG_DIRTY);
2272 ced88553af0b mail_index_sync_sort_flags() now merges flag changes so mail storage Timo Sirainen <tss@iki.fi> parents: 2258 diff changeset	285 ctx->dirty_state = 1;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	286 return 1;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	287 }
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	288
3280 2c72492dfd91 Created mbox_storage and maildir_storage. Timo Sirainen <tss@iki.fi> parents: 3279 diff changeset	289 mail_storage_set_critical(STORAGE(mbox->storage),
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	290 "rename(%s, %s) failed: %m", path, newpath);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	291 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	292 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	293
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	294 static int
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	295 maildir_sync_record_commit_until(struct maildir_index_sync_context *ctx,
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	296 uint32_t last_seq)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	297 {
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	298 struct mail_index_sync_rec *recs;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	299 unsigned int i, count;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	300 uint32_t seq, uid;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	301 int expunged, flag_changed;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	302
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	303 recs = array_get_modifyable(&ctx->sync_recs, &count);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	304 for (seq = recs[0].uid1; seq <= last_seq; seq++) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	305 expunged = flag_changed = FALSE;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	306 for (i = 0; i < count; i++) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	307 if (recs[i].uid1 > seq)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	308 break;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	309
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	310 i_assert(recs[i].uid1 == seq);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	311 switch (recs[i].type) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	312 case MAIL_INDEX_SYNC_TYPE_EXPUNGE:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	313 expunged = TRUE;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	314 break;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	315 case MAIL_INDEX_SYNC_TYPE_FLAGS:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	316 case MAIL_INDEX_SYNC_TYPE_KEYWORD_ADD:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	317 case MAIL_INDEX_SYNC_TYPE_KEYWORD_REMOVE:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	318 case MAIL_INDEX_SYNC_TYPE_KEYWORD_RESET:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	319 flag_changed = TRUE;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	320 break;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	321 case MAIL_INDEX_SYNC_TYPE_APPEND:
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	322 i_unreached();
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	323 break;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	324 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	325 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	326
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	327 if (mail_index_lookup_uid(ctx->view, seq, &uid) < 0)
2033 4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	328 return -1;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	329
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	330 ctx->seq = seq;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	331 if (expunged) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	332 if (maildir_file_do(ctx->mbox, uid,
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	333 maildir_expunge, ctx) < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	334 return -1;
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	335 } else if (flag_changed) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	336 if (maildir_file_do(ctx->mbox, uid,
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	337 maildir_sync_flags, ctx) < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	338 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	339 }
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	340
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	341 for (i = count; i > 0; i--) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	342 if (++recs[i-1].uid1 > recs[i-1].uid2) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	343 array_delete(&ctx->sync_recs, i-1, 1);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	344 recs = array_get_modifyable(&ctx->sync_recs,
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	345 &count);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	346 if (count == 0) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	347 /* all sync_recs committed */
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	348 return 0;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	349 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	350 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	351 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	352 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	353 return 0;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	354 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	355
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	356 static int maildir_sync_record(struct maildir_index_sync_context *ctx,
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	357 const struct mail_index_sync_rec *sync_rec)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	358 {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	359 struct mail_index_view *view = ctx->view;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	360 struct mail_index_sync_rec sync_copy;
2033 4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	361
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	362 if (sync_rec == NULL) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	363 /* deinit */
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	364 while (array_count(&ctx->sync_recs) > 0) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	365 if (maildir_sync_record_commit_until(ctx,
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	366 (uint32_t)-1) < 0)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	367 return -1;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	368 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	369 return 0;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	370 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	371
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	372 if (sync_rec->type == MAIL_INDEX_SYNC_TYPE_APPEND)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	373 return 0; /* ignore */
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	374
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	375 /* convert to sequences to avoid looping through huge holes in
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	376 UID range */
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	377 sync_copy = *sync_rec;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	378 if (mail_index_lookup_uid_range(view, sync_rec->uid1,
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	379 sync_rec->uid2,
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	380 &sync_copy.uid1,
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	381 &sync_copy.uid2) < 0)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	382 return -1;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	383
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	384 while (array_count(&ctx->sync_recs) > 0) {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	385 const struct mail_index_sync_rec *rec =
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	386 array_idx(&ctx->sync_recs, 0);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	387
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	388 i_assert(rec->uid1 <= sync_copy.uid1);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	389 if (rec->uid1 == sync_copy.uid1)
2033 4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	390 break;
4f6b1118a53d Transaction log contains only UIDs now, no more sequences which just mess up Timo Sirainen <tss@iki.fi> parents: 2009 diff changeset	391
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	392 if (maildir_sync_record_commit_until(ctx, sync_copy.uid1-1) < 0)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	393 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	394 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	395
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	396 array_append(&ctx->sync_recs, &sync_copy, 1);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	397 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	398 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	399
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	400 static int maildir_sync_index_records(struct maildir_index_sync_context *ctx)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	401 {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	402 struct mail_index_sync_rec sync_rec;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	403 int ret;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	404
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	405 ret = mail_index_sync_next(ctx->sync_ctx, &sync_rec);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	406 if (ret <= 0)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	407 return ret;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	408
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	409 ARRAY_CREATE(&ctx->sync_recs, pool_datastack_create(),
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	410 struct mail_index_sync_rec, 32);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	411 do {
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	412 if (maildir_sync_record(ctx, &sync_rec) < 0)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	413 return -1;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	414
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	415 ret = mail_index_sync_next(ctx->sync_ctx, &sync_rec);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	416 } while (ret > 0);
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	417
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	418 if (maildir_sync_record(ctx, NULL) < 0)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	419 return -1;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	420 return ret;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	421 }
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	422
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	423 int maildir_sync_last_commit(struct maildir_mailbox *mbox)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	424 {
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	425 struct maildir_index_sync_context ctx;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	426 uint32_t seq;
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	427 uoff_t offset;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	428 int ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	429
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	430 if (mbox->ibox.commit_log_file_seq == 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	431 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	432
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	433 memset(&ctx, 0, sizeof(ctx));
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	434 ctx.mbox = mbox;
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	435
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	436 mbox->syncing_commit = TRUE;
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	437 ret = mail_index_sync_begin(mbox->ibox.index, &ctx.sync_ctx, &ctx.view,
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	438 mbox->ibox.commit_log_file_seq,
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	439 mbox->ibox.commit_log_file_offset,
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	440 FALSE, FALSE);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	441 if (ret > 0) {
2877 3cb483d565a6 Allow specifying to transaction if it's external. Make mail saving Timo Sirainen <tss@iki.fi> parents: 2818 diff changeset	442 ctx.trans = mail_index_transaction_begin(ctx.view, FALSE, TRUE);
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	443 if (maildir_sync_index_records(&ctx) < 0)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	444 ret = -1;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	445 if (mail_index_transaction_commit(ctx.trans, &seq, &offset) < 0)
8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	446 ret = -1;
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	447 if (mail_index_sync_commit(ctx.sync_ctx) < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	448 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	449 }
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	450 mbox->syncing_commit = FALSE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	451
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	452 if (ret == 0) {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	453 mbox->ibox.commit_log_file_seq = 0;
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	454 mbox->ibox.commit_log_file_offset = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	455 } else {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	456 mail_storage_set_index_error(&mbox->ibox);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	457 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	458 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	459 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	460
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	461 static struct maildir_sync_context *
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	462 maildir_sync_context_new(struct maildir_mailbox *mbox)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	463 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	464 struct maildir_sync_context *ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	465
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	466 ctx = t_new(struct maildir_sync_context, 1);
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	467 ctx->mbox = mbox;
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	468 ctx->new_dir = t_strconcat(mbox->path, "/new", NULL);
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	469 ctx->cur_dir = t_strconcat(mbox->path, "/cur", NULL);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	470 return ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	471 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	472
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	473 static void maildir_sync_deinit(struct maildir_sync_context *ctx)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	474 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	475 if (ctx->uidlist_sync_ctx != NULL)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	476 (void)maildir_uidlist_sync_deinit(ctx->uidlist_sync_ctx);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	477 if (ctx->index_sync_ctx != NULL)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	478 maildir_sync_index_abort(ctx->index_sync_ctx);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	479 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	480
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	481 static int maildir_fix_duplicate(struct maildir_mailbox mbox, const char dir,
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	482 const char *old_fname)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	483 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	484 const char new_fname, old_path, *new_path;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	485 int ret = 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	486
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	487 t_push();
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	488
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	489 old_path = t_strconcat(dir, "/", old_fname, NULL);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	490 new_fname = maildir_generate_tmp_filename(&ioloop_timeval);
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	491 new_path = t_strconcat(mbox->path, "/new/", new_fname, NULL);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	492
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	493 if (rename(old_path, new_path) == 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	494 i_warning("Fixed duplicate in %s: %s -> %s",
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	495 mbox->path, old_fname, new_fname);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	496 } else if (errno != ENOENT) {
3280 2c72492dfd91 Created mbox_storage and maildir_storage. Timo Sirainen <tss@iki.fi> parents: 3279 diff changeset	497 mail_storage_set_critical(STORAGE(mbox->storage),
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	498 "rename(%s, %s) failed: %m", old_path, new_path);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	499 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	500 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	501 t_pop();
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	502
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	503 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	504 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	505
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	506 static int maildir_scan_dir(struct maildir_sync_context *ctx, int new_dir)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	507 {
3280 2c72492dfd91 Created mbox_storage and maildir_storage. Timo Sirainen <tss@iki.fi> parents: 3279 diff changeset	508 struct mail_storage *storage = STORAGE(ctx->mbox->storage);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	509 const char *dir;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	510 DIR *dirp;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	511 string_t src, dest;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	512 struct dirent *dp;
3435 6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	513 enum maildir_uidlist_rec_flag flags;
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	514 unsigned int moves = 0;
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	515 int move_new, ret = 1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	516
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	517 dir = new_dir ? ctx->new_dir : ctx->cur_dir;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	518 dirp = opendir(dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	519 if (dirp == NULL) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	520 mail_storage_set_critical(storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	521 "opendir(%s) failed: %m", dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	522 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	523 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	524
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	525 t_push();
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	526 src = t_str_new(1024);
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	527 dest = t_str_new(1024);
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	528
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	529 move_new = new_dir && !mailbox_is_readonly(&ctx->mbox->ibox.box) &&
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	530 !ctx->mbox->ibox.keep_recent;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	531 while ((dp = readdir(dirp)) != NULL) {
2258 087a43e29492 No maildir filename checking after all. Timo Sirainen <tss@iki.fi> parents: 2256 diff changeset	532 if (dp->d_name[0] == '.')
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	533 continue;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	534
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	535 ret = maildir_uidlist_sync_next_pre(ctx->uidlist_sync_ctx,
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	536 dp->d_name);
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	537 if (ret == 0) {
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	538 /* new file and we couldn't lock uidlist, check this
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	539 later in next sync. */
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	540 if (new_dir)
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	541 ctx->mbox->last_new_mtime = 0;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	542 else
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	543 ctx->mbox->dirty_cur_time = ioloop_time;
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	544 continue;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	545 }
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	546 if (ret < 0)
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	547 break;
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1970 diff changeset	548
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	549 flags = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	550 if (move_new) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	551 str_truncate(src, 0);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	552 str_truncate(dest, 0);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	553 str_printfa(src, "%s/%s", ctx->new_dir, dp->d_name);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	554 str_printfa(dest, "%s/%s", ctx->cur_dir, dp->d_name);
3417 b0bdf32564b7 Replaced ':' and ',' character usages with #defines, so they can be changed Timo Sirainen <tss@iki.fi> parents: 3340 diff changeset	555 if (strchr(dp->d_name, MAILDIR_INFO_SEP) == NULL) {
b0bdf32564b7 Replaced ':' and ',' character usages with #defines, so they can be changed Timo Sirainen <tss@iki.fi> parents: 3340 diff changeset	556 str_append(dest, MAILDIR_FLAGS_FULL_SEP);
b0bdf32564b7 Replaced ':' and ',' character usages with #defines, so they can be changed Timo Sirainen <tss@iki.fi> parents: 3340 diff changeset	557 }
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	558 if (rename(str_c(src), str_c(dest)) == 0) {
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	559 /* we moved it - it's \Recent for us */
3435 6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	560 moves++;
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	561 ctx->mbox->dirty_cur_time = ioloop_time;
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	562 flags \|= MAILDIR_UIDLIST_REC_FLAG_MOVED \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	563 MAILDIR_UIDLIST_REC_FLAG_RECENT;
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	564 } else if (ENOTFOUND(errno)) {
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	565 /* someone else moved it already */
3435 6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	566 moves++;
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	567 flags \|= MAILDIR_UIDLIST_REC_FLAG_MOVED;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	568 } else if (ENOSPACE(errno)) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	569 /* not enough disk space, leave here */
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	570 flags \|= MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	571 MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	572 move_new = FALSE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	573 } else {
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	574 flags \|= MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	575 MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	576 mail_storage_set_critical(storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	577 "rename(%s, %s) failed: %m",
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	578 str_c(src), str_c(dest));
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	579 }
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	580 } else if (new_dir) {
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	581 flags \|= MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	582 MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	583 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	584
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	585 ret = maildir_uidlist_sync_next(ctx->uidlist_sync_ctx,
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	586 dp->d_name, flags);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	587 if (ret <= 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	588 if (ret < 0)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	589 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	590
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	591 /* possibly duplicate - try fixing it */
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	592 if (maildir_fix_duplicate(ctx->mbox,
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	593 dir, dp->d_name) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	594 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	595 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	596 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	597 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	598 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	599
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	600 if (closedir(dirp) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	601 mail_storage_set_critical(storage,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	602 "closedir(%s) failed: %m", dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	603 }
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	604
0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	605 t_pop();
3435 6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	606 return ret < 0 ? -1 : (moves <= MAILDIR_RENAME_RESCAN_COUNT ? 0 : 1);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	607 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	608
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	609 static int maildir_sync_quick_check(struct maildir_sync_context *ctx,
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	610 int new_changed_r, int cur_changed_r)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	611 {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	612 struct maildir_mailbox *mbox = ctx->mbox;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	613 struct stat st;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	614 time_t new_mtime, cur_mtime;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	615
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	616 new_changed_r = cur_changed_r = FALSE;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	617
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	618 if (stat(ctx->new_dir, &st) < 0) {
3280 2c72492dfd91 Created mbox_storage and maildir_storage. Timo Sirainen <tss@iki.fi> parents: 3279 diff changeset	619 mail_storage_set_critical(STORAGE(mbox->storage),
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	620 "stat(%s) failed: %m", ctx->new_dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	621 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	622 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	623 new_mtime = st.st_mtime;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	624
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	625 if (stat(ctx->cur_dir, &st) < 0) {
3280 2c72492dfd91 Created mbox_storage and maildir_storage. Timo Sirainen <tss@iki.fi> parents: 3279 diff changeset	626 mail_storage_set_critical(STORAGE(mbox->storage),
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	627 "stat(%s) failed: %m", ctx->cur_dir);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	628 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	629 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	630 cur_mtime = st.st_mtime;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	631
2893 fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	632 /* cur stamp is kept in index, we don't have to sync if
fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	633 someone else has done it and updated the index. */
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	634 mbox->last_cur_mtime =
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	635 mail_index_get_header(mbox->ibox.view)->sync_stamp;
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	636 if (mbox->dirty_cur_time == 0 && cur_mtime != mbox->last_cur_mtime) {
2893 fd431866c674 mail_index_refresh() isn't public anymore, mail_index_view_open_locked() Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	637 /* check if the index has been updated.. */
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	638 if (mail_index_refresh(mbox->ibox.index) < 0) {
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	639 mail_storage_set_index_error(&mbox->ibox);
2064 5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	640 return -1;
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	641 }
5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	642
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	643 mbox->last_cur_mtime =
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	644 mail_index_get_header(mbox->ibox.view)->sync_stamp;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	645 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	646
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	647 if (new_mtime != mbox->last_new_mtime \|\|
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	648 new_mtime >= mbox->last_new_sync_time - MAILDIR_SYNC_SECS) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	649 *new_changed_r = TRUE;
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	650 mbox->last_new_mtime = new_mtime;
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	651 mbox->last_new_sync_time = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	652 }
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	653
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	654 if (cur_mtime != mbox->last_cur_mtime \|\|
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	655 (mbox->dirty_cur_time != 0 &&
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	656 ioloop_time - mbox->dirty_cur_time > MAILDIR_SYNC_SECS)) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	657 /* cur/ changed, or delayed cur/ check */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	658 *cur_changed_r = TRUE;
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	659 mbox->last_cur_mtime = cur_mtime;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	660
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	661 mbox->dirty_cur_time =
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	662 cur_mtime >= ioloop_time - MAILDIR_SYNC_SECS ?
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	663 cur_mtime : 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	664 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	665
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	666 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	667 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	668
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	669 struct maildir_index_sync_context *
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	670 maildir_sync_index_begin(struct maildir_mailbox *mbox)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	671 {
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	672 struct maildir_index_sync_context *sync_ctx;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	673
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	674 sync_ctx = i_new(struct maildir_index_sync_context, 1);
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	675 sync_ctx->mbox = mbox;
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	676
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	677 if (mail_index_sync_begin(mbox->ibox.index, &sync_ctx->sync_ctx,
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	678 &sync_ctx->view, (uint32_t)-1, (uoff_t)-1,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	679 FALSE, FALSE) <= 0) {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	680 mail_storage_set_index_error(&mbox->ibox);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	681 return NULL;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	682 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	683 return sync_ctx;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	684 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	685
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	686 void maildir_sync_index_abort(struct maildir_index_sync_context *sync_ctx)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	687 {
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	688 mail_index_sync_rollback(sync_ctx->sync_ctx);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	689 i_free(sync_ctx);
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	690 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	691
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	692 int maildir_sync_index_finish(struct maildir_index_sync_context *sync_ctx,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	693 int partial)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	694 {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	695 struct maildir_mailbox *mbox = sync_ctx->mbox;
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	696 struct mail_index_view *view = sync_ctx->view;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	697 struct maildir_uidlist_iter_ctx *iter;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	698 struct mail_index_transaction *trans;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	699 const struct mail_index_header *hdr;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	700 const struct mail_index_record *rec;
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	701 pool_t keyword_pool;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	702 uint32_t seq, uid;
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	703 enum maildir_uidlist_rec_flag uflags;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	704 const char *filename;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	705 enum mail_flags flags;
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	706 const char const keywords;
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	707 uint32_t uid_validity, next_uid;
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	708 int ret = 0, full_rescan = FALSE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	709
3322 49071cc19102 If UIDVALIDITY changes, don't invalidate the whole index. Just expunge all Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	710 trans = mail_index_transaction_begin(view, FALSE, TRUE);
49071cc19102 If UIDVALIDITY changes, don't invalidate the whole index. Just expunge all Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	711 sync_ctx->trans = trans;
49071cc19102 If UIDVALIDITY changes, don't invalidate the whole index. Just expunge all Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	712
2892 62d53b49110d Changed mail_index_get_header() to return the header as return value because Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	713 hdr = mail_index_get_header(view);
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	714 uid_validity = maildir_uidlist_get_uid_validity(mbox->uidlist);
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	715 if (uid_validity != hdr->uid_validity &&
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	716 uid_validity != 0 && hdr->uid_validity != 0) {
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	717 /* uidvalidity changed and mailbox isn't being initialized,
3322 49071cc19102 If UIDVALIDITY changes, don't invalidate the whole index. Just expunge all Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	718 reset mailbox so we can add all messages as new */
3280 2c72492dfd91 Created mbox_storage and maildir_storage. Timo Sirainen <tss@iki.fi> parents: 3279 diff changeset	719 mail_storage_set_critical(STORAGE(mbox->storage),
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	720 "Maildir %s sync: UIDVALIDITY changed (%u -> %u)",
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	721 mbox->path, hdr->uid_validity, uid_validity);
3322 49071cc19102 If UIDVALIDITY changes, don't invalidate the whole index. Just expunge all Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	722
3340 9c8220dfde7c Don't try to handle UIDVALIDITY changes nicely after all. It causes Timo Sirainen <tss@iki.fi> parents: 3322 diff changeset	723 mail_index_mark_corrupted(mbox->ibox.index);
9c8220dfde7c Don't try to handle UIDVALIDITY changes nicely after all. It causes Timo Sirainen <tss@iki.fi> parents: 3322 diff changeset	724 return -1;
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	725 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	726
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	727 keyword_pool = pool_alloconly_create("maildir keywords", 128);
73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	728
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	729 seq = 0;
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	730 iter = maildir_uidlist_iter_init(mbox->uidlist);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	731 while (maildir_uidlist_iter_next(iter, &uid, &uflags, &filename)) {
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	732 p_clear(keyword_pool);
73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	733 maildir_filename_get_flags(filename, keyword_pool,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	734 &flags, &keywords);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	735
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	736 if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RECENT) != 0 &&
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	737 (uflags & MAILDIR_UIDLIST_REC_FLAG_NEW_DIR) != 0 &&
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	738 (uflags & MAILDIR_UIDLIST_REC_FLAG_MOVED) == 0) {
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	739 /* mail is recent for next session as well */
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	740 flags \|= MAIL_RECENT;
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	741 }
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	742
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	743 __again:
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	744 seq++;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	745
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	746 if (seq > hdr->messages_count) {
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	747 if (uid < hdr->next_uid) {
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	748 /* most likely a race condition: we read the
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	749 maildir, then someone else expunged messages
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	750 and committed changes to index. so, this
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	751 message shouldn't actually exist. mark it
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	752 racy and check in next sync.
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	753
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	754 the difference between this and the later
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	755 check is that this one happens when messages
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	756 are expunged from the end */
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	757 if ((uflags &
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	758 MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	759 /* partial syncing */
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	760 continue;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	761 }
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	762 if ((uflags &
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	763 MAILDIR_UIDLIST_REC_FLAG_RACING) != 0) {
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	764 mail_storage_set_critical(
3280 2c72492dfd91 Created mbox_storage and maildir_storage. Timo Sirainen <tss@iki.fi> parents: 3279 diff changeset	765 STORAGE(mbox->storage),
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	766 "Maildir %s sync: "
3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	767 "UID < next_uid "
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	768 "(%u < %u, file = %s)",
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	769 mbox->path, uid, hdr->next_uid,
2067 3f97439ba59e Path was missing from Maildir sync-errors. Timo Sirainen <tss@iki.fi> parents: 2064 diff changeset	770 filename);
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	771 mail_index_mark_corrupted(
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	772 mbox->ibox.index);
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	773 ret = -1;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	774 break;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	775 }
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	776 mbox->dirty_cur_time = ioloop_time;
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	777 maildir_uidlist_add_flags(mbox->uidlist,
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	778 filename,
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	779 MAILDIR_UIDLIST_REC_FLAG_RACING);
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	780
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	781 seq--;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	782 continue;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	783 }
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	784
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	785 mail_index_append(trans, uid, &seq);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	786 mail_index_update_flags(trans, seq, MODIFY_REPLACE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	787 flags);
61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	788 // FIXME: set keywords
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	789 continue;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	790 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	791
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	792 if (mail_index_lookup(view, seq, &rec) < 0) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	793 ret = -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	794 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	795 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	796
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	797 if (rec->uid < uid) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	798 /* expunged */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	799 mail_index_expunge(trans, seq);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	800 goto __again;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	801 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	802
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	803 if (rec->uid > uid) {
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	804 /* most likely a race condition: we read the
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	805 maildir, then someone else expunged messages and
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	806 committed changes to index. so, this message
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	807 shouldn't actually exist. mark it racy and check
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	808 in next sync. */
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	809 if ((uflags &
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	810 MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	811 /* partial syncing */
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	812 seq--;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	813 continue;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	814 }
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	815 if ((uflags & MAILDIR_UIDLIST_REC_FLAG_RACING) != 0) {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	816 mail_storage_set_critical(
3280 2c72492dfd91 Created mbox_storage and maildir_storage. Timo Sirainen <tss@iki.fi> parents: 3279 diff changeset	817 STORAGE(mbox->storage),
2570 372d4b90c076 cleanup Timo Sirainen <tss@iki.fi> parents: 2533 diff changeset	818 "Maildir %s sync: "
372d4b90c076 cleanup Timo Sirainen <tss@iki.fi> parents: 2533 diff changeset	819 "UID inserted in the middle of mailbox "
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	820 "(%u > %u, file = %s)",
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	821 mbox->path, rec->uid, uid, filename);
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	822 mail_index_mark_corrupted(mbox->ibox.index);
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	823 ret = -1;
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	824 break;
2064 5cfdc99fab69 Don't use internal last_cur_mtime stamp - it's index-specific so always get Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	825 }
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	826
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	827 mbox->dirty_cur_time = ioloop_time;
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	828 maildir_uidlist_add_flags(mbox->uidlist, filename,
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2051 diff changeset	829 MAILDIR_UIDLIST_REC_FLAG_RACING);
2034 8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	830
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	831 seq--;
8078400fe561 Fix bogus "UID inserted in the middle of mailbox" errors Timo Sirainen <tss@iki.fi> parents: 2033 diff changeset	832 continue;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	833 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	834
2320 8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	835 if ((rec->flags & MAIL_RECENT) != 0) {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	836 index_mailbox_set_recent(&mbox->ibox, seq);
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	837 if (mbox->ibox.keep_recent) {
2320 8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	838 flags \|= MAIL_RECENT;
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	839 } else {
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	840 mail_index_update_flags(trans, seq,
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	841 MODIFY_REMOVE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	842 MAIL_RECENT);
2320 8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	843 }
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	844 }
8a6666a9ac98 Handle recent flags in index file correctly. Fixes recent flag losing when Timo Sirainen <tss@iki.fi> parents: 2272 diff changeset	845
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	846 if ((uflags & MAILDIR_UIDLIST_REC_FLAG_NONSYNCED) != 0) {
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	847 /* partial syncing */
3436 3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	848 if ((flags & MAIL_RECENT) != 0) {
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	849 /* we last saw this mail in new/, but it's
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	850 not there anymore. possibly expunged,
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	851 make sure. */
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	852 full_rescan = TRUE;
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	853 }
2228 d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	854 continue;
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	855 }
d19ba01fb5cd partial syncing fixes Timo Sirainen <tss@iki.fi> parents: 2173 diff changeset	856
1956 d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	857 if ((rec->flags & MAIL_INDEX_MAIL_FLAG_DIRTY) != 0) {
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	858 /* we haven't been able to update maildir with this
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	859 record's flag changes. don't sync them. */
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	860 continue;
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	861 }
d6941cd8afdc Added support for setting dirty flags for messages (TODO: undirty..) Timo Sirainen <tss@iki.fi> parents: 1955 diff changeset	862
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	863 if (((uint8_t)flags & ~MAIL_RECENT) !=
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	864 (rec->flags & (MAIL_FLAGS_MASK^MAIL_RECENT))) {
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	865 /* FIXME: this is wrong if there's pending changes in
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	866 transaction log already. it gets fixed in next sync
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	867 however.. */
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	868 mail_index_update_flags(trans, seq, MODIFY_REPLACE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	869 flags);
2038 df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	870 } else if ((flags & MAIL_RECENT) == 0 &&
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	871 (rec->flags & MAIL_RECENT) != 0) {
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	872 /* just remove recent flag */
df504dad3aec Recent flag fixes. Should work perfectly now with maildir. Timo Sirainen <tss@iki.fi> parents: 2037 diff changeset	873 mail_index_update_flags(trans, seq, MODIFY_REMOVE,
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	874 MAIL_RECENT);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	875 }
3016 61c8d205d887 Initial support for keywords. Syncing to mbox/maildir doesn't work yet. Timo Sirainen <tss@iki.fi> parents: 2936 diff changeset	876 // FIXME: update keywords
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	877 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	878 maildir_uidlist_iter_deinit(iter);
3085 73944f902165 don't allocate memory from data stack uselessly. Timo Sirainen <tss@iki.fi> parents: 3016 diff changeset	879 pool_unref(keyword_pool);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	880
2533 89911d2a9126 Don't break if saving messages to maildir without index files. Timo Sirainen <tss@iki.fi> parents: 2393 diff changeset	881 if (!partial) {
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	882 /* expunge the rest */
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	883 for (seq++; seq <= hdr->messages_count; seq++)
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	884 mail_index_expunge(trans, seq);
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	885 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	886
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	887 /* now, sync the index */
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	888 mbox->syncing_commit = TRUE;
3446 113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	889 if (maildir_sync_index_records(sync_ctx) < 0)
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	890 ret = -1;
113c888cdca1 Merge changes from multiple index sync records into one before actually Timo Sirainen <tss@iki.fi> parents: 3436 diff changeset	891 mbox->syncing_commit = FALSE;
2037 8763032d31bd Set dirty flags through transaction log, not directly. Some other flag Timo Sirainen <tss@iki.fi> parents: 2034 diff changeset	892
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	893 if (mbox->dirty_cur_time == 0 &&
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	894 mbox->last_cur_mtime != (time_t)hdr->sync_stamp) {
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	895 uint32_t sync_stamp = mbox->last_cur_mtime;
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	896
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	897 mail_index_update_header(trans,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	898 offsetof(struct mail_index_header, sync_stamp),
3322 49071cc19102 If UIDVALIDITY changes, don't invalidate the whole index. Just expunge all Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	899 &sync_stamp, sizeof(sync_stamp), TRUE);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	900 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	901
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	902 if (hdr->uid_validity == 0) {
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	903 /* get the initial uidvalidity */
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	904 if (maildir_uidlist_update(mbox->uidlist) < 0)
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	905 ret = -1;
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	906 uid_validity = maildir_uidlist_get_uid_validity(mbox->uidlist);
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	907 if (uid_validity == 0) {
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	908 uid_validity = ioloop_time;
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	909 maildir_uidlist_set_uid_validity(mbox->uidlist,
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	910 uid_validity);
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	911 }
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	912 } else if (uid_validity == 0) {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	913 maildir_uidlist_set_uid_validity(mbox->uidlist,
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	914 hdr->uid_validity);
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	915 }
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	916
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	917 if (uid_validity != hdr->uid_validity && uid_validity != 0) {
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	918 mail_index_update_header(trans,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	919 offsetof(struct mail_index_header, uid_validity),
3322 49071cc19102 If UIDVALIDITY changes, don't invalidate the whole index. Just expunge all Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	920 &uid_validity, sizeof(uid_validity), TRUE);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	921 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	922
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	923 next_uid = maildir_uidlist_get_next_uid(mbox->uidlist);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	924 if (next_uid != 0 && hdr->next_uid != next_uid) {
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	925 mail_index_update_header(trans,
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	926 offsetof(struct mail_index_header, next_uid),
3322 49071cc19102 If UIDVALIDITY changes, don't invalidate the whole index. Just expunge all Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	927 &next_uid, sizeof(next_uid), FALSE);
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	928 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	929
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	930 if (ret < 0) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	931 mail_index_transaction_rollback(trans);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	932 mail_index_sync_rollback(sync_ctx->sync_ctx);
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	933 } else {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	934 uint32_t seq;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	935 uoff_t offset;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	936
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	937 if (mail_index_transaction_commit(trans, &seq, &offset) < 0)
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	938 ret = -1;
2009 2e77ee652e2e When committing with no changes, don't try to sync them later. Timo Sirainen <tss@iki.fi> parents: 1988 diff changeset	939 else if (seq != 0) {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	940 mbox->ibox.commit_log_file_seq = seq;
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	941 mbox->ibox.commit_log_file_offset = offset;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	942 }
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	943 if (mail_index_sync_commit(sync_ctx->sync_ctx) < 0)
2393 d598c82a3c22 Renamed mail_index_sync_end() to _commit() and added _rollback(). Fixed mbox Timo Sirainen <tss@iki.fi> parents: 2373 diff changeset	944 ret = -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	945 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	946
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	947 if (ret == 0) {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	948 mbox->ibox.commit_log_file_seq = 0;
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	949 mbox->ibox.commit_log_file_offset = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	950 } else {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	951 mail_storage_set_index_error(&mbox->ibox);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	952 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	953
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	954 i_free(sync_ctx);
3436 3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	955 return ret < 0 ? -1 : (full_rescan ? 0 : 1);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	956 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	957
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	958 static int maildir_sync_context(struct maildir_sync_context *ctx, int forced)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	959 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	960 int ret, new_changed, cur_changed;
3436 3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	961 int full_rescan = FALSE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	962
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	963 if (!forced) {
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	964 if (maildir_sync_quick_check(ctx, &new_changed, &cur_changed) < 0)
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	965 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	966
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	967 if (!new_changed && !cur_changed)
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	968 return 0;
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	969 } else {
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	970 new_changed = cur_changed = TRUE;
e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	971 }
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1944 diff changeset	972
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	973 /*
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	974 Locking, locking, locking.. Wasn't maildir supposed to be lockless?
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	975
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	976 We can get here either as beginning a real maildir sync, or when
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	977 committing changes to maildir but a file was lost (maybe renamed).
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	978
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	979 So, we're going to need two locks. One for index and one for
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	980 uidlist. To avoid deadlocking do the index lock first.
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	981
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	982 uidlist is needed only for figuring out UIDs for newly seen files,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	983 so theoretically we wouldn't need to lock it unless there are new
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	984 files. It has a few problems though, assuming the index lock didn't
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	985 already protect it (eg. in-memory indexes):
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	986
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	987 1. Just because you see a new file which doesn't exist in uidlist
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	988 file, doesn't mean that the file really exists anymore, or that
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	989 your readdir() lists all new files. Meaning that this is possible:
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	990
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	991 A: opendir(), readdir() -> new file ...
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	992 -- new files are written to the maildir --
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	993 B: opendir(), readdir() -> new file, lock uidlist,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	994 readdir() -> another new file, rewrite uidlist, unlock
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	995 A: ... lock uidlist, readdir() -> nothing left, rewrite uidlist,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	996 unlock
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	997
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	998 The second time running A didn't see the two new files. To handle
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	999 this correctly, it must not remove the new unseen files from
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1000 uidlist. This is possible to do, but adds extra complexity.
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1001
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1002 2. If another process is rename()ing files while we are
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1003 readdir()ing, it's possible that readdir() never lists some files,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1004 causing Dovecot to assume they were expunged. In next sync they
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1005 would show up again, but client could have already been notified of
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1006 that and they would show up under new UIDs, so the damage is
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1007 already done.
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1008
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1009 Both of the problems can be avoided if we simply lock the uidlist
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1010 before syncing and keep it until sync is finished. Typically this
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1011 would happen in any case, as there is the index lock..
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1012
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1013 The second case is still a problem with external changes though,
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1014 because maildir doesn't require any kind of locking. Luckily this
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1015 problem rarely happens except under high amount of modifications.
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1016 */
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1017
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1018 if (!ctx->mbox->syncing_commit) {
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1019 ctx->index_sync_ctx = maildir_sync_index_begin(ctx->mbox);
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1020 if (ctx->index_sync_ctx == NULL)
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1021 return -1;
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1022 }
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1023
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1024 if ((ret = maildir_uidlist_lock(ctx->mbox->uidlist)) <= 0) {
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1025 /* failure / timeout. if forced is TRUE, we could still go
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1026 forward and check only for renamed files, but is it worth
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1027 the trouble? .. */
a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1028 return ret;
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	1029 }
2123 e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	1030
e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	1031 ctx->partial = !cur_changed;
e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	1032 ctx->uidlist_sync_ctx =
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1033 maildir_uidlist_sync_init(ctx->mbox->uidlist, ctx->partial);
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2067 diff changeset	1034
3435 6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	1035 while ((ret = maildir_scan_dir(ctx, TRUE)) > 0) {
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	1036 /* rename()d at least some files, which might have caused some
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	1037 other files to be missed. check again (see
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	1038 MAILDIR_RENAME_RESCAN_COUNT). */
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	1039 }
6f7ce690358e If we have rename()d more than 5 files from new/ to cur/, rescan the Timo Sirainen <tss@iki.fi> parents: 3417 diff changeset	1040 if (ret < 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1041 return -1;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1042 if (cur_changed) {
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1043 if (maildir_scan_dir(ctx, FALSE) < 0)
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1044 return -1;
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1045 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1046
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	1047 /* finish uidlist syncing, but keep it still locked */
2123 e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	1048 maildir_uidlist_sync_finish(ctx->uidlist_sync_ctx);
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1049 if (!ctx->mbox->syncing_commit) {
3436 3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1050 ret = maildir_sync_index_finish(ctx->index_sync_ctx,
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1051 ctx->partial);
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1052 if (ret < 0) {
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1053 ctx->index_sync_ctx = NULL;
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	1054 return -1;
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1055 }
3436 3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1056 if (ret == 0)
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1057 full_rescan = TRUE;
2818 a758a5b542bb Always protect maildir syncing with uidlist lock. Before we only tried to Timo Sirainen <tss@iki.fi> parents: 2816 diff changeset	1058 ctx->index_sync_ctx = NULL;
2173 9438951e243f Don't crash if we're syncing last commit to maildir, but some file was lost Timo Sirainen <tss@iki.fi> parents: 2140 diff changeset	1059 }
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	1060
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1061 ret = maildir_uidlist_sync_deinit(ctx->uidlist_sync_ctx);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1062 ctx->uidlist_sync_ctx = NULL;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1063
3436 3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1064 return ret < 0 ? -1 : (full_rescan ? 0 : 1);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1065 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1066
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1067 int maildir_storage_sync_force(struct maildir_mailbox *mbox)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1068 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1069 struct maildir_sync_context *ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1070 int ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1071
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1072 ctx = maildir_sync_context_new(mbox);
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	1073 ret = maildir_sync_context(ctx, TRUE);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1074 maildir_sync_deinit(ctx);
3436 3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1075 return ret < 0 ? -1 : 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1076 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1077
2322 aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	1078 struct mailbox_sync_context *
aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	1079 maildir_storage_sync_init(struct mailbox *box, enum mailbox_sync_flags flags)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1080 {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1081 struct maildir_mailbox mbox = (struct maildir_mailbox )box;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1082 struct maildir_sync_context *ctx;
2322 aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	1083 int ret = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1084
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1085 if ((flags & MAILBOX_SYNC_FLAG_FAST) == 0 \|\|
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1086 mbox->ibox.sync_last_check + MAILBOX_FULL_SYNC_INTERVAL <=
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1087 ioloop_time) {
b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1088 mbox->ibox.sync_last_check = ioloop_time;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1089
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3203 diff changeset	1090 ctx = maildir_sync_context_new(mbox);
2140 e2cd51b99359 "readonly sync" -> "forced sync" Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	1091 ret = maildir_sync_context(ctx, FALSE);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1092 maildir_sync_deinit(ctx);
3436 3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1093
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1094 if (ret == 0) {
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1095 /* lost some files from new/, see if thery're in cur/ */
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1096 ret = maildir_storage_sync_force(mbox);
3c51658d6846 We didn't notice if messages were deleted directly from new/. Timo Sirainen <tss@iki.fi> parents: 3435 diff changeset	1097 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1098 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1099
2322 aae574ed7f4c Broke mailbox_sync() into iterator. Timo Sirainen <tss@iki.fi> parents: 2320 diff changeset	1100 return index_mailbox_sync_init(box, flags, ret < 0);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1101 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-storage/index/maildir/maildir-sync.c @ 3446:113c888cdca1 HEAD