dovecot/original-hg/dovecot-1.2: src/lib-storage/index/maildir/maildir-uidlist.c annotate

annotate src/lib-storage/index/maildir/maildir-uidlist.c @ 9488:e47eb506eebd HEAD

maildir: Changed index/uidlist mismatch from assert-crashing to logging an error.

author	Timo Sirainen <tss@iki.fi>
date	Mon, 23 Nov 2009 12:19:24 -0500
parents	7956cc1086e1
children	c9cc50405ca4

rev	line source
8590 b9faf4db2a9f Updated copyright notices to include year 2009. Timo Sirainen <tss@iki.fi> parents: 8573 diff changeset	1 /* Copyright (c) 2003-2009 Dovecot authors, see the included COPYING file */
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	2
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	3 /*
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	4 Version 1 format has been used for most versions of Dovecot up to v1.0.x.
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	5 It's also compatible with Courier IMAP's courierimapuiddb file.
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	6 The format is:
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	7
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	8 header: 1 <uid validity> <next uid>
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	9 entry: <uid> <filename>
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	10
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	11 --
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	12
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	13 Version 2 format was written by a few development Dovecot versions, but
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	14 v1.0.x still parses the format. The format has <flags> field after <uid>.
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	15
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	16 --
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	17
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	18 Version 3 format is an extensible format used by Dovecot v1.1 and later.
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	19 It's also parsed by v1.0.2 (and later). The format is:
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	20
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	21 header: 3 [<key><value> ...]
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	22 entry: <uid> [<key><value> ...] :<filename>
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	23
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	24 See enum maildir_uidlist_*_ext_key for used keys.
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	25 */
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	26
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	27 #include "lib.h"
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	28 #include "array.h"
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	29 #include "hash.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	30 #include "istream.h"
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	31 #include "ostream.h"
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	32 #include "str.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	33 #include "file-dotlock.h"
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	34 #include "close-keep-errno.h"
4071 cd3d26cf124a Renamed safe-open.* to nfs-workarounds., safe_open() to nfs_safe_open() and Timo Sirainen <tss@iki.fi>* parents: 4070 diff changeset	35 #include "nfs-workarounds.h"
9166 60c8ea46c740 Use eacces_get_error() in more places for handling EACCES errors. Timo Sirainen <tss@iki.fi> parents: 9018 diff changeset	36 #include "eacces-error.h"
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	37 #include "maildir-storage.h"
5365 35f18edd5f17 If uidlist file doesn't exist but index files do, resync the maildir to Timo Sirainen <tss@iki.fi> parents: 5176 diff changeset	38 #include "maildir-sync.h"
5899 f29b93c0519c Moved maildir filename related functions to maildir-filename.c Timo Sirainen <tss@iki.fi> parents: 5892 diff changeset	39 #include "maildir-filename.h"
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	40 #include "maildir-uidlist.h"
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	41
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	42 #include <stdio.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	43 #include <stdlib.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	44 #include <sys/stat.h>
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	45
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	46 /* NFS: How many times to retry reading dovecot-uidlist file if ESTALE
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	47 error occurs in the middle of reading it */
4071 cd3d26cf124a Renamed safe-open.* to nfs-workarounds., safe_open() to nfs_safe_open() and Timo Sirainen <tss@iki.fi>* parents: 4070 diff changeset	48 #define UIDLIST_ESTALE_RETRY_COUNT NFS_ESTALE_RETRY_COUNT
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	49
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	50 #define UIDLIST_VERSION 3
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	51 #define UIDLIST_COMPRESS_PERCENTAGE 75
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	52
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	53 #define UIDLIST_IS_LOCKED(uidlist) \
3447 96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	54 ((uidlist)->lock_count > 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	55
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	56 struct maildir_uidlist_rec {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	57 uint32_t uid;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	58 uint32_t flags;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	59 char *filename;
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	60 unsigned char extensions; / <data>\0[<data>\0 ...]\0 */
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	61 };
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	62 ARRAY_DEFINE_TYPE(maildir_uidlist_rec_p, struct maildir_uidlist_rec *);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	63
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	64 struct maildir_uidlist {
3279 b698ae839a18 Moved mbox/maildir-specific variables from struct index_mailbox to Timo Sirainen <tss@iki.fi> parents: 3133 diff changeset	65 struct maildir_mailbox *mbox;
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	66 struct index_mailbox *ibox;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	67 char *path;
3447 96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	68
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	69 int fd;
db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	70 dev_t fd_dev;
db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	71 ino_t fd_ino;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	72 off_t fd_size;
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	73
3447 96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	74 unsigned int lock_count;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	75
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	76 struct dotlock_settings dotlock_settings;
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	77 struct dotlock *dotlock;
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	78
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	79 pool_t record_pool;
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	80 ARRAY_TYPE(maildir_uidlist_rec_p) records;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	81 struct hash_table *files;
5913 699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	82 unsigned int change_counter;
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	83
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	84 unsigned int version;
2565 e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	85 unsigned int uid_validity, next_uid, prev_read_uid, last_seen_uid;
9262 5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	86 unsigned int hdr_next_uid;
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	87 unsigned int read_records_count, read_line_count;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	88 uoff_t last_read_offset;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	89 string_t *hdr_extensions;
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	90
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	91 unsigned int recreate:1;
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	92 unsigned int initial_read:1;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	93 unsigned int initial_hdr_read:1;
8303 d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	94 unsigned int retry_rewind:1;
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	95 unsigned int locked_refresh:1;
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	96 unsigned int unsorted:1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	97 };
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	98
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	99 struct maildir_uidlist_sync_ctx {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	100 struct maildir_uidlist *uidlist;
5393 434047e53586 When saving/copying mails, we must not do any sorting based on the filename Timo Sirainen <tss@iki.fi> parents: 5388 diff changeset	101 enum maildir_uidlist_sync_flags sync_flags;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	102
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	103 pool_t record_pool;
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	104 ARRAY_TYPE(maildir_uidlist_rec_p) records;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	105 struct hash_table *files;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	106
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	107 unsigned int first_unwritten_pos, first_nouid_pos;
1973 f6d2dba3b78b and yet more syncing fixes.. Timo Sirainen <tss@iki.fi> parents: 1971 diff changeset	108 unsigned int new_files_count;
7271 4ac4e9ba1124 Make sure we don't use a stale first_unwritten_pos to optimize Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	109 unsigned int finish_change_counter;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	110
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	111 unsigned int partial:1;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	112 unsigned int finished:1;
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	113 unsigned int changed:1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	114 unsigned int failed:1;
6880 724c8f12eed2 maildir_storage_sync_force(): Don't wait on trying to lock uidlist. Just try Timo Sirainen <tss@iki.fi> parents: 6859 diff changeset	115 unsigned int locked:1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	116 };
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	117
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	118 struct maildir_uidlist_iter_ctx {
5913 699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	119 struct maildir_uidlist *uidlist;
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	120 struct maildir_uidlist_rec const next, const end;
5913 699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	121
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	122 unsigned int change_counter;
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	123 uint32_t prev_uid;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	124 };
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	125
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	126 static bool maildir_uidlist_iter_next_rec(struct maildir_uidlist_iter_ctx *ctx,
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	127 struct maildir_uidlist_rec **rec_r);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	128
2271 15d959d262a4 UIDs for appended messages can now be assigned all at once. Maildir now Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	129 static int maildir_uidlist_lock_timeout(struct maildir_uidlist *uidlist,
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	130 bool nonblock, bool refresh,
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	131 bool refresh_when_locked)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	132 {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	133 struct mailbox *box = &uidlist->ibox->box;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	134 const char control_dir, path;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	135 mode_t old_mask;
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	136 const enum dotlock_create_flags dotlock_flags =
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	137 nonblock ? DOTLOCK_CREATE_FLAG_NONBLOCK : 0;
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	138 int i, ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	139
3447 96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	140 if (uidlist->lock_count > 0) {
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	141 if (!uidlist->locked_refresh && refresh_when_locked) {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	142 if (maildir_uidlist_refresh(uidlist) < 0)
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	143 return -1;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	144 }
3447 96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	145 uidlist->lock_count++;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	146 return 1;
3447 96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	147 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	148
9296 22119217ab5d dbox: Don't crash when trying to update migrated maildir's uidlist. Timo Sirainen <tss@iki.fi> parents: 9280 diff changeset	149 index_storage_lock_notify_reset(uidlist->ibox);
8992 9f91cc822f34 maildir: Don't show "* OK" messages about lock waits too early. Timo Sirainen <tss@iki.fi> parents: 8943 diff changeset	150
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	151 control_dir = mailbox_list_get_path(box->storage->list, box->name,
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	152 MAILBOX_LIST_PATH_TYPE_CONTROL);
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	153 path = t_strconcat(control_dir, "/" MAILDIR_UIDLIST_NAME, NULL);
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	154
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	155 for (i = 0;; i++) {
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	156 old_mask = umask(0777 & ~box->file_create_mode);
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	157 ret = file_dotlock_create(&uidlist->dotlock_settings, path,
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	158 dotlock_flags, &uidlist->dotlock);
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	159 umask(old_mask);
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	160 if (ret > 0)
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	161 break;
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	162
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	163 /* failure */
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	164 if (ret == 0) {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	165 mail_storage_set_error(box->storage,
5613 f717fb4b31c0 Error handling rewrite. Timo Sirainen <tss@iki.fi> parents: 5584 diff changeset	166 MAIL_ERROR_TEMP, MAIL_ERRSTR_LOCK_TIMEOUT);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	167 return 0;
4152 e2edd333c473 Added MAILBOX_OPEN_KEEP_LOCKED flag to mailbox opening and implemented it Timo Sirainen <tss@iki.fi> parents: 4072 diff changeset	168 }
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	169 if (errno != ENOENT \|\| i == MAILDIR_DELETE_RETRY_COUNT \|\|
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	170 uidlist->mbox == NULL) {
9166 60c8ea46c740 Use eacces_get_error() in more places for handling EACCES errors. Timo Sirainen <tss@iki.fi> parents: 9018 diff changeset	171 if (errno == EACCES) {
60c8ea46c740 Use eacces_get_error() in more places for handling EACCES errors. Timo Sirainen <tss@iki.fi> parents: 9018 diff changeset	172 mail_storage_set_critical(box->storage, "%s",
60c8ea46c740 Use eacces_get_error() in more places for handling EACCES errors. Timo Sirainen <tss@iki.fi> parents: 9018 diff changeset	173 eacces_error_get_creating("file_dotlock_create", path));
60c8ea46c740 Use eacces_get_error() in more places for handling EACCES errors. Timo Sirainen <tss@iki.fi> parents: 9018 diff changeset	174 } else {
60c8ea46c740 Use eacces_get_error() in more places for handling EACCES errors. Timo Sirainen <tss@iki.fi> parents: 9018 diff changeset	175 mail_storage_set_critical(box->storage,
60c8ea46c740 Use eacces_get_error() in more places for handling EACCES errors. Timo Sirainen <tss@iki.fi> parents: 9018 diff changeset	176 "file_dotlock_create(%s) failed: %m",
60c8ea46c740 Use eacces_get_error() in more places for handling EACCES errors. Timo Sirainen <tss@iki.fi> parents: 9018 diff changeset	177 path);
60c8ea46c740 Use eacces_get_error() in more places for handling EACCES errors. Timo Sirainen <tss@iki.fi> parents: 9018 diff changeset	178 }
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	179 return -1;
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	180 }
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	181 /* the control dir doesn't exist. create it unless the whole
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	182 mailbox was just deleted. */
8772 dfcb8a6a4f5f maildir: We didn't properly handle missing control directory at startup. Timo Sirainen <tss@iki.fi> parents: 8703 diff changeset	183 if (!maildir_set_deleted(uidlist->mbox))
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	184 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	185 }
2121 0840edf34f37 Locking fixes. use less memory Timo Sirainen <tss@iki.fi> parents: 2094 diff changeset	186
3447 96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	187 uidlist->lock_count++;
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	188 uidlist->locked_refresh = FALSE;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	189
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	190 if (refresh) {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	191 /* make sure we have the latest changes before
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	192 changing anything */
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	193 if (maildir_uidlist_refresh(uidlist) < 0) {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	194 maildir_uidlist_unlock(uidlist);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	195 return -1;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	196 }
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	197 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	198 return 1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	199 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	200
2271 15d959d262a4 UIDs for appended messages can now be assigned all at once. Maildir now Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	201 int maildir_uidlist_lock(struct maildir_uidlist *uidlist)
15d959d262a4 UIDs for appended messages can now be assigned all at once. Maildir now Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	202 {
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	203 return maildir_uidlist_lock_timeout(uidlist, FALSE, TRUE, FALSE);
2271 15d959d262a4 UIDs for appended messages can now be assigned all at once. Maildir now Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	204 }
15d959d262a4 UIDs for appended messages can now be assigned all at once. Maildir now Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	205
15d959d262a4 UIDs for appended messages can now be assigned all at once. Maildir now Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	206 int maildir_uidlist_try_lock(struct maildir_uidlist *uidlist)
15d959d262a4 UIDs for appended messages can now be assigned all at once. Maildir now Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	207 {
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	208 return maildir_uidlist_lock_timeout(uidlist, TRUE, TRUE, FALSE);
2271 15d959d262a4 UIDs for appended messages can now be assigned all at once. Maildir now Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	209 }
15d959d262a4 UIDs for appended messages can now be assigned all at once. Maildir now Timo Sirainen <tss@iki.fi> parents: 2123 diff changeset	210
5176 db7b28a3ffbd If pop3_lock_session=yes, update maildir/dbox uidlist lock file every 10 Timo Sirainen <tss@iki.fi> parents: 5064 diff changeset	211 int maildir_uidlist_lock_touch(struct maildir_uidlist *uidlist)
db7b28a3ffbd If pop3_lock_session=yes, update maildir/dbox uidlist lock file every 10 Timo Sirainen <tss@iki.fi> parents: 5064 diff changeset	212 {
db7b28a3ffbd If pop3_lock_session=yes, update maildir/dbox uidlist lock file every 10 Timo Sirainen <tss@iki.fi> parents: 5064 diff changeset	213 i_assert(UIDLIST_IS_LOCKED(uidlist));
db7b28a3ffbd If pop3_lock_session=yes, update maildir/dbox uidlist lock file every 10 Timo Sirainen <tss@iki.fi> parents: 5064 diff changeset	214
db7b28a3ffbd If pop3_lock_session=yes, update maildir/dbox uidlist lock file every 10 Timo Sirainen <tss@iki.fi> parents: 5064 diff changeset	215 return file_dotlock_touch(uidlist->dotlock);
db7b28a3ffbd If pop3_lock_session=yes, update maildir/dbox uidlist lock file every 10 Timo Sirainen <tss@iki.fi> parents: 5064 diff changeset	216 }
db7b28a3ffbd If pop3_lock_session=yes, update maildir/dbox uidlist lock file every 10 Timo Sirainen <tss@iki.fi> parents: 5064 diff changeset	217
3863 55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool. Timo Sirainen <tss@iki.fi> parents: 3534 diff changeset	218 bool maildir_uidlist_is_locked(struct maildir_uidlist *uidlist)
3447 96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	219 {
96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	220 return UIDLIST_IS_LOCKED(uidlist);
96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	221 }
96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	222
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	223 void maildir_uidlist_unlock(struct maildir_uidlist *uidlist)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	224 {
3447 96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	225 i_assert(uidlist->lock_count > 0);
96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	226
96de02ea7482 Keywords are stored in maildir filename and maildir-keywords file Timo Sirainen <tss@iki.fi> parents: 3280 diff changeset	227 if (--uidlist->lock_count > 0)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	228 return;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	229
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	230 uidlist->locked_refresh = FALSE;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	231 (void)file_dotlock_delete(&uidlist->dotlock);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	232 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	233
8796 94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	234 static bool dotlock_callback(unsigned int secs_left, bool stale, void *context)
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	235 {
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	236 struct index_mailbox *ibox = context;
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	237
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	238 index_storage_lock_notify(ibox, stale ?
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	239 MAILBOX_LOCK_NOTIFY_MAILBOX_OVERRIDE :
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	240 MAILBOX_LOCK_NOTIFY_MAILBOX_ABORT,
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	241 secs_left);
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	242 return TRUE;
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	243 }
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	244
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	245 struct maildir_uidlist *
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	246 maildir_uidlist_init_readonly(struct index_mailbox *ibox)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	247 {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	248 struct mailbox *box = &ibox->box;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	249 struct maildir_uidlist *uidlist;
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	250 const char *control_dir;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	251
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	252 control_dir = mailbox_list_get_path(box->storage->list, box->name,
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	253 MAILBOX_LIST_PATH_TYPE_CONTROL);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	254
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	255 uidlist = i_new(struct maildir_uidlist, 1);
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	256 uidlist->fd = -1;
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	257 uidlist->ibox = ibox;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	258 uidlist->path = i_strconcat(control_dir, "/"MAILDIR_UIDLIST_NAME, NULL);
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	259 i_array_init(&uidlist->records, 128);
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	260 uidlist->files = hash_table_create(default_pool, default_pool, 4096,
f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	261 maildir_filename_base_hash,
f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	262 maildir_filename_base_cmp);
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	263 uidlist->next_uid = 1;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	264 uidlist->hdr_extensions = str_new(default_pool, 128);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	265
5938 805f2527a982 If use_io_notify dotlock setting is enabled, use I/O notify loop for waiting Timo Sirainen <tss@iki.fi> parents: 5933 diff changeset	266 uidlist->dotlock_settings.use_io_notify = TRUE;
4969 5c18ac362f65 More dotlock_use_excl uses. Timo Sirainen <tss@iki.fi> parents: 4960 diff changeset	267 uidlist->dotlock_settings.use_excl_lock =
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	268 (box->storage->flags & MAIL_STORAGE_FLAG_DOTLOCK_USE_EXCL) != 0;
6592 b6631550cf84 dotlocks: Flush NFS attribute cache only if nfs_flush setting is TRUE. Timo Sirainen <tss@iki.fi> parents: 6442 diff changeset	269 uidlist->dotlock_settings.nfs_flush =
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	270 (box->storage->flags &
6592 b6631550cf84 dotlocks: Flush NFS attribute cache only if nfs_flush setting is TRUE. Timo Sirainen <tss@iki.fi> parents: 6442 diff changeset	271 MAIL_STORAGE_FLAG_NFS_FLUSH_STORAGE) != 0;
7976 d4b7ef797828 Maildir: Uidlist stale timeout macro is now globally available. Timo Sirainen <tss@iki.fi> parents: 7940 diff changeset	272 uidlist->dotlock_settings.timeout =
d4b7ef797828 Maildir: Uidlist stale timeout macro is now globally available. Timo Sirainen <tss@iki.fi> parents: 7940 diff changeset	273 MAILDIR_UIDLIST_LOCK_STALE_TIMEOUT + 2;
d4b7ef797828 Maildir: Uidlist stale timeout macro is now globally available. Timo Sirainen <tss@iki.fi> parents: 7940 diff changeset	274 uidlist->dotlock_settings.stale_timeout =
d4b7ef797828 Maildir: Uidlist stale timeout macro is now globally available. Timo Sirainen <tss@iki.fi> parents: 7940 diff changeset	275 MAILDIR_UIDLIST_LOCK_STALE_TIMEOUT;
8796 94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	276 uidlist->dotlock_settings.callback = dotlock_callback;
94c815002d84 maildir: When locking uidlist, send OK notifies if it's taking a long time. Timo Sirainen <tss@iki.fi> parents: 8772 diff changeset	277 uidlist->dotlock_settings.context = ibox;
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	278
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	279 return uidlist;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	280 }
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	281
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	282 struct maildir_uidlist maildir_uidlist_init(struct maildir_mailbox mbox)
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	283 {
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	284 struct maildir_uidlist *uidlist;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	285
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	286 uidlist = maildir_uidlist_init_readonly(&mbox->ibox);
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	287 uidlist->mbox = mbox;
4808 93bc9770f938 Initial code for separation of mailbox accessing and directory layout Timo Sirainen <tss@iki.fi> parents: 4451 diff changeset	288 uidlist->dotlock_settings.temp_prefix = mbox->storage->temp_prefix;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	289 return uidlist;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	290 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	291
5915 171c3f89295c fd leak fix Timo Sirainen <tss@iki.fi> parents: 5913 diff changeset	292 static void maildir_uidlist_close(struct maildir_uidlist *uidlist)
171c3f89295c fd leak fix Timo Sirainen <tss@iki.fi> parents: 5913 diff changeset	293 {
8371 b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	294 struct mail_storage *storage = uidlist->ibox->box.storage;
b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	295
5915 171c3f89295c fd leak fix Timo Sirainen <tss@iki.fi> parents: 5913 diff changeset	296 if (uidlist->fd != -1) {
8371 b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	297 if (close(uidlist->fd) < 0) {
b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	298 mail_storage_set_critical(storage,
b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	299 "close(%s) failed: %m", uidlist->path);
b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	300 }
5915 171c3f89295c fd leak fix Timo Sirainen <tss@iki.fi> parents: 5913 diff changeset	301 uidlist->fd = -1;
171c3f89295c fd leak fix Timo Sirainen <tss@iki.fi> parents: 5913 diff changeset	302 uidlist->fd_ino = 0;
171c3f89295c fd leak fix Timo Sirainen <tss@iki.fi> parents: 5913 diff changeset	303 }
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	304 uidlist->last_read_offset = 0;
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	305 uidlist->read_line_count = 0;
5915 171c3f89295c fd leak fix Timo Sirainen <tss@iki.fi> parents: 5913 diff changeset	306 }
171c3f89295c fd leak fix Timo Sirainen <tss@iki.fi> parents: 5913 diff changeset	307
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	308 static void maildir_uidlist_reset(struct maildir_uidlist *uidlist)
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	309 {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	310 maildir_uidlist_close(uidlist);
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	311 uidlist->last_seen_uid = 0;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	312 uidlist->initial_hdr_read = FALSE;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	313
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	314 hash_table_clear(uidlist->files, FALSE);
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	315 array_clear(&uidlist->records);
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	316 }
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	317
6409 7f733ba453bf Changed _deinit() APIs to take * pointer and set it to NULL. Timo Sirainen <tss@iki.fi> parents: 6344 diff changeset	318 void maildir_uidlist_deinit(struct maildir_uidlist **_uidlist)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	319 {
6409 7f733ba453bf Changed _deinit() APIs to take * pointer and set it to NULL. Timo Sirainen <tss@iki.fi> parents: 6344 diff changeset	320 struct maildir_uidlist uidlist = _uidlist;
7f733ba453bf Changed _deinit() APIs to take * pointer and set it to NULL. Timo Sirainen <tss@iki.fi> parents: 6344 diff changeset	321
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	322 i_assert(!UIDLIST_IS_LOCKED(uidlist));
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	323
6409 7f733ba453bf Changed _deinit() APIs to take * pointer and set it to NULL. Timo Sirainen <tss@iki.fi> parents: 6344 diff changeset	324 *_uidlist = NULL;
6657 5da6da5cea95 Make sure uidlist gets written if we only added some extension fields to it. Timo Sirainen <tss@iki.fi> parents: 6599 diff changeset	325 maildir_uidlist_update(uidlist);
5915 171c3f89295c fd leak fix Timo Sirainen <tss@iki.fi> parents: 5913 diff changeset	326 maildir_uidlist_close(uidlist);
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	327
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	328 hash_table_destroy(&uidlist->files);
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	329 if (uidlist->record_pool != NULL)
6428 7cad076906eb pool_unref() now takes ** pointer. Timo Sirainen <tss@iki.fi> parents: 6417 diff changeset	330 pool_unref(&uidlist->record_pool);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	331
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	332 array_free(&uidlist->records);
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	333 str_free(&uidlist->hdr_extensions);
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	334 i_free(uidlist->path);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	335 i_free(uidlist);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	336 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	337
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	338 static int maildir_uid_cmp(const void p1, const void p2)
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	339 {
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	340 const struct maildir_uidlist_rec const rec1 = p1, const rec2 = p2;
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	341
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	342 return (rec1)->uid < (rec2)->uid ? -1 :
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	343 (rec1)->uid > (rec2)->uid ? 1 : 0;
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	344 }
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	345
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	346 static void ATTR_FORMAT(2, 3)
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	347 maildir_uidlist_set_corrupted(struct maildir_uidlist *uidlist,
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	348 const char *fmt, ...)
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	349 {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	350 struct mail_storage *storage = uidlist->ibox->box.storage;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	351 va_list args;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	352
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	353 va_start(args, fmt);
8303 d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	354 if (uidlist->retry_rewind) {
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	355 mail_storage_set_critical(storage,
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	356 "Broken or unexpectedly changed file %s "
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	357 "line %u: %s - re-reading from beginning",
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	358 uidlist->path, uidlist->read_line_count,
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	359 t_strdup_vprintf(fmt, args));
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	360 } else {
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	361 mail_storage_set_critical(storage, "Broken file %s line %u: %s",
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	362 uidlist->path, uidlist->read_line_count,
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	363 t_strdup_vprintf(fmt, args));
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	364 }
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	365 va_end(args);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	366 }
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	367
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	368 static void maildir_uidlist_update_hdr(struct maildir_uidlist *uidlist,
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	369 const struct stat *st)
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	370 {
7588 ffa9c0dc27a7 dbox: Crashfix when using old maildir files. Timo Sirainen <tss@iki.fi> parents: 7558 diff changeset	371 struct maildir_index_header *mhdr;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	372
7588 ffa9c0dc27a7 dbox: Crashfix when using old maildir files. Timo Sirainen <tss@iki.fi> parents: 7558 diff changeset	373 if (uidlist->mbox == NULL) {
ffa9c0dc27a7 dbox: Crashfix when using old maildir files. Timo Sirainen <tss@iki.fi> parents: 7558 diff changeset	374 /* dbox is using this */
ffa9c0dc27a7 dbox: Crashfix when using old maildir files. Timo Sirainen <tss@iki.fi> parents: 7558 diff changeset	375 return;
ffa9c0dc27a7 dbox: Crashfix when using old maildir files. Timo Sirainen <tss@iki.fi> parents: 7558 diff changeset	376 }
ffa9c0dc27a7 dbox: Crashfix when using old maildir files. Timo Sirainen <tss@iki.fi> parents: 7558 diff changeset	377
ffa9c0dc27a7 dbox: Crashfix when using old maildir files. Timo Sirainen <tss@iki.fi> parents: 7558 diff changeset	378 mhdr = &uidlist->mbox->maildir_hdr;
7940 87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	379 if (mhdr->uidlist_mtime == 0) {
87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	380 if (!uidlist->initial_read)
87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	381 (void)maildir_uidlist_refresh(uidlist);
87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	382 if (uidlist->version != UIDLIST_VERSION) {
87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	383 /* upgrading from older verson. don't update the
87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	384 uidlist times until it uses the new format */
87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	385 uidlist->recreate = TRUE;
87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	386 return;
87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	387 }
87214cbd824c maildir: Convert dovecot-uidlist files to new format immediately when Timo Sirainen <tss@iki.fi> parents: 7866 diff changeset	388 }
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	389 mhdr->uidlist_mtime = st->st_mtime;
7530 e29dafc501cc Compile fix Timo Sirainen <tss@iki.fi> parents: 7529 diff changeset	390 mhdr->uidlist_mtime_nsecs = ST_MTIME_NSEC(*st);
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	391 mhdr->uidlist_size = st->st_size;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	392 }
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	393
8304 ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	394 static unsigned int
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	395 maildir_uidlist_records_array_delete(struct maildir_uidlist *uidlist,
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	396 struct maildir_uidlist_rec *rec)
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	397 {
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	398 struct maildir_uidlist_rec const recs, const pos;
8304 ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	399 unsigned int idx, count;
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	400
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	401 recs = array_get(&uidlist->records, &count);
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	402 pos = bsearch(&rec, recs, count, sizeof(*recs), maildir_uid_cmp);
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	403 i_assert(pos != NULL);
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	404
8304 ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	405 idx = pos - recs;
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	406 array_delete(&uidlist->records, idx, 1);
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	407 return idx;
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	408 }
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	409
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	410 static bool
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	411 maildir_uidlist_read_extended(struct maildir_uidlist *uidlist,
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	412 const char **line_p,
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	413 struct maildir_uidlist_rec *rec)
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	414 {
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	415 const char start, line = *line_p;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	416 buffer_t *buf;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	417
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	418 buf = buffer_create_dynamic(pool_datastack_create(), 128);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	419 while (line != '\0' && line != ':') {
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	420 /* skip over an extension field */
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	421 start = line;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	422 while (line != ' ' && line != '\0') line++;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	423 buffer_append(buf, start, line - start);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	424 buffer_append_c(buf, '\0');
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	425 while (*line == ' ') line++;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	426 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	427
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	428 if (buf->used > 0) {
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	429 /* save the extensions */
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	430 buffer_append_c(buf, '\0');
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	431 rec->extensions = p_malloc(uidlist->record_pool, buf->used);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	432 memcpy(rec->extensions, buf->data, buf->used);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	433 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	434
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	435 if (*line == ':')
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	436 line++;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	437 if (*line == '\0')
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	438 return FALSE;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	439
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	440 *line_p = line;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	441 return TRUE;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	442 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	443
7556 ded075753258 Code cleanup Timo Sirainen <tss@iki.fi> parents: 7550 diff changeset	444 static bool maildir_uidlist_next(struct maildir_uidlist *uidlist,
ded075753258 Code cleanup Timo Sirainen <tss@iki.fi> parents: 7550 diff changeset	445 const char *line)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	446 {
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	447 struct maildir_uidlist_rec rec, old_rec, const recs;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	448 unsigned int count;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	449 uint32_t uid;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	450
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	451 uid = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	452 while (line >= '0' && line <= '9') {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	453 uid = uid10 + (line - '0');
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	454 line++;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	455 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	456
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	457 if (uid == 0 \|\| *line != ' ') {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	458 /* invalid file */
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	459 maildir_uidlist_set_corrupted(uidlist, "Invalid data: %s",
c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	460 line);
7556 ded075753258 Code cleanup Timo Sirainen <tss@iki.fi> parents: 7550 diff changeset	461 return FALSE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	462 }
2565 e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	463 if (uid <= uidlist->prev_read_uid) {
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	464 maildir_uidlist_set_corrupted(uidlist,
8303 d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	465 "UIDs not ordered (%u >= %u)",
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	466 uid, uidlist->prev_read_uid);
7556 ded075753258 Code cleanup Timo Sirainen <tss@iki.fi> parents: 7550 diff changeset	467 return FALSE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	468 }
9211 219ee9858156 maildir: Added some checks for UIDs reaching 2^31-1. Timo Sirainen <tss@iki.fi> parents: 9168 diff changeset	469 if (uid >= (uint32_t)-1) {
219ee9858156 maildir: Added some checks for UIDs reaching 2^31-1. Timo Sirainen <tss@iki.fi> parents: 9168 diff changeset	470 maildir_uidlist_set_corrupted(uidlist,
219ee9858156 maildir: Added some checks for UIDs reaching 2^31-1. Timo Sirainen <tss@iki.fi> parents: 9168 diff changeset	471 "UID too high (%u)", uid);
219ee9858156 maildir: Added some checks for UIDs reaching 2^31-1. Timo Sirainen <tss@iki.fi> parents: 9168 diff changeset	472 return FALSE;
219ee9858156 maildir: Added some checks for UIDs reaching 2^31-1. Timo Sirainen <tss@iki.fi> parents: 9168 diff changeset	473 }
2565 e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	474 uidlist->prev_read_uid = uid;
e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	475
e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	476 if (uid <= uidlist->last_seen_uid) {
e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	477 /* we already have this */
7556 ded075753258 Code cleanup Timo Sirainen <tss@iki.fi> parents: 7550 diff changeset	478 return TRUE;
2565 e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	479 }
e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	480 uidlist->last_seen_uid = uid;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	481
5958 5825961b9862 Make sure that uid < next_uid if uidlist version is 1. Timo Sirainen <tss@iki.fi> parents: 5957 diff changeset	482 if (uid >= uidlist->next_uid && uidlist->version == 1) {
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	483 maildir_uidlist_set_corrupted(uidlist,
c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	484 "UID larger than next_uid (%u >= %u)",
c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	485 uid, uidlist->next_uid);
7556 ded075753258 Code cleanup Timo Sirainen <tss@iki.fi> parents: 7550 diff changeset	486 return FALSE;
5958 5825961b9862 Make sure that uid < next_uid if uidlist version is 1. Timo Sirainen <tss@iki.fi> parents: 5957 diff changeset	487 }
5825961b9862 Make sure that uid < next_uid if uidlist version is 1. Timo Sirainen <tss@iki.fi> parents: 5957 diff changeset	488
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	489 rec = p_new(uidlist->record_pool, struct maildir_uidlist_rec, 1);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	490 rec->uid = uid;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	491 rec->flags = MAILDIR_UIDLIST_REC_FLAG_NONSYNCED;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	492
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	493 while (*line == ' ') line++;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	494
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	495 if (uidlist->version == UIDLIST_VERSION) {
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	496 /* read extended fields */
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	497 bool ret;
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	498
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7171 diff changeset	499 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	500 ret = maildir_uidlist_read_extended(uidlist, &line,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	501 rec);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7171 diff changeset	502 } T_END;
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	503 if (!ret) {
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	504 maildir_uidlist_set_corrupted(uidlist,
c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	505 "Invalid extended fields: %s", line);
7556 ded075753258 Code cleanup Timo Sirainen <tss@iki.fi> parents: 7550 diff changeset	506 return FALSE;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	507 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	508 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	509
7557 5532bc0eb7bc If dovecot-uidlist contains a filename with '/', log an error and recreate Timo Sirainen <tss@iki.fi> parents: 7556 diff changeset	510 if (strchr(line, '/') != NULL) {
5532bc0eb7bc If dovecot-uidlist contains a filename with '/', log an error and recreate Timo Sirainen <tss@iki.fi> parents: 7556 diff changeset	511 maildir_uidlist_set_corrupted(uidlist,
5532bc0eb7bc If dovecot-uidlist contains a filename with '/', log an error and recreate Timo Sirainen <tss@iki.fi> parents: 7556 diff changeset	512 "%s: Broken filename at line %u: %s",
5532bc0eb7bc If dovecot-uidlist contains a filename with '/', log an error and recreate Timo Sirainen <tss@iki.fi> parents: 7556 diff changeset	513 uidlist->path, uidlist->read_line_count, line);
7558 a1ba58f7d695 Minor cleanup to previous commit. Timo Sirainen <tss@iki.fi> parents: 7557 diff changeset	514 return FALSE;
7557 5532bc0eb7bc If dovecot-uidlist contains a filename with '/', log an error and recreate Timo Sirainen <tss@iki.fi> parents: 7556 diff changeset	515 }
5532bc0eb7bc If dovecot-uidlist contains a filename with '/', log an error and recreate Timo Sirainen <tss@iki.fi> parents: 7556 diff changeset	516
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	517 old_rec = hash_table_lookup(uidlist->files, line);
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	518 if (old_rec == NULL) {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	519 /* no conflicts */
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	520 } else if (old_rec->uid == uid) {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	521 /* most likely this is a record we saved ourself, but couldn't
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	522 update last_seen_uid because uidlist wasn't refreshed while
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	523 it was locked.
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	524
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	525 another possibility is a duplicate file record. currently
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	526 it would be a bug, but not that big of a deal. also perhaps
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	527 in future such duplicate lines could be used to update
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	528 extended fields. so just let it through anyway.
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	529
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	530 we'll waste a bit of memory here by allocating the record
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	531 twice, but that's not really a problem. */
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	532 rec->filename = old_rec->filename;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	533 hash_table_insert(uidlist->files, rec->filename, rec);
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	534 uidlist->unsorted = TRUE;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	535 return TRUE;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	536 } else {
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	537 /* This can happen if expunged file is moved back and the file
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	538 was appended to uidlist. */
7529 6ab2ccae0868 maildir uidlist: Include old and new UID in "Duplicate file entry" warning. Timo Sirainen <tss@iki.fi> parents: 7528 diff changeset	539 i_warning("%s: Duplicate file entry at line %u: "
6ab2ccae0868 maildir uidlist: Include old and new UID in "Duplicate file entry" warning. Timo Sirainen <tss@iki.fi> parents: 7528 diff changeset	540 "%s (uid %u -> %u)",
6ab2ccae0868 maildir uidlist: Include old and new UID in "Duplicate file entry" warning. Timo Sirainen <tss@iki.fi> parents: 7528 diff changeset	541 uidlist->path, uidlist->read_line_count, line,
6ab2ccae0868 maildir uidlist: Include old and new UID in "Duplicate file entry" warning. Timo Sirainen <tss@iki.fi> parents: 7528 diff changeset	542 old_rec->uid, uid);
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	543 /* Delete the old UID */
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	544 maildir_uidlist_records_array_delete(uidlist, old_rec);
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	545 /* Replace the old record with this new one */
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	546 old_rec = rec;
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	547 rec = old_rec;
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	548 uidlist->recreate = TRUE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	549 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	550
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	551 recs = array_get(&uidlist->records, &count);
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	552 if (count > 0 && recs[count-1]->uid > uid) {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	553 /* we most likely have some records in the array that we saved
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	554 ourself without refreshing uidlist */
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	555 uidlist->unsorted = TRUE;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	556 }
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	557
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	558 rec->filename = p_strdup(uidlist->record_pool, line);
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	559 hash_table_insert(uidlist->files, rec->filename, rec);
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	560 array_append(&uidlist->records, &rec, 1);
7556 ded075753258 Code cleanup Timo Sirainen <tss@iki.fi> parents: 7550 diff changeset	561 return TRUE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	562 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	563
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	564 static int maildir_uidlist_read_header(struct maildir_uidlist *uidlist,
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	565 struct istream *input)
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	566 {
9262 5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	567 unsigned int uid_validity = 0, next_uid = 0;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	568 string_t *ext_hdr;
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	569 const char *line;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	570 char key;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	571
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	572 line = i_stream_read_next_line(input);
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	573 if (line == NULL) {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	574 /* I/O error / empty file */
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	575 return input->stream_errno == 0 ? 0 : -1;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	576 }
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	577 uidlist->read_line_count = 1;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	578
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	579 if (line < '0' \|\| line > '9' \|\| line[1] != ' ') {
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	580 maildir_uidlist_set_corrupted(uidlist,
c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	581 "Corrupted header (invalid version number)");
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	582 return 0;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	583 }
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	584
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	585 uidlist->version = *line - '0';
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	586 line += 2;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	587
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	588 switch (uidlist->version) {
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	589 case 1:
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	590 if (sscanf(line, "%u %u", &uid_validity, &next_uid) != 2) {
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	591 maildir_uidlist_set_corrupted(uidlist,
c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	592 "Corrupted header (version 1)");
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	593 return 0;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	594 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	595 break;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	596 case UIDLIST_VERSION:
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	597 ext_hdr = uidlist->hdr_extensions;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	598 str_truncate(ext_hdr, 0);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7171 diff changeset	599 while (*line != '\0') T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	600 const char *value;
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	601
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	602 key = *line;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	603 value = ++line;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	604 while (line != '\0' && line != ' ') line++;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	605 value = t_strdup_until(value, line);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	606
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	607 switch (key) {
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	608 case MAILDIR_UIDLIST_HDR_EXT_UID_VALIDITY:
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	609 uid_validity = strtoul(value, NULL, 10);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	610 break;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	611 case MAILDIR_UIDLIST_HDR_EXT_NEXT_UID:
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	612 next_uid = strtoul(value, NULL, 10);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	613 break;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	614 default:
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	615 if (str_len(ext_hdr) > 0)
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	616 str_append_c(ext_hdr, ' ');
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	617 str_printfa(ext_hdr, "%c%s", key, value);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	618 break;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	619 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	620
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	621 while (*line == ' ') line++;
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7171 diff changeset	622 } T_END;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	623 break;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	624 default:
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	625 maildir_uidlist_set_corrupted(uidlist, "Unsupported version %u",
c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	626 uidlist->version);
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	627 return 0;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	628 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	629
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	630 if (uid_validity == 0 \|\| next_uid == 0) {
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	631 maildir_uidlist_set_corrupted(uidlist,
c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	632 "Broken header (uidvalidity = %u, next_uid=%u)",
c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	633 uid_validity, next_uid);
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	634 return 0;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	635 }
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	636
9262 5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	637 if (uid_validity == uidlist->uid_validity &&
5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	638 next_uid < uidlist->hdr_next_uid) {
5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	639 maildir_uidlist_set_corrupted(uidlist,
5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	640 "next_uid header was lowered (%u -> %u)",
5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	641 uidlist->hdr_next_uid, next_uid);
5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	642 return 0;
5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	643 }
5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	644
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	645 uidlist->uid_validity = uid_validity;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	646 uidlist->next_uid = next_uid;
9262 5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	647 uidlist->hdr_next_uid = next_uid;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	648 return 1;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	649 }
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	650
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	651 static void maildir_uidlist_records_sort_by_uid(struct maildir_uidlist *uidlist)
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	652 {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	653 struct maildir_uidlist_rec **recs;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	654 unsigned int count;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	655
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	656 recs = array_get_modifiable(&uidlist->records, &count);
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	657 qsort(recs, count, sizeof(*recs), maildir_uid_cmp);
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	658
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	659 uidlist->unsorted = FALSE;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	660 }
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	661
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	662 static int
4072 7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	663 maildir_uidlist_update_read(struct maildir_uidlist *uidlist,
7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	664 bool *retry_r, bool try_retry)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	665 {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	666 struct mail_storage *storage = uidlist->ibox->box.storage;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	667 const char *line;
9262 5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	668 uint32_t orig_next_uid, orig_uid_validity;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	669 struct istream *input;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	670 struct stat st;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	671 uoff_t last_read_offset;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	672 int fd, ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	673
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	674 *retry_r = FALSE;
db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	675
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	676 if (uidlist->fd == -1) {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	677 fd = nfs_safe_open(uidlist->path, O_RDWR);
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	678 if (fd == -1) {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	679 if (errno != ENOENT) {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	680 mail_storage_set_critical(storage,
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	681 "open(%s) failed: %m", uidlist->path);
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	682 return -1;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	683 }
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	684 return 0;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	685 }
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	686 last_read_offset = 0;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	687 } else {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	688 /* the file was updated */
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	689 fd = uidlist->fd;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	690 if (lseek(fd, 0, SEEK_SET) < 0) {
8703 e07901202df1 nfs+maildir: lseek() may also return ESTALE failure, handle it. Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	691 if (errno == ESTALE && try_retry) {
e07901202df1 nfs+maildir: lseek() may also return ESTALE failure, handle it. Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	692 *retry_r = TRUE;
e07901202df1 nfs+maildir: lseek() may also return ESTALE failure, handle it. Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	693 return -1;
e07901202df1 nfs+maildir: lseek() may also return ESTALE failure, handle it. Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	694 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	695 mail_storage_set_critical(storage,
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	696 "lseek(%s) failed: %m", uidlist->path);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	697 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	698 }
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	699 uidlist->fd = -1;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	700 uidlist->fd_ino = 0;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	701 last_read_offset = uidlist->last_read_offset;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	702 uidlist->last_read_offset = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	703 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	704
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	705 if (fstat(fd, &st) < 0) {
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	706 close_keep_errno(fd);
4072 7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	707 if (errno == ESTALE && try_retry) {
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	708 *retry_r = TRUE;
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	709 return -1;
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	710 }
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	711 mail_storage_set_critical(storage,
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	712 "fstat(%s) failed: %m", uidlist->path);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	713 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	714 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	715
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	716 if (uidlist->record_pool == NULL) {
4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	717 uidlist->record_pool =
4980 8e88ecc64563 debug: MEMPOOL_GROWING prefix in alloconly pool names means that when Timo Sirainen <tss@iki.fi> parents: 4969 diff changeset	718 pool_alloconly_create(MEMPOOL_GROWING
8e88ecc64563 debug: MEMPOOL_GROWING prefix in alloconly pool names means that when Timo Sirainen <tss@iki.fi> parents: 4969 diff changeset	719 "uidlist record_pool",
1931 35a947481130 fixes Timo Sirainen <tss@iki.fi> parents: 1915 diff changeset	720 nearest_power(st.st_size -
35a947481130 fixes Timo Sirainen <tss@iki.fi> parents: 1915 diff changeset	721 st.st_size/8));
35a947481130 fixes Timo Sirainen <tss@iki.fi> parents: 1915 diff changeset	722 }
35a947481130 fixes Timo Sirainen <tss@iki.fi> parents: 1915 diff changeset	723
6162 896cc473c1f0 Renamed i_stream_create_file() to i_stream_create_fd(). Timo Sirainen <tss@iki.fi> parents: 6161 diff changeset	724 input = i_stream_create_fd(fd, 4096, FALSE);
7169 b9d78196e9ba Optimization fix: We always read dovecot-uidlist from the beginning of the Timo Sirainen <tss@iki.fi> parents: 7166 diff changeset	725 i_stream_seek(input, last_read_offset);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	726
9262 5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	727 orig_uid_validity = uidlist->uid_validity;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	728 orig_next_uid = uidlist->next_uid;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	729 ret = input->v_offset != 0 ? 1 :
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	730 maildir_uidlist_read_header(uidlist, input);
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	731 if (ret > 0) {
2565 e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	732 uidlist->prev_read_uid = 0;
5913 699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	733 uidlist->change_counter++;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	734 uidlist->read_records_count = 0;
8303 d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	735 uidlist->retry_rewind = last_read_offset != 0 && try_retry;
2094 8f46cf3e9b5a small fixes Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	736
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	737 ret = 1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	738 while ((line = i_stream_read_next_line(input)) != NULL) {
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	739 uidlist->read_records_count++;
7166 c8f5955a4cf8 Log errors in dovecot-uidlist better. Timo Sirainen <tss@iki.fi> parents: 7140 diff changeset	740 uidlist->read_line_count++;
2565 e14c2a36a0a2 "UIDs not ordered in file" errors could have occured sometimes wrongly. Timo Sirainen <tss@iki.fi> parents: 2531 diff changeset	741 if (!maildir_uidlist_next(uidlist, line)) {
8303 d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	742 if (!uidlist->retry_rewind)
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	743 ret = 0;
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	744 else {
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	745 ret = -1;
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	746 *retry_r = TRUE;
d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	747 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	748 break;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	749 }
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	750 }
8303 d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	751 uidlist->retry_rewind = FALSE;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	752 if (input->stream_errno != 0)
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	753 ret = -1;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	754
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	755 if (uidlist->unsorted) {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	756 uidlist->recreate = TRUE;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	757 maildir_uidlist_records_sort_by_uid(uidlist);
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	758 }
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	759 if (uidlist->next_uid <= uidlist->prev_read_uid)
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	760 uidlist->next_uid = uidlist->prev_read_uid + 1;
9280 25c9df95fda6 maildir: Don't recreate dovecot-uidlist all the time. Timo Sirainen <tss@iki.fi> parents: 9274 diff changeset	761 if (ret > 0 && uidlist->uid_validity != orig_uid_validity &&
25c9df95fda6 maildir: Don't recreate dovecot-uidlist all the time. Timo Sirainen <tss@iki.fi> parents: 9274 diff changeset	762 orig_uid_validity != 0) {
9262 5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	763 uidlist->recreate = TRUE;
5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	764 } else if (ret > 0 && uidlist->next_uid < orig_next_uid) {
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	765 mail_storage_set_critical(storage,
9262 5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	766 "%s: next_uid was lowered (%u -> %u, hdr=%u)",
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	767 uidlist->path, orig_next_uid,
9262 5a4875e92829 maildir uidlist: Added better next_uid error tracking. Timo Sirainen <tss@iki.fi> parents: 9211 diff changeset	768 uidlist->next_uid, uidlist->hdr_next_uid);
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	769 uidlist->recreate = TRUE;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	770 uidlist->next_uid = orig_next_uid;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	771 }
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	772 }
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	773
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	774 if (ret == 0) {
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	775 /* file is broken */
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	776 (void)unlink(uidlist->path);
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	777 } else if (ret > 0) {
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	778 /* success */
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	779 uidlist->fd = fd;
db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	780 uidlist->fd_dev = st.st_dev;
db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	781 uidlist->fd_ino = st.st_ino;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	782 uidlist->fd_size = st.st_size;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	783 uidlist->last_read_offset = input->v_offset;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	784 maildir_uidlist_update_hdr(uidlist, &st);
8303 d3b37327936e Maildir: Re-read uidlist from the beginning before deciding it's corrupted. Timo Sirainen <tss@iki.fi> parents: 8260 diff changeset	785 } else if (!*retry_r) {
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	786 /* I/O error */
4072 7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	787 if (input->stream_errno == ESTALE && try_retry)
7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	788 *retry_r = TRUE;
7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	789 else {
7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	790 errno = input->stream_errno;
7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	791 mail_storage_set_critical(storage,
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	792 "read(%s) failed: %m", uidlist->path);
4072 7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	793 }
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	794 uidlist->last_read_offset = 0;
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	795 }
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	796
4070 71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of Timo Sirainen <tss@iki.fi> parents: 4023 diff changeset	797 i_stream_destroy(&input);
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	798 if (ret <= 0) {
8371 b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	799 if (close(fd) < 0) {
b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	800 mail_storage_set_critical(storage,
b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	801 "close(%s) failed: %m", uidlist->path);
b46cf0979768 Use mail_storage_set_critical() instead of i_error() directly. Timo Sirainen <tss@iki.fi> parents: 8304 diff changeset	802 }
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	803 }
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	804 return ret;
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	805 }
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	806
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	807 static int
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	808 maildir_uidlist_stat(struct maildir_uidlist uidlist, struct stat st_r)
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	809 {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	810 struct mail_storage *storage = uidlist->ibox->box.storage;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	811
6859 64e0f45e8dc3 Delayed NFS attribute cache flushing had some bug, so removed it for now. Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	812 if ((storage->flags & MAIL_STORAGE_FLAG_NFS_FLUSH_STORAGE) != 0) {
64e0f45e8dc3 Delayed NFS attribute cache flushing had some bug, so removed it for now. Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	813 nfs_flush_file_handle_cache(uidlist->path);
64e0f45e8dc3 Delayed NFS attribute cache flushing had some bug, so removed it for now. Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	814 nfs_flush_attr_cache_unlocked(uidlist->path);
64e0f45e8dc3 Delayed NFS attribute cache flushing had some bug, so removed it for now. Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	815 }
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	816 if (nfs_safe_stat(uidlist->path, st_r) < 0) {
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	817 if (errno != ENOENT) {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	818 mail_storage_set_critical(storage,
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	819 "stat(%s) failed: %m", uidlist->path);
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	820 return -1;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	821 }
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	822 return 0;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	823 }
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	824 return 1;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	825 }
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	826
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	827 static int
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	828 maildir_uidlist_has_changed(struct maildir_uidlist uidlist, bool recreated_r)
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	829 {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	830 struct mail_storage *storage = uidlist->ibox->box.storage;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	831 struct stat st;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	832 int ret;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	833
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	834 *recreated_r = FALSE;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	835
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	836 if ((ret = maildir_uidlist_stat(uidlist, &st)) <= 0)
9373 f1a29b13f7dc maildir: Don't force a resync when saving. Timo Sirainen <tss@iki.fi> parents: 9296 diff changeset	837 return ret < 0 ? -1 : 1;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	838
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	839 if (st.st_ino != uidlist->fd_ino \|\|
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	840 !CMP_DEV_T(st.st_dev, uidlist->fd_dev)) {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	841 /* file recreated */
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	842 *recreated_r = TRUE;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	843 return 1;
6797 a59deefc552f NFS attribute cache flushing fixes. nfs_flush_attr_cache() takes now Timo Sirainen <tss@iki.fi> parents: 6777 diff changeset	844 }
a59deefc552f NFS attribute cache flushing fixes. nfs_flush_attr_cache() takes now Timo Sirainen <tss@iki.fi> parents: 6777 diff changeset	845
6859 64e0f45e8dc3 Delayed NFS attribute cache flushing had some bug, so removed it for now. Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	846 if ((storage->flags & MAIL_STORAGE_FLAG_NFS_FLUSH_STORAGE) != 0) {
6798 e70686f19f67 Don't flush attribute cache unless we're locked and we really need to know Timo Sirainen <tss@iki.fi> parents: 6797 diff changeset	847 /* NFS: either the file hasn't been changed, or it has already
e70686f19f67 Don't flush attribute cache unless we're locked and we really need to know Timo Sirainen <tss@iki.fi> parents: 6797 diff changeset	848 been deleted and the inodes just happen to be the same.
e70686f19f67 Don't flush attribute cache unless we're locked and we really need to know Timo Sirainen <tss@iki.fi> parents: 6797 diff changeset	849 check if the fd is still valid. */
6829 dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	850 if (fstat(uidlist->fd, &st) < 0) {
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	851 if (errno == ESTALE) {
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	852 *recreated_r = TRUE;
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	853 return 1;
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	854 }
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	855 mail_storage_set_critical(storage,
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	856 "fstat(%s) failed: %m", uidlist->path);
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	857 return -1;
6798 e70686f19f67 Don't flush attribute cache unless we're locked and we really need to know Timo Sirainen <tss@iki.fi> parents: 6797 diff changeset	858 }
6797 a59deefc552f NFS attribute cache flushing fixes. nfs_flush_attr_cache() takes now Timo Sirainen <tss@iki.fi> parents: 6777 diff changeset	859 }
a59deefc552f NFS attribute cache flushing fixes. nfs_flush_attr_cache() takes now Timo Sirainen <tss@iki.fi> parents: 6777 diff changeset	860
a59deefc552f NFS attribute cache flushing fixes. nfs_flush_attr_cache() takes now Timo Sirainen <tss@iki.fi> parents: 6777 diff changeset	861 if (st.st_size != uidlist->fd_size) {
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	862 /* file modified but not recreated */
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	863 return 1;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	864 } else {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	865 /* unchanged */
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	866 return 0;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	867 }
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	868 }
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	869
6859 64e0f45e8dc3 Delayed NFS attribute cache flushing had some bug, so removed it for now. Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	870 int maildir_uidlist_refresh(struct maildir_uidlist *uidlist)
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	871 {
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	872 unsigned int i;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	873 bool retry, recreated;
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	874 int ret;
b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	875
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	876 if (uidlist->fd != -1) {
6859 64e0f45e8dc3 Delayed NFS attribute cache flushing had some bug, so removed it for now. Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	877 ret = maildir_uidlist_has_changed(uidlist, &recreated);
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	878 if (ret <= 0) {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	879 if (UIDLIST_IS_LOCKED(uidlist))
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	880 uidlist->locked_refresh = TRUE;
9373 f1a29b13f7dc maildir: Don't force a resync when saving. Timo Sirainen <tss@iki.fi> parents: 9296 diff changeset	881 return ret < 0 ? -1 : 1;
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	882 }
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	883
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	884 if (recreated)
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	885 maildir_uidlist_close(uidlist);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	886 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	887
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	888 for (i = 0; ; i++) {
4072 7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	889 ret = maildir_uidlist_update_read(uidlist, &retry,
7e6acdd8d18d If we reach the NFS_ESTALE_RETRY_COUNT, give an ESTALE error message as Timo Sirainen <tss@iki.fi> parents: 4071 diff changeset	890 i < UIDLIST_ESTALE_RETRY_COUNT);
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	891 if (!retry)
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	892 break;
6797 a59deefc552f NFS attribute cache flushing fixes. nfs_flush_attr_cache() takes now Timo Sirainen <tss@iki.fi> parents: 6777 diff changeset	893 /* ESTALE - try reopening and rereading */
a59deefc552f NFS attribute cache flushing fixes. nfs_flush_attr_cache() takes now Timo Sirainen <tss@iki.fi> parents: 6777 diff changeset	894 maildir_uidlist_close(uidlist);
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	895 }
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	896 if (ret >= 0) {
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	897 uidlist->initial_read = TRUE;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	898 uidlist->initial_hdr_read = TRUE;
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	899 if (UIDLIST_IS_LOCKED(uidlist))
6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	900 uidlist->locked_refresh = TRUE;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	901 }
4023 b19ccbaa3802 Try to handle ESTALE NFS errors the best way we can. Timo Sirainen <timo.sirainen@movial.fi> parents: 3981 diff changeset	902 return ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	903 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	904
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	905 int maildir_uidlist_refresh_fast_init(struct maildir_uidlist *uidlist)
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	906 {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	907 const struct maildir_index_header *mhdr = &uidlist->mbox->maildir_hdr;
8943 206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	908 struct mail_index *index = uidlist->mbox->ibox.index;
206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	909 struct mail_index_view *view;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	910 const struct mail_index_header *hdr;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	911 struct stat st;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	912 int ret;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	913
8943 206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	914 i_assert(UIDLIST_IS_LOCKED(uidlist));
206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	915
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	916 if (uidlist->fd != -1)
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	917 return maildir_uidlist_refresh(uidlist);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	918
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	919 if ((ret = maildir_uidlist_stat(uidlist, &st)) < 0)
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	920 return ret;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	921
9274 693fa1d6160a maildir: Don't access uninitialized variables. Timo Sirainen <tss@iki.fi> parents: 9262 diff changeset	922 if (ret > 0 && st.st_size == mhdr->uidlist_size &&
7866 e4884450ecf4 Compiler warning fix Timo Sirainen <tss@iki.fi> parents: 7611 diff changeset	923 st.st_mtime == (time_t)mhdr->uidlist_mtime &&
8943 206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	924 ST_NTIMES_EQUAL(ST_MTIME_NSEC(st), mhdr->uidlist_mtime_nsecs) &&
206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	925 (!mail_index_is_in_memory(index) \|\| st.st_mtime < ioloop_time-1)) {
206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	926 /* index is up-to-date. look up the uidvalidity and next-uid
206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	927 from it. we'll need to create a new view temporarily to
206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	928 make sure we get the latest values. */
206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	929 view = mail_index_view_open(index);
206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	930 hdr = mail_index_get_header(view);
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	931 uidlist->uid_validity = hdr->uid_validity;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	932 uidlist->next_uid = hdr->next_uid;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	933 uidlist->initial_hdr_read = TRUE;
8943 206af25d21bc maildir: When saving messages, in some race conditions we might have written a duplicate UID. Timo Sirainen <tss@iki.fi> parents: 8876 diff changeset	934 mail_index_view_close(&view);
9428 ebdba086e3b1 maildir: Saving a mail after fast syncing caused slow sync the next time. Timo Sirainen <tss@iki.fi> parents: 9373 diff changeset	935
ebdba086e3b1 maildir: Saving a mail after fast syncing caused slow sync the next time. Timo Sirainen <tss@iki.fi> parents: 9373 diff changeset	936 if (UIDLIST_IS_LOCKED(uidlist))
ebdba086e3b1 maildir: Saving a mail after fast syncing caused slow sync the next time. Timo Sirainen <tss@iki.fi> parents: 9373 diff changeset	937 uidlist->locked_refresh = TRUE;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	938 return 1;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	939 } else {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	940 return maildir_uidlist_refresh(uidlist);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	941 }
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	942 }
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	943
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	944 static int
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	945 maildir_uidlist_lookup_rec(struct maildir_uidlist *uidlist, uint32_t uid,
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	946 unsigned int *idx_r,
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	947 struct maildir_uidlist_rec **rec_r)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	948 {
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	949 struct maildir_uidlist_rec const recs;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	950 unsigned int idx, left_idx, right_idx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	951
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	952 if (!uidlist->initial_read) {
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	953 /* first time we need to read uidlist */
6859 64e0f45e8dc3 Delayed NFS attribute cache flushing had some bug, so removed it for now. Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	954 if (maildir_uidlist_refresh(uidlist) < 0)
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	955 return -1;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	956 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	957
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	958 idx = left_idx = 0;
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	959 recs = array_get(&uidlist->records, &right_idx);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	960 while (left_idx < right_idx) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	961 idx = (left_idx + right_idx) / 2;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	962
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	963 if (recs[idx]->uid < uid)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	964 left_idx = idx+1;
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	965 else if (recs[idx]->uid > uid)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	966 right_idx = idx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	967 else {
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	968 *idx_r = idx;
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	969 *rec_r = recs[idx];
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	970 return 1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	971 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	972 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	973
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	974 if (idx > 0) idx--;
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	975 *idx_r = idx;
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	976 return 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	977 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	978
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	979 int maildir_uidlist_lookup(struct maildir_uidlist *uidlist, uint32_t uid,
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	980 enum maildir_uidlist_rec_flag *flags_r,
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	981 const char **fname_r)
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	982 {
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	983 int ret;
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	984
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	985 ret = maildir_uidlist_lookup_nosync(uidlist, uid, flags_r, fname_r);
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	986 if (ret <= 0) {
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	987 if (ret < 0)
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	988 return -1;
7550 d920ec986c34 Maildir: Moved uidlist refreshing to maildir_uidlist_lookup() where it Timo Sirainen <tss@iki.fi> parents: 7538 diff changeset	989 if (uidlist->fd != -1 \|\| uidlist->mbox == NULL) {
d920ec986c34 Maildir: Moved uidlist refreshing to maildir_uidlist_lookup() where it Timo Sirainen <tss@iki.fi> parents: 7538 diff changeset	990 /* refresh uidlist and check again in case it was added
d920ec986c34 Maildir: Moved uidlist refreshing to maildir_uidlist_lookup() where it Timo Sirainen <tss@iki.fi> parents: 7538 diff changeset	991 after the last mailbox sync */
d920ec986c34 Maildir: Moved uidlist refreshing to maildir_uidlist_lookup() where it Timo Sirainen <tss@iki.fi> parents: 7538 diff changeset	992 if (maildir_uidlist_refresh(uidlist) < 0)
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	993 return -1;
7550 d920ec986c34 Maildir: Moved uidlist refreshing to maildir_uidlist_lookup() where it Timo Sirainen <tss@iki.fi> parents: 7538 diff changeset	994 } else {
d920ec986c34 Maildir: Moved uidlist refreshing to maildir_uidlist_lookup() where it Timo Sirainen <tss@iki.fi> parents: 7538 diff changeset	995 /* the uidlist doesn't exist. */
d920ec986c34 Maildir: Moved uidlist refreshing to maildir_uidlist_lookup() where it Timo Sirainen <tss@iki.fi> parents: 7538 diff changeset	996 if (maildir_storage_sync_force(uidlist->mbox, uid) < 0)
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	997 return -1;
7550 d920ec986c34 Maildir: Moved uidlist refreshing to maildir_uidlist_lookup() where it Timo Sirainen <tss@iki.fi> parents: 7538 diff changeset	998 }
5365 35f18edd5f17 If uidlist file doesn't exist but index files do, resync the maildir to Timo Sirainen <tss@iki.fi> parents: 5176 diff changeset	999
35f18edd5f17 If uidlist file doesn't exist but index files do, resync the maildir to Timo Sirainen <tss@iki.fi> parents: 5176 diff changeset	1000 /* try again */
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1001 ret = maildir_uidlist_lookup_nosync(uidlist, uid,
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1002 flags_r, fname_r);
5365 35f18edd5f17 If uidlist file doesn't exist but index files do, resync the maildir to Timo Sirainen <tss@iki.fi> parents: 5176 diff changeset	1003 }
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1004
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1005 return ret;
6933 ad16a4582033 If file is lost, don't try to look for it until stack overflows. Timo Sirainen <tss@iki.fi> parents: 6886 diff changeset	1006 }
ad16a4582033 If file is lost, don't try to look for it until stack overflows. Timo Sirainen <tss@iki.fi> parents: 6886 diff changeset	1007
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1008 int maildir_uidlist_lookup_nosync(struct maildir_uidlist *uidlist, uint32_t uid,
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1009 enum maildir_uidlist_rec_flag *flags_r,
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1010 const char **fname_r)
6933 ad16a4582033 If file is lost, don't try to look for it until stack overflows. Timo Sirainen <tss@iki.fi> parents: 6886 diff changeset	1011 {
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1012 struct maildir_uidlist_rec *rec;
6933 ad16a4582033 If file is lost, don't try to look for it until stack overflows. Timo Sirainen <tss@iki.fi> parents: 6886 diff changeset	1013 unsigned int idx;
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1014 int ret;
6933 ad16a4582033 If file is lost, don't try to look for it until stack overflows. Timo Sirainen <tss@iki.fi> parents: 6886 diff changeset	1015
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1016 if ((ret = maildir_uidlist_lookup_rec(uidlist, uid, &idx, &rec)) <= 0)
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1017 return ret;
6933 ad16a4582033 If file is lost, don't try to look for it until stack overflows. Timo Sirainen <tss@iki.fi> parents: 6886 diff changeset	1018
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1019 *flags_r = rec->flags;
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1020 *fname_r = rec->filename;
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1021 return 1;
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1022 }
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1023
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1024 const char *
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1025 maildir_uidlist_lookup_ext(struct maildir_uidlist *uidlist, uint32_t uid,
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1026 enum maildir_uidlist_rec_ext_key key)
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1027 {
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1028 struct maildir_uidlist_rec *rec;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1029 unsigned int idx;
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1030 const unsigned char *p;
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1031 int ret;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1032
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1033 ret = maildir_uidlist_lookup_rec(uidlist, uid, &idx, &rec);
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1034 if (ret <= 0 \|\| rec->extensions == NULL)
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1035 return NULL;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1036
8844 94dbe538cdb7 Removed some unnecessary code. Timo Sirainen <tss@iki.fi> parents: 8796 diff changeset	1037 p = rec->extensions;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1038 while (*p != '\0') {
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1039 /* <key><value>\0 */
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1040 if (*p == (char)key)
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1041 return (const char *)p + 1;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1042
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1043 p += strlen((const char *)p) + 1;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1044 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1045 return NULL;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1046 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1047
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	1048 uint32_t maildir_uidlist_get_uid_validity(struct maildir_uidlist *uidlist)
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	1049 {
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	1050 return uidlist->uid_validity;
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	1051 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	1052
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	1053 uint32_t maildir_uidlist_get_next_uid(struct maildir_uidlist *uidlist)
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	1054 {
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1055 return !uidlist->initial_hdr_read ? 0 : uidlist->next_uid;
2050 ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	1056 }
ee1095ccfd23 Index header changes now go through transaction log. Removed the kludgy Timo Sirainen <tss@iki.fi> parents: 2039 diff changeset	1057
5932 6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1058 void maildir_uidlist_set_uid_validity(struct maildir_uidlist *uidlist,
6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1059 uint32_t uid_validity)
6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1060 {
6344 e45ba2950722 Added asserts against uid_validity==0. Timo Sirainen <tss@iki.fi> parents: 6162 diff changeset	1061 i_assert(uid_validity != 0);
e45ba2950722 Added asserts against uid_validity==0. Timo Sirainen <tss@iki.fi> parents: 6162 diff changeset	1062
5932 6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1063 uidlist->uid_validity = uid_validity;
6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1064 }
6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1065
6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1066 void maildir_uidlist_set_next_uid(struct maildir_uidlist *uidlist,
6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1067 uint32_t next_uid, bool force)
6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1068 {
9211 219ee9858156 maildir: Added some checks for UIDs reaching 2^31-1. Timo Sirainen <tss@iki.fi> parents: 9168 diff changeset	1069 if (uidlist->next_uid < next_uid \|\| force) {
219ee9858156 maildir: Added some checks for UIDs reaching 2^31-1. Timo Sirainen <tss@iki.fi> parents: 9168 diff changeset	1070 i_assert(next_uid != 0);
5932 6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1071 uidlist->next_uid = next_uid;
9211 219ee9858156 maildir: Added some checks for UIDs reaching 2^31-1. Timo Sirainen <tss@iki.fi> parents: 9168 diff changeset	1072 }
5932 6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1073 }
6ac8d6c93d34 Handle next_uid changes better. Timo Sirainen <tss@iki.fi> parents: 5923 diff changeset	1074
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	1075 static void
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	1076 maildir_uidlist_set_ext_real(struct maildir_uidlist *uidlist, uint32_t uid,
5950 49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1077 enum maildir_uidlist_rec_ext_key key,
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1078 const char *value)
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1079 {
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1080 struct maildir_uidlist_rec *rec;
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1081 unsigned int idx;
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1082 const unsigned char *p;
5950 49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1083 buffer_t *buf;
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1084 unsigned int len;
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1085 int ret;
5950 49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1086
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1087 ret = maildir_uidlist_lookup_rec(uidlist, uid, &idx, &rec);
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1088 if (ret <= 0) {
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1089 if (ret < 0)
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1090 return;
0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1091
7013 7966e025ac0f maildir_uidlist_set_ext() assert-crashed when trying to set it to expunged Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1092 /* maybe it's a new message */
7966e025ac0f maildir_uidlist_set_ext() assert-crashed when trying to set it to expunged Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1093 if (maildir_uidlist_refresh(uidlist) < 0)
7966e025ac0f maildir_uidlist_set_ext() assert-crashed when trying to set it to expunged Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1094 return;
9017 0aa17f3e4a6d Maildir: Handle uidlist errors better. Timo Sirainen <tss@iki.fi> parents: 9016 diff changeset	1095 if (maildir_uidlist_lookup_rec(uidlist, uid, &idx, &rec) <= 0) {
7013 7966e025ac0f maildir_uidlist_set_ext() assert-crashed when trying to set it to expunged Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1096 /* message is already expunged, ignore */
7966e025ac0f maildir_uidlist_set_ext() assert-crashed when trying to set it to expunged Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1097 return;
7966e025ac0f maildir_uidlist_set_ext() assert-crashed when trying to set it to expunged Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1098 }
7966e025ac0f maildir_uidlist_set_ext() assert-crashed when trying to set it to expunged Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1099 }
5950 49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1100
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1101 buf = buffer_create_dynamic(pool_datastack_create(), 128);
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1102
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1103 /* copy existing extensions, except for the one we're updating */
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1104 if (rec->extensions != NULL) {
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1105 p = rec->extensions;
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1106 while (*p != '\0') {
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1107 /* <key><value>\0 */
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1108 len = strlen((const char *)p) + 1;
5950 49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1109 if (*p != (char)key)
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1110 buffer_append(buf, p, len);
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1111 p += len;
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1112 }
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1113 }
7538 abdf9e06a956 Maildir: If dovecot-uidlist has wrong cached virtual size, remove it when we Timo Sirainen <tss@iki.fi> parents: 7530 diff changeset	1114 if (value != NULL) {
abdf9e06a956 Maildir: If dovecot-uidlist has wrong cached virtual size, remove it when we Timo Sirainen <tss@iki.fi> parents: 7530 diff changeset	1115 buffer_append_c(buf, key);
abdf9e06a956 Maildir: If dovecot-uidlist has wrong cached virtual size, remove it when we Timo Sirainen <tss@iki.fi> parents: 7530 diff changeset	1116 buffer_append(buf, value, strlen(value) + 1);
abdf9e06a956 Maildir: If dovecot-uidlist has wrong cached virtual size, remove it when we Timo Sirainen <tss@iki.fi> parents: 7530 diff changeset	1117 }
5950 49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1118 buffer_append_c(buf, '\0');
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1119
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1120 rec->extensions = p_malloc(uidlist->record_pool, buf->used);
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1121 memcpy(rec->extensions, buf->data, buf->used);
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1122
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1123 if (rec->uid != (uint32_t)-1) {
6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1124 /* message already exists in uidlist, need to recreate it */
6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1125 uidlist->recreate = TRUE;
6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1126 }
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	1127 }
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	1128
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	1129 void maildir_uidlist_set_ext(struct maildir_uidlist *uidlist, uint32_t uid,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	1130 enum maildir_uidlist_rec_ext_key key,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	1131 const char *value)
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	1132 {
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7171 diff changeset	1133 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6933 diff changeset	1134 maildir_uidlist_set_ext_real(uidlist, uid, key, value);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7171 diff changeset	1135 } T_END;
5950 49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1136 }
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1137
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1138 static int maildir_uidlist_write_fd(struct maildir_uidlist *uidlist, int fd,
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1139 const char *path, unsigned int first_idx,
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1140 uoff_t *file_size_r)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1141 {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1142 struct mail_storage *storage = uidlist->ibox->box.storage;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1143 struct maildir_uidlist_iter_ctx *iter;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1144 struct ostream *output;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1145 struct maildir_uidlist_rec *rec;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1146 string_t *str;
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1147 const unsigned char *p;
2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1148 unsigned int len;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1149 int ret;
4152 e2edd333c473 Added MAILBOX_OPEN_KEEP_LOCKED flag to mailbox opening and implemented it Timo Sirainen <tss@iki.fi> parents: 4072 diff changeset	1150
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1151 i_assert(fd != -1);
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1152
6161 c62f7ee79446 Split o_stream_create_file() to _create_fd() and _create_fd_file(). Timo Sirainen <tss@iki.fi> parents: 6142 diff changeset	1153 output = o_stream_create_fd_file(fd, (uoff_t)-1, FALSE);
7125 7ee2bee70d6a Buffer writes to dovecot-uidlist instead of doing lots of small write()s. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1154 o_stream_cork(output);
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1155 str = t_str_new(512);
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	1156
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1157 if (output->offset == 0) {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1158 i_assert(first_idx == 0);
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1159 uidlist->version = UIDLIST_VERSION;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1160
6344 e45ba2950722 Added asserts against uid_validity==0. Timo Sirainen <tss@iki.fi> parents: 6162 diff changeset	1161 i_assert(uidlist->uid_validity != 0);
e45ba2950722 Added asserts against uid_validity==0. Timo Sirainen <tss@iki.fi> parents: 6162 diff changeset	1162 i_assert(uidlist->next_uid > 0);
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1163 str_printfa(str, "%u V%u N%u", uidlist->version,
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1164 uidlist->uid_validity, uidlist->next_uid);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1165 if (str_len(uidlist->hdr_extensions) > 0) {
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1166 str_append_c(str, ' ');
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1167 str_append_str(str, uidlist->hdr_extensions);
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1168 }
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1169 str_append_c(str, '\n');
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1170 o_stream_send(output, str_data(str), str_len(str));
3472 db29cc6754d5 Store new/ directory's timestamp in sync_size header in index (kludgy..). Timo Sirainen <tss@iki.fi> parents: 3447 diff changeset	1171 }
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1172
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1173 iter = maildir_uidlist_iter_init(uidlist);
7271 4ac4e9ba1124 Make sure we don't use a stale first_unwritten_pos to optimize Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	1174 i_assert(first_idx <= array_count(&uidlist->records));
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1175 iter->next += first_idx;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1176
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1177 while (maildir_uidlist_iter_next_rec(iter, &rec)) {
8014 cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1178 uidlist->read_records_count++;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1179 str_truncate(str, 0);
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1180 str_printfa(str, "%u", rec->uid);
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1181 if (rec->extensions != NULL) {
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1182 for (p = rec->extensions; *p != '\0'; ) {
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1183 len = strlen((const char *)p);
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1184 str_append_c(str, ' ');
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1185 str_append_n(str, p, len);
2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1186 p += len + 1;
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1187 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1188 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1189 str_printfa(str, " :%s\n", rec->filename);
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1190 o_stream_send(output, str_data(str), str_len(str));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1191 }
6409 7f733ba453bf Changed _deinit() APIs to take * pointer and set it to NULL. Timo Sirainen <tss@iki.fi> parents: 6344 diff changeset	1192 maildir_uidlist_iter_deinit(&iter);
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1193 o_stream_flush(output);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1194
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1195 ret = output->stream_errno == 0 ? 0 : -1;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1196
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1197 *file_size_r = output->offset;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1198 o_stream_unref(&output);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1199
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1200 if (ret < 0) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1201 mail_storage_set_critical(storage,
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1202 "o_stream_send(%s) failed: %m", path);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1203 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1204 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1205
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1206 if (!uidlist->ibox->fsync_disable) {
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1207 if (fdatasync(fd) < 0) {
5064 db3f662588a4 Made one more fsync() optional with fsync_disable. Timo Sirainen <tss@iki.fi> parents: 5033 diff changeset	1208 mail_storage_set_critical(storage,
6738 c3e6d7d96f1a Error message fix: s/fsync/fdatasync/ Timo Sirainen <tss@iki.fi> parents: 6657 diff changeset	1209 "fdatasync(%s) failed: %m", path);
5064 db3f662588a4 Made one more fsync() optional with fsync_disable. Timo Sirainen <tss@iki.fi> parents: 5033 diff changeset	1210 return -1;
db3f662588a4 Made one more fsync() optional with fsync_disable. Timo Sirainen <tss@iki.fi> parents: 5033 diff changeset	1211 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1212 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1213 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1214 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1215
9429 7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1216 static void
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1217 maildir_uidlist_records_drop_expunges(struct maildir_uidlist *uidlist)
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1218 {
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1219 struct mail_index_view *view;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1220 struct maildir_uidlist_rec const recs;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1221 ARRAY_TYPE(maildir_uidlist_rec_p) new_records;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1222 const struct mail_index_header *hdr;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1223 const struct mail_index_record *rec;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1224 unsigned int i, count;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1225 uint32_t seq;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1226
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1227 mail_index_refresh(uidlist->mbox->ibox.index);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1228 view = mail_index_view_open(uidlist->mbox->ibox.index);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1229 count = array_count(&uidlist->records);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1230 hdr = mail_index_get_header(view);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1231 if (count * UIDLIST_COMPRESS_PERCENTAGE / 100 <= hdr->messages_count) {
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1232 /* too much trouble to be worth it */
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1233 mail_index_view_close(&view);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1234 return;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1235 }
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1236
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1237 i_array_init(&new_records, hdr->messages_count + 64);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1238 recs = array_get(&uidlist->records, &count);
9488 e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1239 for (i = 0, seq = 1; i < count && seq <= hdr->messages_count; ) {
9429 7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1240 rec = mail_index_lookup(view, seq);
9488 e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1241 if (recs[i]->uid < rec->uid) {
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1242 /* expunged entry */
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1243 i++;
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1244 } else if (recs[i]->uid > rec->uid) {
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1245 i_error("%s: uid=%u exists in index, "
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1246 "but not in uidlist",
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1247 uidlist->path, rec->uid);
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1248 seq++;
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1249 } else {
9429 7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1250 array_append(&new_records, &recs[i], 1);
9488 e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1251 seq++; i++;
9429 7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1252 }
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1253 }
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1254
9488 e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1255 for (; seq <= hdr->messages_count; seq++) {
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1256 rec = mail_index_lookup(view, seq);
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1257 i_error("%s: uid=%u exists in index tail, but not in uidlist",
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1258 uidlist->path, rec->uid);
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1259 }
e47eb506eebd maildir: Changed index/uidlist mismatch from assert-crashing to logging an error. Timo Sirainen <tss@iki.fi> parents: 9429 diff changeset	1260
9429 7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1261 /* drop messages expunged at the end of index */
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1262 while (i < count && recs[i]->uid < hdr->next_uid)
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1263 i++;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1264 /* view might not be completely up-to-date, so preserve any
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1265 messages left */
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1266 for (; i < count; i++)
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1267 array_append(&new_records, &recs[i], 1);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1268
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1269 array_free(&uidlist->records);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1270 uidlist->records = new_records;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1271
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1272 mail_index_view_close(&view);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1273 }
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1274
5919 cc439e4e99cb cleanup Timo Sirainen <tss@iki.fi> parents: 5918 diff changeset	1275 static int maildir_uidlist_recreate(struct maildir_uidlist *uidlist)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1276 {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1277 struct mailbox *box = &uidlist->ibox->box;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1278 const char control_dir, temp_path;
5891 db959f0d94e3 Leave uidlist fd open so we can check if uidlist has been rewritten from Timo Sirainen <tss@iki.fi> parents: 5890 diff changeset	1279 struct stat st;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1280 mode_t old_mask;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1281 uoff_t file_size;
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1282 int i, fd, ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1283
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1284 i_assert(uidlist->initial_read);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1285
9429 7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1286 maildir_uidlist_records_drop_expunges(uidlist);
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1287
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1288 control_dir = mailbox_list_get_path(box->storage->list, box->name,
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1289 MAILBOX_LIST_PATH_TYPE_CONTROL);
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1290 temp_path = t_strconcat(control_dir,
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1291 "/" MAILDIR_UIDLIST_NAME ".tmp", NULL);
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1292
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1293 for (i = 0;; i++) {
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1294 old_mask = umask(0777 & ~box->file_create_mode);
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1295 fd = open(temp_path, O_RDWR \| O_CREAT \| O_TRUNC, 0777);
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1296 umask(old_mask);
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1297 if (fd != -1)
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1298 break;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1299
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1300 if (errno != ENOENT \|\| i == MAILDIR_DELETE_RETRY_COUNT \|\|
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1301 uidlist->mbox == NULL) {
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1302 mail_storage_set_critical(box->storage,
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1303 "open(%s, O_CREAT) failed: %m", temp_path);
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1304 return -1;
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1305 }
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1306 /* the control dir doesn't exist. create it unless the whole
186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1307 mailbox was just deleted. */
8772 dfcb8a6a4f5f maildir: We didn't properly handle missing control directory at startup. Timo Sirainen <tss@iki.fi> parents: 8703 diff changeset	1308 if (!maildir_set_deleted(uidlist->mbox))
6777 186b164a9579 Delay creating directories until we really need them. Timo Sirainen <tss@iki.fi> parents: 6738 diff changeset	1309 return -1;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1310 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1311
9168 2bbf175bb6d3 Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9166 diff changeset	1312 if (box->file_create_gid != (gid_t)-1 &&
2bbf175bb6d3 Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9166 diff changeset	1313 fchown(fd, (uid_t)-1, box->file_create_gid) < 0) {
2bbf175bb6d3 Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9166 diff changeset	1314 if (errno == EPERM) {
2bbf175bb6d3 Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9166 diff changeset	1315 mail_storage_set_critical(box->storage, "%s",
2bbf175bb6d3 Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9166 diff changeset	1316 eperm_error_get_chgrp("fchown", temp_path,
2bbf175bb6d3 Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9166 diff changeset	1317 box->file_create_gid,
2bbf175bb6d3 Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9166 diff changeset	1318 box->file_create_gid_origin));
2bbf175bb6d3 Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9166 diff changeset	1319 } else {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1320 mail_storage_set_critical(box->storage,
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1321 "fchown(%s) failed: %m", temp_path);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1322 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1323 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1324
8014 cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1325 uidlist->read_records_count = 0;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1326 ret = maildir_uidlist_write_fd(uidlist, fd, temp_path, 0, &file_size);
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1327 if (ret == 0) {
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1328 if (rename(temp_path, uidlist->path) < 0) {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1329 mail_storage_set_critical(box->storage,
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1330 "rename(%s, %s) failed: %m",
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1331 temp_path, uidlist->path);
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1332 ret = -1;
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1333 }
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1334 }
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1335
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1336 if (ret < 0) {
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1337 if (unlink(temp_path) < 0) {
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1338 mail_storage_set_critical(box->storage,
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1339 "unlink(%s) failed: %m", temp_path);
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1340 }
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1341 } else if (fstat(fd, &st) < 0) {
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1342 mail_storage_set_critical(box->storage,
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1343 "fstat(%s) failed: %m", temp_path);
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1344 ret = -1;
7135 9fef306a0d95 Changed assert to first check if dotlock had been overridden, and if so log Timo Sirainen <tss@iki.fi> parents: 7125 diff changeset	1345 } else if (file_size != (uoff_t)st.st_size) {
9fef306a0d95 Changed assert to first check if dotlock had been overridden, and if so log Timo Sirainen <tss@iki.fi> parents: 7125 diff changeset	1346 i_assert(!file_dotlock_is_locked(uidlist->dotlock));
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1347 mail_storage_set_critical(box->storage,
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1348 "Maildir uidlist dotlock overridden: %s",
7135 9fef306a0d95 Changed assert to first check if dotlock had been overridden, and if so log Timo Sirainen <tss@iki.fi> parents: 7125 diff changeset	1349 uidlist->path);
9fef306a0d95 Changed assert to first check if dotlock had been overridden, and if so log Timo Sirainen <tss@iki.fi> parents: 7125 diff changeset	1350 ret = -1;
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1351 } else {
5956 fc75386a57ab fd leak fix Timo Sirainen <tss@iki.fi> parents: 5950 diff changeset	1352 maildir_uidlist_close(uidlist);
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1353 uidlist->fd = fd;
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1354 uidlist->fd_dev = st.st_dev;
525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1355 uidlist->fd_ino = st.st_ino;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1356 uidlist->fd_size = st.st_size;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1357 uidlist->last_read_offset = st.st_size;
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1358 uidlist->recreate = FALSE;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1359 maildir_uidlist_update_hdr(uidlist, &st);
5917 525b449313a7 Don't use uidlist lock to update uidlist. Have a separate real dotlock for Timo Sirainen <tss@iki.fi> parents: 5915 diff changeset	1360 }
7528 4a0e2927a232 Maildir error handling fixes if appending to uidlist failed. Timo Sirainen <tss@iki.fi> parents: 7526 diff changeset	1361 if (ret < 0)
4a0e2927a232 Maildir error handling fixes if appending to uidlist failed. Timo Sirainen <tss@iki.fi> parents: 7526 diff changeset	1362 (void)close(fd);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1363 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1364 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1365
5950 49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1366 int maildir_uidlist_update(struct maildir_uidlist *uidlist)
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1367 {
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1368 int ret;
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1369
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1370 if (!uidlist->recreate)
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1371 return 0;
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1372
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1373 if (maildir_uidlist_lock(uidlist) <= 0)
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1374 return -1;
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1375 ret = maildir_uidlist_recreate(uidlist);
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1376 maildir_uidlist_unlock(uidlist);
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1377 return ret;
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1378 }
49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1379
8014 cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1380 static bool maildir_uidlist_want_compress(struct maildir_uidlist_sync_ctx *ctx)
cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1381 {
9429 7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1382 struct mail_index_view *view = ctx->uidlist->mbox->ibox.view;
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1383 unsigned int min_rewrite_count, messages_count;
8014 cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1384
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1385 if (!ctx->uidlist->locked_refresh)
8015 73b4fe95a6dd Maildir: Check more correctly when we should update/recreate uidlist. Timo Sirainen <tss@iki.fi> parents: 8014 diff changeset	1386 return FALSE;
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1387 if (ctx->uidlist->recreate)
6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1388 return TRUE;
8015 73b4fe95a6dd Maildir: Check more correctly when we should update/recreate uidlist. Timo Sirainen <tss@iki.fi> parents: 8014 diff changeset	1389
8014 cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1390 min_rewrite_count =
cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1391 (ctx->uidlist->read_records_count + ctx->new_files_count) *
cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1392 UIDLIST_COMPRESS_PERCENTAGE / 100;
9429 7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1393 messages_count = I_MIN(mail_index_view_get_messages_count(view),
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1394 array_count(&ctx->uidlist->records));
7956cc1086e1 maildir_very_dirty_syncs=yes: Make sure uidlist shrinks eventually. Timo Sirainen <tss@iki.fi> parents: 9428 diff changeset	1395 return min_rewrite_count >= messages_count;
8014 cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1396 }
cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1397
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1398 static bool maildir_uidlist_want_recreate(struct maildir_uidlist_sync_ctx *ctx)
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1399 {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1400 struct maildir_uidlist *uidlist = ctx->uidlist;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1401
9428 ebdba086e3b1 maildir: Saving a mail after fast syncing caused slow sync the next time. Timo Sirainen <tss@iki.fi> parents: 9373 diff changeset	1402 if (!uidlist->locked_refresh \|\| !uidlist->initial_read)
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1403 return FALSE;
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1404
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1405 if (ctx->finish_change_counter != uidlist->change_counter)
8015 73b4fe95a6dd Maildir: Check more correctly when we should update/recreate uidlist. Timo Sirainen <tss@iki.fi> parents: 8014 diff changeset	1406 return TRUE;
73b4fe95a6dd Maildir: Check more correctly when we should update/recreate uidlist. Timo Sirainen <tss@iki.fi> parents: 8014 diff changeset	1407 if (uidlist->fd == -1 \|\| uidlist->version != UIDLIST_VERSION)
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1408 return TRUE;
8014 cb2650c20a58 Recreate dovecot-uidlist if updating it would shrink the file more than 25%. Timo Sirainen <tss@iki.fi> parents: 7996 diff changeset	1409 return maildir_uidlist_want_compress(ctx);
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1410 }
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1411
5950 49aebb3028f7 Added support for updating uidlist record extensions. Timo Sirainen <tss@iki.fi> parents: 5944 diff changeset	1412 static int maildir_uidlist_sync_update(struct maildir_uidlist_sync_ctx *ctx)
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1413 {
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1414 struct maildir_uidlist *uidlist = ctx->uidlist;
8260 110afc84fbb1 Maildir/dbox: Try harder to assign unique UIDVALIDITY values to mailboxes. Timo Sirainen <tss@iki.fi> parents: 8015 diff changeset	1415 struct mail_storage *storage = uidlist->ibox->box.storage;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1416 struct stat st;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1417 uoff_t file_size;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1418
5959 5513182aed67 One more UIDVALIDITY=0 fix Timo Sirainen <tss@iki.fi> parents: 5958 diff changeset	1419 if (uidlist->uid_validity == 0) {
5513182aed67 One more UIDVALIDITY=0 fix Timo Sirainen <tss@iki.fi> parents: 5958 diff changeset	1420 /* saving a message to a newly created maildir. */
5513182aed67 One more UIDVALIDITY=0 fix Timo Sirainen <tss@iki.fi> parents: 5958 diff changeset	1421 const struct mail_index_header *hdr;
5513182aed67 One more UIDVALIDITY=0 fix Timo Sirainen <tss@iki.fi> parents: 5958 diff changeset	1422
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1423 hdr = mail_index_get_header(uidlist->ibox->view);
5959 5513182aed67 One more UIDVALIDITY=0 fix Timo Sirainen <tss@iki.fi> parents: 5958 diff changeset	1424 uidlist->uid_validity = hdr->uid_validity != 0 ?
8260 110afc84fbb1 Maildir/dbox: Try harder to assign unique UIDVALIDITY values to mailboxes. Timo Sirainen <tss@iki.fi> parents: 8015 diff changeset	1425 hdr->uid_validity :
110afc84fbb1 Maildir/dbox: Try harder to assign unique UIDVALIDITY values to mailboxes. Timo Sirainen <tss@iki.fi> parents: 8015 diff changeset	1426 maildir_get_uidvalidity_next(storage);
5959 5513182aed67 One more UIDVALIDITY=0 fix Timo Sirainen <tss@iki.fi> parents: 5958 diff changeset	1427 }
5513182aed67 One more UIDVALIDITY=0 fix Timo Sirainen <tss@iki.fi> parents: 5958 diff changeset	1428
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1429
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1430 if (maildir_uidlist_want_recreate(ctx))
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1431 return maildir_uidlist_recreate(uidlist);
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1432
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1433 if (uidlist->fd == -1) {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1434 /* NOREFRESH flag used. we're just appending some messages. */
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1435 i_assert(uidlist->initial_hdr_read);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1436
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1437 uidlist->fd = nfs_safe_open(uidlist->path, O_RDWR);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1438 if (uidlist->fd == -1) {
8260 110afc84fbb1 Maildir/dbox: Try harder to assign unique UIDVALIDITY values to mailboxes. Timo Sirainen <tss@iki.fi> parents: 8015 diff changeset	1439 mail_storage_set_critical(storage,
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1440 "open(%s) failed: %m", uidlist->path);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1441 return -1;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1442 }
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1443 }
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1444
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1445 i_assert(ctx->first_unwritten_pos != (unsigned int)-1);
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1446
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1447 if (lseek(uidlist->fd, 0, SEEK_END) < 0) {
8260 110afc84fbb1 Maildir/dbox: Try harder to assign unique UIDVALIDITY values to mailboxes. Timo Sirainen <tss@iki.fi> parents: 8015 diff changeset	1448 mail_storage_set_critical(storage,
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1449 "lseek(%s) failed: %m", uidlist->path);
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1450 return -1;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1451 }
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1452
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1453 if (maildir_uidlist_write_fd(uidlist, uidlist->fd, uidlist->path,
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1454 ctx->first_unwritten_pos, &file_size) < 0)
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1455 return -1;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1456
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1457 if (fstat(uidlist->fd, &st) < 0) {
8260 110afc84fbb1 Maildir/dbox: Try harder to assign unique UIDVALIDITY values to mailboxes. Timo Sirainen <tss@iki.fi> parents: 8015 diff changeset	1458 mail_storage_set_critical(storage,
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1459 "fstat(%s) failed: %m", uidlist->path);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1460 return -1;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1461 }
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1462 if ((uoff_t)st.st_size != file_size) {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1463 i_warning("%s: file size changed unexpectedly after write",
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1464 uidlist->path);
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1465 } else if (uidlist->locked_refresh) {
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1466 uidlist->fd_size = st.st_size;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1467 uidlist->last_read_offset = st.st_size;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1468 maildir_uidlist_update_hdr(uidlist, &st);
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1469 }
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1470 return 0;
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1471 }
fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1472
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1473 static void maildir_uidlist_mark_all(struct maildir_uidlist *uidlist,
3863 55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool. Timo Sirainen <tss@iki.fi> parents: 3534 diff changeset	1474 bool nonsynced)
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1475 {
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1476 struct maildir_uidlist_rec **recs;
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1477 unsigned int i, count;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1478
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1479 recs = array_get_modifiable(&uidlist->records, &count);
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1480 if (nonsynced) {
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1481 for (i = 0; i < count; i++)
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1482 recs[i]->flags \|= MAILDIR_UIDLIST_REC_FLAG_NONSYNCED;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1483 } else {
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1484 for (i = 0; i < count; i++)
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1485 recs[i]->flags &= ~MAILDIR_UIDLIST_REC_FLAG_NONSYNCED;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1486 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1487 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1488
8554 c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1489 static int maildir_uidlist_sync_lock(struct maildir_uidlist *uidlist,
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1490 enum maildir_uidlist_sync_flags sync_flags,
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1491 bool *locked_r)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1492 {
8554 c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1493 bool nonblock, refresh;
3530 e9695ec7925b Recursive maildir uidlist syncs caused assert crashes. Also did some Timo Sirainen <tss@iki.fi> parents: 3472 diff changeset	1494 int ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1495
8554 c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1496 *locked_r = FALSE;
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1497
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1498 if ((sync_flags & MAILDIR_UIDLIST_SYNC_NOLOCK) != 0) {
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1499 if (maildir_uidlist_refresh(uidlist) < 0)
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1500 return -1;
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1501 return 1;
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1502 }
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1503
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1504 nonblock = (sync_flags & MAILDIR_UIDLIST_SYNC_TRYLOCK) != 0;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1505 refresh = (sync_flags & MAILDIR_UIDLIST_SYNC_NOREFRESH) == 0;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1506
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1507 ret = maildir_uidlist_lock_timeout(uidlist, nonblock, refresh, refresh);
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1508 if (ret <= 0) {
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1509 if (ret < 0 \|\| !nonblock)
6880 724c8f12eed2 maildir_storage_sync_force(): Don't wait on trying to lock uidlist. Just try Timo Sirainen <tss@iki.fi> parents: 6859 diff changeset	1510 return ret;
6886 82420ed23379 If doing a lockless forced sync, track the UID we're trying to find. If we Timo Sirainen <tss@iki.fi> parents: 6884 diff changeset	1511
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1512 /* couldn't lock it */
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1513 if ((sync_flags & MAILDIR_UIDLIST_SYNC_FORCE) == 0)
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1514 return 0;
c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1515 /* forcing the sync anyway */
6886 82420ed23379 If doing a lockless forced sync, track the UID we're trying to find. If we Timo Sirainen <tss@iki.fi> parents: 6884 diff changeset	1516 if (maildir_uidlist_refresh(uidlist) < 0)
82420ed23379 If doing a lockless forced sync, track the UID we're trying to find. If we Timo Sirainen <tss@iki.fi> parents: 6884 diff changeset	1517 return -1;
7515 c14a2b0a3126 Keep track of dovecot-uidlist mtime+size in index file. If it's up-to-date Timo Sirainen <tss@iki.fi> parents: 7271 diff changeset	1518 } else {
8554 c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1519 *locked_r = TRUE;
6880 724c8f12eed2 maildir_storage_sync_force(): Don't wait on trying to lock uidlist. Just try Timo Sirainen <tss@iki.fi> parents: 6859 diff changeset	1520 }
8554 c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1521 return 1;
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1522 }
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1523
8876 88aeadb32151 Added maildir_very_dirty_syncs setting. Timo Sirainen <tss@iki.fi> parents: 8844 diff changeset	1524 void maildir_uidlist_set_all_nonsynced(struct maildir_uidlist *uidlist)
88aeadb32151 Added maildir_very_dirty_syncs setting. Timo Sirainen <tss@iki.fi> parents: 8844 diff changeset	1525 {
88aeadb32151 Added maildir_very_dirty_syncs setting. Timo Sirainen <tss@iki.fi> parents: 8844 diff changeset	1526 maildir_uidlist_mark_all(uidlist, TRUE);
88aeadb32151 Added maildir_very_dirty_syncs setting. Timo Sirainen <tss@iki.fi> parents: 8844 diff changeset	1527 }
88aeadb32151 Added maildir_very_dirty_syncs setting. Timo Sirainen <tss@iki.fi> parents: 8844 diff changeset	1528
8554 c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1529 int maildir_uidlist_sync_init(struct maildir_uidlist *uidlist,
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1530 enum maildir_uidlist_sync_flags sync_flags,
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1531 struct maildir_uidlist_sync_ctx **sync_ctx_r)
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1532 {
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1533 struct maildir_uidlist_sync_ctx *ctx;
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1534 bool locked;
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1535 int ret;
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1536
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1537 ret = maildir_uidlist_sync_lock(uidlist, sync_flags, &locked);
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1538 if (ret <= 0)
c438437b709a maildir: Allow opening mailbox and expunging messages when uidlist couldn't be locked (e.g. out of quota). Timo Sirainen <tss@iki.fi> parents: 8371 diff changeset	1539 return ret;
3530 e9695ec7925b Recursive maildir uidlist syncs caused assert crashes. Also did some Timo Sirainen <tss@iki.fi> parents: 3472 diff changeset	1540
e9695ec7925b Recursive maildir uidlist syncs caused assert crashes. Also did some Timo Sirainen <tss@iki.fi> parents: 3472 diff changeset	1541 *sync_ctx_r = ctx = i_new(struct maildir_uidlist_sync_ctx, 1);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1542 ctx->uidlist = uidlist;
5393 434047e53586 When saving/copying mails, we must not do any sorting based on the filename Timo Sirainen <tss@iki.fi> parents: 5388 diff changeset	1543 ctx->sync_flags = sync_flags;
6886 82420ed23379 If doing a lockless forced sync, track the UID we're trying to find. If we Timo Sirainen <tss@iki.fi> parents: 6884 diff changeset	1544 ctx->partial = !locked \|\|
82420ed23379 If doing a lockless forced sync, track the UID we're trying to find. If we Timo Sirainen <tss@iki.fi> parents: 6884 diff changeset	1545 (sync_flags & MAILDIR_UIDLIST_SYNC_PARTIAL) != 0;
82420ed23379 If doing a lockless forced sync, track the UID we're trying to find. If we Timo Sirainen <tss@iki.fi> parents: 6884 diff changeset	1546 ctx->locked = locked;
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1547 ctx->first_unwritten_pos = (unsigned int)-1;
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1548 ctx->first_nouid_pos = (unsigned int)-1;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1549
5393 434047e53586 When saving/copying mails, we must not do any sorting based on the filename Timo Sirainen <tss@iki.fi> parents: 5388 diff changeset	1550 if (ctx->partial) {
6028 68be663e79ec Fixed reassigning new UIDs to more than one message. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	1551 if ((sync_flags & MAILDIR_UIDLIST_SYNC_KEEP_STATE) == 0) {
68be663e79ec Fixed reassigning new UIDs to more than one message. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	1552 /* initially mark all nonsynced */
68be663e79ec Fixed reassigning new UIDs to more than one message. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	1553 maildir_uidlist_mark_all(uidlist, TRUE);
68be663e79ec Fixed reassigning new UIDs to more than one message. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	1554 }
3530 e9695ec7925b Recursive maildir uidlist syncs caused assert crashes. Also did some Timo Sirainen <tss@iki.fi> parents: 3472 diff changeset	1555 return 1;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1556 }
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1557 i_assert(uidlist->locked_refresh);
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1558
4980 8e88ecc64563 debug: MEMPOOL_GROWING prefix in alloconly pool names means that when Timo Sirainen <tss@iki.fi> parents: 4969 diff changeset	1559 ctx->record_pool = pool_alloconly_create(MEMPOOL_GROWING
8e88ecc64563 debug: MEMPOOL_GROWING prefix in alloconly pool names means that when Timo Sirainen <tss@iki.fi> parents: 4969 diff changeset	1560 "maildir_uidlist_sync", 16384);
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1561 ctx->files = hash_table_create(default_pool, ctx->record_pool, 4096,
f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1562 maildir_filename_base_hash,
f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1563 maildir_filename_base_cmp);
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1564
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1565 i_array_init(&ctx->records, array_count(&uidlist->records));
3530 e9695ec7925b Recursive maildir uidlist syncs caused assert crashes. Also did some Timo Sirainen <tss@iki.fi> parents: 3472 diff changeset	1566 return 1;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1567 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1568
5921 00bdb1f546d3 minor cleanup Timo Sirainen <tss@iki.fi> parents: 5919 diff changeset	1569 static void
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1570 maildir_uidlist_sync_next_partial(struct maildir_uidlist_sync_ctx *ctx,
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1571 const char *filename,
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1572 enum maildir_uidlist_rec_flag flags)
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1573 {
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1574 struct maildir_uidlist *uidlist = ctx->uidlist;
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1575 struct maildir_uidlist_rec *rec;
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1576
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1577 /* we'll update uidlist directly */
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1578 rec = hash_table_lookup(uidlist->files, filename);
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1579 if (rec == NULL) {
6800 ca849e28a062 NFS: Delay flushing dovecot-uidlist's attribute cache until we find an Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1580 /* doesn't exist in uidlist */
6880 724c8f12eed2 maildir_storage_sync_force(): Don't wait on trying to lock uidlist. Just try Timo Sirainen <tss@iki.fi> parents: 6859 diff changeset	1581 if (!ctx->locked) {
724c8f12eed2 maildir_storage_sync_force(): Don't wait on trying to lock uidlist. Just try Timo Sirainen <tss@iki.fi> parents: 6859 diff changeset	1582 /* we can't add it, so just ignore it */
724c8f12eed2 maildir_storage_sync_force(): Don't wait on trying to lock uidlist. Just try Timo Sirainen <tss@iki.fi> parents: 6859 diff changeset	1583 return;
724c8f12eed2 maildir_storage_sync_force(): Don't wait on trying to lock uidlist. Just try Timo Sirainen <tss@iki.fi> parents: 6859 diff changeset	1584 }
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1585 if (ctx->first_nouid_pos == (unsigned int)-1)
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1586 ctx->first_nouid_pos = array_count(&uidlist->records);
1973 f6d2dba3b78b and yet more syncing fixes.. Timo Sirainen <tss@iki.fi> parents: 1971 diff changeset	1587 ctx->new_files_count++;
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1588 ctx->changed = TRUE;
1960 b9ea4db3f8e0 crashfix Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	1589
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	1590 if (uidlist->record_pool == NULL) {
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	1591 uidlist->record_pool =
4980 8e88ecc64563 debug: MEMPOOL_GROWING prefix in alloconly pool names means that when Timo Sirainen <tss@iki.fi> parents: 4969 diff changeset	1592 pool_alloconly_create(MEMPOOL_GROWING
8e88ecc64563 debug: MEMPOOL_GROWING prefix in alloconly pool names means that when Timo Sirainen <tss@iki.fi> parents: 4969 diff changeset	1593 "uidlist record_pool",
2051 596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	1594 1024);
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	1595 }
596267d8e2e2 Some more UIDVALIDITY issues fixed. Timo Sirainen <tss@iki.fi> parents: 2050 diff changeset	1596
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1597 rec = p_new(uidlist->record_pool,
4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1598 struct maildir_uidlist_rec, 1);
1973 f6d2dba3b78b and yet more syncing fixes.. Timo Sirainen <tss@iki.fi> parents: 1971 diff changeset	1599 rec->uid = (uint32_t)-1;
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1600 array_append(&uidlist->records, &rec, 1);
5913 699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1601 uidlist->change_counter++;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1602 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1603
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1604 rec->flags = (rec->flags \| flags) & ~MAILDIR_UIDLIST_REC_FLAG_NONSYNCED;
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1605 rec->filename = p_strdup(uidlist->record_pool, filename);
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1606 hash_table_insert(uidlist->files, rec->filename, rec);
6029 10042803d65e Crashfixes to handling new messages. Timo Sirainen <tss@iki.fi> parents: 6028 diff changeset	1607
10042803d65e Crashfixes to handling new messages. Timo Sirainen <tss@iki.fi> parents: 6028 diff changeset	1608 ctx->finished = FALSE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1609 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1610
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1611 static unsigned char ext_dup(pool_t pool, const unsigned char extensions)
7170 36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1612 {
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1613 unsigned char *ret;
7170 36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1614
36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1615 if (extensions == NULL)
36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1616 return NULL;
36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1617
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7171 diff changeset	1618 T_BEGIN {
7170 36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1619 unsigned int len;
36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1620
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1621 for (len = 0; extensions[len] != '\0'; len++) {
2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1622 while (extensions[len] != '\0') len++;
7170 36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1623 }
7171 2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1624 ret = p_malloc(pool, len + 1);
2e60288f0d75 Previous garbage fix was still incorrect. Fixed it and also changed Timo Sirainen <tss@iki.fi> parents: 7170 diff changeset	1625 memcpy(ret, extensions, len);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7171 diff changeset	1626 } T_END;
7170 36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1627 return ret;
36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1628 }
36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1629
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1630 int maildir_uidlist_sync_next(struct maildir_uidlist_sync_ctx *ctx,
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1631 const char *filename,
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1632 enum maildir_uidlist_rec_flag flags)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1633 {
1960 b9ea4db3f8e0 crashfix Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	1634 struct maildir_uidlist *uidlist = ctx->uidlist;
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1635 struct maildir_uidlist_rec rec, old_rec;
7996 fa90c913eae4 Maildir: If filename has CR or LF, log a warning and ignore it instead of Timo Sirainen <tss@iki.fi> parents: 7976 diff changeset	1636 const char *p;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1637
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1638 if (ctx->failed)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1639 return -1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1640
7996 fa90c913eae4 Maildir: If filename has CR or LF, log a warning and ignore it instead of Timo Sirainen <tss@iki.fi> parents: 7976 diff changeset	1641 for (p = filename; *p != '\0'; p++) {
fa90c913eae4 Maildir: If filename has CR or LF, log a warning and ignore it instead of Timo Sirainen <tss@iki.fi> parents: 7976 diff changeset	1642 if (p == 13 \|\| p == 10) {
fa90c913eae4 Maildir: If filename has CR or LF, log a warning and ignore it instead of Timo Sirainen <tss@iki.fi> parents: 7976 diff changeset	1643 i_warning("Maildir %s: Ignoring a file with #0x%x: %s",
fa90c913eae4 Maildir: If filename has CR or LF, log a warning and ignore it instead of Timo Sirainen <tss@iki.fi> parents: 7976 diff changeset	1644 uidlist->mbox->path, *p, filename);
fa90c913eae4 Maildir: If filename has CR or LF, log a warning and ignore it instead of Timo Sirainen <tss@iki.fi> parents: 7976 diff changeset	1645 return 1;
fa90c913eae4 Maildir: If filename has CR or LF, log a warning and ignore it instead of Timo Sirainen <tss@iki.fi> parents: 7976 diff changeset	1646 }
fa90c913eae4 Maildir: If filename has CR or LF, log a warning and ignore it instead of Timo Sirainen <tss@iki.fi> parents: 7976 diff changeset	1647 }
fa90c913eae4 Maildir: If filename has CR or LF, log a warning and ignore it instead of Timo Sirainen <tss@iki.fi> parents: 7976 diff changeset	1648
5921 00bdb1f546d3 minor cleanup Timo Sirainen <tss@iki.fi> parents: 5919 diff changeset	1649 if (ctx->partial) {
00bdb1f546d3 minor cleanup Timo Sirainen <tss@iki.fi> parents: 5919 diff changeset	1650 maildir_uidlist_sync_next_partial(ctx, filename, flags);
00bdb1f546d3 minor cleanup Timo Sirainen <tss@iki.fi> parents: 5919 diff changeset	1651 return 1;
00bdb1f546d3 minor cleanup Timo Sirainen <tss@iki.fi> parents: 5919 diff changeset	1652 }
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1653
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1654 rec = hash_table_lookup(ctx->files, filename);
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1655 if (rec != NULL) {
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1656 if ((rec->flags & (MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1657 MAILDIR_UIDLIST_REC_FLAG_MOVED)) == 0) {
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1658 /* possibly duplicate */
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1659 return 0;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1660 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1661
6037 d911d943438e Recent flag handling rewrite. Still not perfect with maildir. Timo Sirainen <tss@iki.fi> parents: 6034 diff changeset	1662 /* probably was in new/ and now we're seeing it in cur/.
d911d943438e Recent flag handling rewrite. Still not perfect with maildir. Timo Sirainen <tss@iki.fi> parents: 6034 diff changeset	1663 remove new/moved flags so if this happens again we'll know
d911d943438e Recent flag handling rewrite. Still not perfect with maildir. Timo Sirainen <tss@iki.fi> parents: 6034 diff changeset	1664 to check for duplicates. */
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1665 rec->flags &= ~(MAILDIR_UIDLIST_REC_FLAG_NEW_DIR \|
777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1666 MAILDIR_UIDLIST_REC_FLAG_MOVED);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1667 } else {
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1668 old_rec = hash_table_lookup(uidlist->files, filename);
2123 e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	1669 i_assert(old_rec != NULL \|\| UIDLIST_IS_LOCKED(uidlist));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1670
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1671 rec = p_new(ctx->record_pool, struct maildir_uidlist_rec, 1);
1960 b9ea4db3f8e0 crashfix Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	1672
6442 6f12401a149e Crashfix Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1673 if (old_rec != NULL) {
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1674 rec = old_rec;
6442 6f12401a149e Crashfix Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1675 rec->extensions =
7170 36092ec27396 Extensions weren't copied in memory correctly, possibly causing garbage to Timo Sirainen <tss@iki.fi> parents: 7169 diff changeset	1676 ext_dup(ctx->record_pool, rec->extensions);
6442 6f12401a149e Crashfix Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1677 } else {
1973 f6d2dba3b78b and yet more syncing fixes.. Timo Sirainen <tss@iki.fi> parents: 1971 diff changeset	1678 rec->uid = (uint32_t)-1;
f6d2dba3b78b and yet more syncing fixes.. Timo Sirainen <tss@iki.fi> parents: 1971 diff changeset	1679 ctx->new_files_count++;
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1680 ctx->changed = TRUE;
6037 d911d943438e Recent flag handling rewrite. Still not perfect with maildir. Timo Sirainen <tss@iki.fi> parents: 6034 diff changeset	1681 /* didn't exist in uidlist, it's recent */
d911d943438e Recent flag handling rewrite. Still not perfect with maildir. Timo Sirainen <tss@iki.fi> parents: 6034 diff changeset	1682 flags \|= MAILDIR_UIDLIST_REC_FLAG_RECENT;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1683 }
1971 e7244b4dd03c sync fix Timo Sirainen <tss@iki.fi> parents: 1962 diff changeset	1684
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1685 array_append(&ctx->records, &rec, 1);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1686 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1687
2003 d1a2c2ae7226 sync fix Timo Sirainen <tss@iki.fi> parents: 1988 diff changeset	1688 rec->flags = (rec->flags \| flags) & ~MAILDIR_UIDLIST_REC_FLAG_NONSYNCED;
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1689 rec->filename = p_strdup(ctx->record_pool, filename);
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1690 hash_table_insert(ctx->files, rec->filename, rec);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1691 return 1;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1692 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1693
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1694 void maildir_uidlist_sync_remove(struct maildir_uidlist_sync_ctx *ctx,
239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1695 const char *filename)
239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1696 {
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1697 struct maildir_uidlist_rec *rec;
8304 ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1698 unsigned int idx;
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1699
239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1700 i_assert(ctx->partial);
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1701 i_assert(ctx->uidlist->locked_refresh);
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1702
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1703 rec = hash_table_lookup(ctx->uidlist->files, filename);
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1704 i_assert(rec != NULL);
6029 10042803d65e Crashfixes to handling new messages. Timo Sirainen <tss@iki.fi> parents: 6028 diff changeset	1705 i_assert(rec->uid != (uint32_t)-1);
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1706
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1707 hash_table_remove(ctx->uidlist->files, filename);
8304 ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1708 idx = maildir_uidlist_records_array_delete(ctx->uidlist, rec);
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1709
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1710 if (ctx->first_unwritten_pos != (unsigned int)-1) {
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1711 i_assert(ctx->first_unwritten_pos > idx);
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1712 ctx->first_unwritten_pos--;
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1713 }
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1714 if (ctx->first_nouid_pos != (unsigned int)-1) {
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1715 i_assert(ctx->first_nouid_pos > idx);
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1716 ctx->first_nouid_pos--;
ab22ace44190 Maildir: Added more strict asserts. Timo Sirainen <tss@iki.fi> parents: 8303 diff changeset	1717 }
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1718
239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1719 ctx->changed = TRUE;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1720 ctx->uidlist->recreate = TRUE;
5923 239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1721 }
239078f515a8 Added maildir_uidlist_sync_remove() Timo Sirainen <tss@iki.fi> parents: 5922 diff changeset	1722
4397 5cbabd4ccd9c Don't go fixing duplicate maildir filenames without properly checking that Timo Sirainen <tss@iki.fi> parents: 4238 diff changeset	1723 const char *
5cbabd4ccd9c Don't go fixing duplicate maildir filenames without properly checking that Timo Sirainen <tss@iki.fi> parents: 4238 diff changeset	1724 maildir_uidlist_sync_get_full_filename(struct maildir_uidlist_sync_ctx *ctx,
5cbabd4ccd9c Don't go fixing duplicate maildir filenames without properly checking that Timo Sirainen <tss@iki.fi> parents: 4238 diff changeset	1725 const char *filename)
5cbabd4ccd9c Don't go fixing duplicate maildir filenames without properly checking that Timo Sirainen <tss@iki.fi> parents: 4238 diff changeset	1726 {
5cbabd4ccd9c Don't go fixing duplicate maildir filenames without properly checking that Timo Sirainen <tss@iki.fi> parents: 4238 diff changeset	1727 struct maildir_uidlist_rec *rec;
5cbabd4ccd9c Don't go fixing duplicate maildir filenames without properly checking that Timo Sirainen <tss@iki.fi> parents: 4238 diff changeset	1728
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1729 rec = hash_table_lookup(ctx->files, filename);
4397 5cbabd4ccd9c Don't go fixing duplicate maildir filenames without properly checking that Timo Sirainen <tss@iki.fi> parents: 4238 diff changeset	1730 return rec == NULL ? NULL : rec->filename;
5cbabd4ccd9c Don't go fixing duplicate maildir filenames without properly checking that Timo Sirainen <tss@iki.fi> parents: 4238 diff changeset	1731 }
5cbabd4ccd9c Don't go fixing duplicate maildir filenames without properly checking that Timo Sirainen <tss@iki.fi> parents: 4238 diff changeset	1732
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1733 bool maildir_uidlist_get_uid(struct maildir_uidlist *uidlist,
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1734 const char filename, uint32_t uid_r)
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1735 {
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1736 struct maildir_uidlist_rec *rec;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1737
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1738 rec = hash_table_lookup(uidlist->files, filename);
6599 a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1739 if (rec == NULL)
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1740 return FALSE;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1741
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1742 *uid_r = rec->uid;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1743 return TRUE;
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1744 }
a6f51026b969 Added maildir_uidlist_init_readonly() and maildir_uidlist_get_uid(). Timo Sirainen <tss@iki.fi> parents: 6598 diff changeset	1745
4960 4714befe2604 Added maildir_copy_preserve_filename setting. Timo Sirainen <timo.sirainen@movial.fi> parents: 4808 diff changeset	1746 const char *
4714befe2604 Added maildir_copy_preserve_filename setting. Timo Sirainen <timo.sirainen@movial.fi> parents: 4808 diff changeset	1747 maildir_uidlist_get_full_filename(struct maildir_uidlist *uidlist,
4714befe2604 Added maildir_copy_preserve_filename setting. Timo Sirainen <timo.sirainen@movial.fi> parents: 4808 diff changeset	1748 const char *filename)
4714befe2604 Added maildir_copy_preserve_filename setting. Timo Sirainen <timo.sirainen@movial.fi> parents: 4808 diff changeset	1749 {
4714befe2604 Added maildir_copy_preserve_filename setting. Timo Sirainen <timo.sirainen@movial.fi> parents: 4808 diff changeset	1750 struct maildir_uidlist_rec *rec;
4714befe2604 Added maildir_copy_preserve_filename setting. Timo Sirainen <timo.sirainen@movial.fi> parents: 4808 diff changeset	1751
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1752 rec = hash_table_lookup(uidlist->files, filename);
4960 4714befe2604 Added maildir_copy_preserve_filename setting. Timo Sirainen <timo.sirainen@movial.fi> parents: 4808 diff changeset	1753 return rec == NULL ? NULL : rec->filename;
4714befe2604 Added maildir_copy_preserve_filename setting. Timo Sirainen <timo.sirainen@movial.fi> parents: 4808 diff changeset	1754 }
4714befe2604 Added maildir_copy_preserve_filename setting. Timo Sirainen <timo.sirainen@movial.fi> parents: 4808 diff changeset	1755
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1756 static int maildir_time_cmp(const void p1, const void p2)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1757 {
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1758 const struct maildir_uidlist_rec const rec1 = p1, const rec2 = p2;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1759
5903 5d0827cd9839 Added maildir_filename_sort_cmp() which sorts maildir filenames by time and Timo Sirainen <tss@iki.fi> parents: 5902 diff changeset	1760 return maildir_filename_sort_cmp((rec1)->filename, (rec2)->filename);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1761 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1762
6029 10042803d65e Crashfixes to handling new messages. Timo Sirainen <tss@iki.fi> parents: 6028 diff changeset	1763 static void maildir_uidlist_assign_uids(struct maildir_uidlist_sync_ctx *ctx)
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1764 {
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1765 struct maildir_uidlist_rec **recs;
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1766 unsigned int dest, count;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1767
5393 434047e53586 When saving/copying mails, we must not do any sorting based on the filename Timo Sirainen <tss@iki.fi> parents: 5388 diff changeset	1768 i_assert(UIDLIST_IS_LOCKED(ctx->uidlist));
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1769 i_assert(ctx->first_nouid_pos != (unsigned int)-1);
2094 8f46cf3e9b5a small fixes Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	1770
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1771 if (ctx->first_unwritten_pos == (unsigned int)-1)
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1772 ctx->first_unwritten_pos = ctx->first_nouid_pos;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1773
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1774 /* sort new files and assign UIDs for them */
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1775 recs = array_get_modifiable(&ctx->uidlist->records, &count);
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1776 qsort(recs + ctx->first_nouid_pos, count - ctx->first_nouid_pos,
5892 36056f044f1f Removed unused MAILDIR_UIDLIST_SYNC_ORDERED flag. Timo Sirainen <tss@iki.fi> parents: 5891 diff changeset	1777 sizeof(*recs), maildir_time_cmp);
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1778
dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1779 for (dest = ctx->first_nouid_pos; dest < count; dest++) {
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1780 i_assert(recs[dest]->uid == (uint32_t)-1);
9211 219ee9858156 maildir: Added some checks for UIDs reaching 2^31-1. Timo Sirainen <tss@iki.fi> parents: 9168 diff changeset	1781 i_assert(ctx->uidlist->next_uid < (uint32_t)-1);
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1782 recs[dest]->uid = ctx->uidlist->next_uid++;
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1783 recs[dest]->flags &= ~MAILDIR_UIDLIST_REC_FLAG_MOVED;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1784 }
2567 6c802bcb957e fix Timo Sirainen <tss@iki.fi> parents: 2565 diff changeset	1785
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1786 if (ctx->uidlist->locked_refresh)
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1787 ctx->uidlist->last_seen_uid = ctx->uidlist->next_uid-1;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1788
6029 10042803d65e Crashfixes to handling new messages. Timo Sirainen <tss@iki.fi> parents: 6028 diff changeset	1789 ctx->new_files_count = 0;
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1790 ctx->first_nouid_pos = (unsigned int)-1;
6029 10042803d65e Crashfixes to handling new messages. Timo Sirainen <tss@iki.fi> parents: 6028 diff changeset	1791 ctx->uidlist->change_counter++;
7271 4ac4e9ba1124 Make sure we don't use a stale first_unwritten_pos to optimize Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	1792 ctx->finish_change_counter = ctx->uidlist->change_counter;
1954 2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1793 }
2f6e137cdc44 Syncing optimizations. Timo Sirainen <tss@iki.fi> parents: 1947 diff changeset	1794
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1795 static void maildir_uidlist_swap(struct maildir_uidlist_sync_ctx *ctx)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1796 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1797 struct maildir_uidlist *uidlist = ctx->uidlist;
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1798 struct maildir_uidlist_rec **recs;
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1799 unsigned int count;
1960 b9ea4db3f8e0 crashfix Timo Sirainen <tss@iki.fi> parents: 1954 diff changeset	1800
2094 8f46cf3e9b5a small fixes Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	1801 /* buffer is unsorted, sort it by UID */
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1802 recs = array_get_modifiable(&ctx->records, &count);
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1803 qsort(recs, count, sizeof(*recs), maildir_uid_cmp);
1947 777da553d1d3 Recent-flag should work now Timo Sirainen <tss@iki.fi> parents: 1940 diff changeset	1804
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1805 array_free(&uidlist->records);
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1806 uidlist->records = ctx->records;
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1807 ctx->records.arr.buffer = NULL;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1808
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1809 hash_table_destroy(&uidlist->files);
1931 35a947481130 fixes Timo Sirainen <tss@iki.fi> parents: 1915 diff changeset	1810 uidlist->files = ctx->files;
35a947481130 fixes Timo Sirainen <tss@iki.fi> parents: 1915 diff changeset	1811 ctx->files = NULL;
35a947481130 fixes Timo Sirainen <tss@iki.fi> parents: 1915 diff changeset	1812
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1813 if (uidlist->record_pool != NULL)
6428 7cad076906eb pool_unref() now takes ** pointer. Timo Sirainen <tss@iki.fi> parents: 6417 diff changeset	1814 pool_unref(&uidlist->record_pool);
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1815 uidlist->record_pool = ctx->record_pool;
4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1816 ctx->record_pool = NULL;
4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1817
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1818 if (ctx->new_files_count != 0) {
6034 dcaa8d5296a9 Fixes to handling appends and unexpunges. Timo Sirainen <tss@iki.fi> parents: 6029 diff changeset	1819 ctx->first_nouid_pos = count - ctx->new_files_count;
6029 10042803d65e Crashfixes to handling new messages. Timo Sirainen <tss@iki.fi> parents: 6028 diff changeset	1820 maildir_uidlist_assign_uids(ctx);
7271 4ac4e9ba1124 Make sure we don't use a stale first_unwritten_pos to optimize Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	1821 } else {
4ac4e9ba1124 Make sure we don't use a stale first_unwritten_pos to optimize Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	1822 ctx->uidlist->change_counter++;
5933 fdc7e47ccea3 dovecot-uidlist can now be updated by appending to it. It's recreated only Timo Sirainen <tss@iki.fi> parents: 5932 diff changeset	1823 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1824 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1825
2123 e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	1826 void maildir_uidlist_sync_finish(struct maildir_uidlist_sync_ctx *ctx)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1827 {
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1975 diff changeset	1828 if (!ctx->partial) {
2123 e01de478882f locking fixes Timo Sirainen <tss@iki.fi> parents: 2121 diff changeset	1829 if (!ctx->failed)
1978 6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1975 diff changeset	1830 maildir_uidlist_swap(ctx);
6303ef092c5b mbox code compiles again, but syncing is only partially implemented so Timo Sirainen <tss@iki.fi> parents: 1975 diff changeset	1831 } else {
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1832 if (ctx->new_files_count != 0 && !ctx->failed) {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1833 i_assert(ctx->changed);
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1834 i_assert(ctx->locked);
6029 10042803d65e Crashfixes to handling new messages. Timo Sirainen <tss@iki.fi> parents: 6028 diff changeset	1835 maildir_uidlist_assign_uids(ctx);
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1836 }
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1837 }
2094 8f46cf3e9b5a small fixes Timo Sirainen <tss@iki.fi> parents: 2053 diff changeset	1838
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	1839 ctx->finished = TRUE;
7611 5f481022db04 maildir: Write dovecot-uidlist before syncing index to make sure that any Timo Sirainen <tss@iki.fi> parents: 7588 diff changeset	1840
5f481022db04 maildir: Write dovecot-uidlist before syncing index to make sure that any Timo Sirainen <tss@iki.fi> parents: 7588 diff changeset	1841 i_assert(ctx->locked \|\| !ctx->changed);
9016 6770f46971af Maildir saving: Fixed race condition bugs in uidlist handling, causing files to be given new UIDs sometimes. Timo Sirainen <tss@iki.fi> parents: 8992 diff changeset	1842 if ((ctx->changed \|\| maildir_uidlist_want_compress(ctx)) &&
7611 5f481022db04 maildir: Write dovecot-uidlist before syncing index to make sure that any Timo Sirainen <tss@iki.fi> parents: 7588 diff changeset	1843 !ctx->failed && ctx->locked) T_BEGIN {
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1844 if (maildir_uidlist_sync_update(ctx) < 0) {
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1845 /* we couldn't write everything we wanted. make sure
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1846 we don't continue using those UIDs */
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1847 maildir_uidlist_reset(ctx->uidlist);
7611 5f481022db04 maildir: Write dovecot-uidlist before syncing index to make sure that any Timo Sirainen <tss@iki.fi> parents: 7588 diff changeset	1848 ctx->failed = TRUE;
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1849 }
7611 5f481022db04 maildir: Write dovecot-uidlist before syncing index to make sure that any Timo Sirainen <tss@iki.fi> parents: 7588 diff changeset	1850 } T_END;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	1851 }
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	1852
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1853 int maildir_uidlist_sync_deinit(struct maildir_uidlist_sync_ctx **_ctx,
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1854 bool success)
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	1855 {
4238 3c8b191b0019 Adding mail to index while saving it had a race condition. Fixing it Timo Sirainen <timo.sirainen@movial.fi> parents: 4225 diff changeset	1856 struct maildir_uidlist_sync_ctx ctx = _ctx;
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1857 int ret;
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	1858
4238 3c8b191b0019 Adding mail to index while saving it had a race condition. Fixing it Timo Sirainen <timo.sirainen@movial.fi> parents: 4225 diff changeset	1859 *_ctx = NULL;
3c8b191b0019 Adding mail to index while saving it had a race condition. Fixing it Timo Sirainen <timo.sirainen@movial.fi> parents: 4225 diff changeset	1860
9018 0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1861 if (!success)
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1862 ctx->failed = TRUE;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1863 ret = ctx->failed ? -1 : 0;
0bb192fe0abd Maildir: More fixes to uidlist handling. Timo Sirainen <tss@iki.fi> parents: 9017 diff changeset	1864
1984 9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	1865 if (!ctx->finished)
9c159272f721 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1978 diff changeset	1866 maildir_uidlist_sync_finish(ctx);
1988 739b469b99e2 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1987 diff changeset	1867 if (ctx->partial)
739b469b99e2 syncing fixes Timo Sirainen <tss@iki.fi> parents: 1987 diff changeset	1868 maildir_uidlist_mark_all(ctx->uidlist, FALSE);
6880 724c8f12eed2 maildir_storage_sync_force(): Don't wait on trying to lock uidlist. Just try Timo Sirainen <tss@iki.fi> parents: 6859 diff changeset	1869 if (ctx->locked)
724c8f12eed2 maildir_storage_sync_force(): Don't wait on trying to lock uidlist. Just try Timo Sirainen <tss@iki.fi> parents: 6859 diff changeset	1870 maildir_uidlist_unlock(ctx->uidlist);
1940 876bd2849a00 fixes Timo Sirainen <tss@iki.fi> parents: 1934 diff changeset	1871
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1872 if (ctx->files != NULL)
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1873 hash_table_destroy(&ctx->files);
1962 4775ffa73874 Syncing should be fixed now. Timo Sirainen <tss@iki.fi> parents: 1960 diff changeset	1874 if (ctx->record_pool != NULL)
6428 7cad076906eb pool_unref() now takes ** pointer. Timo Sirainen <tss@iki.fi> parents: 6417 diff changeset	1875 pool_unref(&ctx->record_pool);
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1876 if (array_is_created(&ctx->records))
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1877 array_free(&ctx->records);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1878 i_free(ctx);
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1879 return ret;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1880 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1881
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1882 void maildir_uidlist_add_flags(struct maildir_uidlist *uidlist,
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1883 const char *filename,
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1884 enum maildir_uidlist_rec_flag flags)
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1885 {
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1886 struct maildir_uidlist_rec *rec;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1887
8573 f9166a09423a Renamed hash_() to hash_table_() to avoid conflicts with OSX's strhash.h Timo Sirainen <tss@iki.fi> parents: 8554 diff changeset	1888 rec = hash_table_lookup(uidlist->files, filename);
2053 66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1889 i_assert(rec != NULL);
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1890
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1891 rec->flags \|= flags;
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1892 }
66f5d28f9b27 race condition fixes Timo Sirainen <tss@iki.fi> parents: 2052 diff changeset	1893
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1894 struct maildir_uidlist_iter_ctx *
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1895 maildir_uidlist_iter_init(struct maildir_uidlist *uidlist)
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1896 {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1897 struct maildir_uidlist_iter_ctx *ctx;
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1898 unsigned int count;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1899
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1900 ctx = i_new(struct maildir_uidlist_iter_ctx, 1);
5913 699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1901 ctx->uidlist = uidlist;
5890 989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1902 ctx->next = array_get(&uidlist->records, &count);
989abd5017a4 Changed record buffers to arrays. Timo Sirainen <tss@iki.fi> parents: 5740 diff changeset	1903 ctx->end = ctx->next + count;
5913 699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1904 ctx->change_counter = ctx->uidlist->change_counter;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1905 return ctx;
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1906 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1907
5913 699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1908 static void
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1909 maildir_uidlist_iter_update_idx(struct maildir_uidlist_iter_ctx *ctx)
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1910 {
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1911 unsigned int old_rev_idx, idx, count;
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1912
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1913 old_rev_idx = ctx->end - ctx->next;
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1914 ctx->next = array_get(&ctx->uidlist->records, &count);
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1915 ctx->end = ctx->next + count;
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1916
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1917 idx = old_rev_idx >= count ? 0 :
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1918 count - old_rev_idx;
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1919 while (idx < count && ctx->next[idx]->uid <= ctx->prev_uid)
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1920 idx++;
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1921 while (idx > 0 && ctx->next[idx-1]->uid > ctx->prev_uid)
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1922 idx--;
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1923
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1924 ctx->next += idx;
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1925 }
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1926
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1927 static bool maildir_uidlist_iter_next_rec(struct maildir_uidlist_iter_ctx *ctx,
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1928 struct maildir_uidlist_rec **rec_r)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1929 {
5922 79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1930 struct maildir_uidlist_rec *rec;
79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1931
5913 699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1932 if (ctx->change_counter != ctx->uidlist->change_counter)
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1933 maildir_uidlist_iter_update_idx(ctx);
699356ffe549 If uidlist changes while it's being iterated, don't crash. Timo Sirainen <tss@iki.fi> parents: 5903 diff changeset	1934
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1935 if (ctx->next == ctx->end)
5942 bf536b8dbc51 maildir_uidlist_iter_next() should return bool, not int. Timo Sirainen <tss@iki.fi> parents: 5941 diff changeset	1936 return FALSE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1937
5922 79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1938 rec = *ctx->next;
79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1939 i_assert(rec->uid != (uint32_t)-1);
79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1940
79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1941 ctx->prev_uid = rec->uid;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1942 ctx->next++;
5922 79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1943
5944 760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1944 *rec_r = rec;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1945 return TRUE;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1946 }
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1947
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1948 bool maildir_uidlist_iter_next(struct maildir_uidlist_iter_ctx *ctx,
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1949 uint32_t *uid_r,
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1950 enum maildir_uidlist_rec_flag *flags_r,
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1951 const char **filename_r)
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1952 {
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1953 struct maildir_uidlist_rec *rec;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1954
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1955 if (!maildir_uidlist_iter_next_rec(ctx, &rec))
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1956 return FALSE;
760f910004d9 Created a new extensible version 3 dovecot-uidlist file. Timo Sirainen <tss@iki.fi> parents: 5942 diff changeset	1957
5922 79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1958 *uid_r = rec->uid;
79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1959 *flags_r = rec->flags;
79513299d42c code cleanup Timo Sirainen <tss@iki.fi> parents: 5921 diff changeset	1960 *filename_r = rec->filename;
5942 bf536b8dbc51 maildir_uidlist_iter_next() should return bool, not int. Timo Sirainen <tss@iki.fi> parents: 5941 diff changeset	1961 return TRUE;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1962 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1963
6409 7f733ba453bf Changed _deinit() APIs to take * pointer and set it to NULL. Timo Sirainen <tss@iki.fi> parents: 6344 diff changeset	1964 void maildir_uidlist_iter_deinit(struct maildir_uidlist_iter_ctx **_ctx)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1965 {
6409 7f733ba453bf Changed _deinit() APIs to take * pointer and set it to NULL. Timo Sirainen <tss@iki.fi> parents: 6344 diff changeset	1966 i_free(*_ctx);
7f733ba453bf Changed _deinit() APIs to take * pointer and set it to NULL. Timo Sirainen <tss@iki.fi> parents: 6344 diff changeset	1967 *_ctx = NULL;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	1968 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-storage/index/maildir/maildir-uidlist.c @ 9488:e47eb506eebd HEAD