dovecot/core-2.2: src/lib-index/mail-index-strmap.c annotate

annotate src/lib-index/mail-index-strmap.c @ 22576:707ae9de3812

lib: istream-multiplex - Minor code cleanup Avoid propagating the error twice, and avoid any confusion about what "got" actually contains.

author	Timo Sirainen <timo.sirainen@dovecot.fi>
date	Thu, 05 Oct 2017 20:24:11 +0300
parents	2e2563132d5f
children	cb108f786fb4

rev	line source
21390 2e2563132d5f Updated copyright notices to include the year 2017. Stephan Bosch <stephan.bosch@dovecot.fi> parents: 21389 diff changeset	1 /* Copyright (c) 2008-2017 Dovecot authors, see the included COPYING file */
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	2
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	3 #include "lib.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	4 #include "array.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	5 #include "bsearch-insert-pos.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	6 #include "istream.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	7 #include "ostream.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	8 #include "file-lock.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	9 #include "file-dotlock.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	10 #include "crc32.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	11 #include "safe-mkstemp.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	12 #include "str.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	13 #include "mail-index-private.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	14 #include "mail-index-strmap.h"
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	15
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	16 #include <stdio.h>
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	17
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	18 struct mail_index_strmap {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	19 struct mail_index *index;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	20 char *path;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	21 int fd;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	22 struct istream *input;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	23
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	24 struct file_lock *file_lock;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	25 struct dotlock *dotlock;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	26 struct dotlock_settings dotlock_settings;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	27 };
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	28
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	29 struct mail_index_strmap_view {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	30 struct mail_index_strmap *strmap;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	31 struct mail_index_view *view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	32
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	33 ARRAY_TYPE(mail_index_strmap_rec) recs;
14920 a097ef0a9d6d Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Timo Sirainen <tss@iki.fi> parents: 14688 diff changeset	34 ARRAY(uint32_t) recs_crc32;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	35 struct hash2_table *hash;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	36
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	37 mail_index_strmap_key_cmp_t *key_compare;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	38 mail_index_strmap_rec_cmp_t *rec_compare;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	39 mail_index_strmap_remap_t *remap_cb;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	40 void *cb_context;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	41
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	42 uoff_t last_read_block_offset;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	43 uint32_t last_read_uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	44 uint32_t last_added_uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	45 uint32_t total_ref_count;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	46
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	47 uint32_t last_ref_index;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	48 uint32_t next_str_idx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	49 uint32_t lost_expunged_uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	50
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	51 unsigned int desynced:1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	52 };
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	53
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	54 struct mail_index_strmap_read_context {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	55 struct mail_index_strmap_view *view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	56
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	57 struct istream *input;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	58 uoff_t end_offset;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	59 uint32_t highest_str_idx;
17433 2d5c66e8d77c mail-index: make uid_lookup_idx hold a sequence number, not a 0-based C array index Phil Carmody <phil@dovecot.fi> parents: 17351 diff changeset	60 uint32_t uid_lookup_seq;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	61 uint32_t lost_expunged_uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	62
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	63 const unsigned char data, end, *str_idx_base;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	64 struct mail_index_strmap_rec rec;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	65 uint32_t next_ref_index;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	66 unsigned int rec_size;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	67
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	68 unsigned int too_large_uids:1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	69 };
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	70
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	71 struct mail_index_strmap_view_sync {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	72 struct mail_index_strmap_view *view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	73 };
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	74
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	75 struct mail_index_strmap_hash_key {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	76 const char *str;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	77 uint32_t crc32;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	78 };
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	79
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	80 /* number of bytes required to store one string idx */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	81 #define STRMAP_FILE_STRIDX_SIZE (sizeof(uint32_t)*2)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	82
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	83 /* renumber the string indexes when highest string idx becomes larger than
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	84 <number of indexes>STRMAP_FILE_MAX_STRIDX_MULTIPLIER /
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	85 #define STRMAP_FILE_MAX_STRIDX_MULTIPLIER 2
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	86
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	87 #define STRIDX_MUST_RENUMBER(highest_idx, n_unique_indexes) \
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	88 (highest_idx > n_unique_indexes * STRMAP_FILE_MAX_STRIDX_MULTIPLIER)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	89
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	90 #define MAIL_INDEX_STRMAP_TIMEOUT_SECS 10
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	91
17351 7f6c5e27b908 treewide sparse cleanup - make single-unit-only data static Phil Carmody <phil@dovecot.fi> parents: 17340 diff changeset	92 static const struct dotlock_settings default_dotlock_settings = {
10410 b757dab45756 Removed MEMBER() macro. Require C99 style struct initializer. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	93 .timeout = MAIL_INDEX_STRMAP_TIMEOUT_SECS,
b757dab45756 Removed MEMBER() macro. Require C99 style struct initializer. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	94 .stale_timeout = 30
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	95 };
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	96
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	97 struct mail_index_strmap *
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	98 mail_index_strmap_init(struct mail_index index, const char suffix)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	99 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	100 struct mail_index_strmap *strmap;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	101
10781 0b139aae6276 lib-storage: Thread strmap must not be opened before index is opened. Timo Sirainen <tss@iki.fi> parents: 10640 diff changeset	102 i_assert(index->open_count > 0);
0b139aae6276 lib-storage: Thread strmap must not be opened before index is opened. Timo Sirainen <tss@iki.fi> parents: 10640 diff changeset	103
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	104 strmap = i_new(struct mail_index_strmap, 1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	105 strmap->index = index;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	106 strmap->path = i_strconcat(index->filepath, suffix, NULL);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	107 strmap->fd = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	108
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	109 strmap->dotlock_settings = default_dotlock_settings;
9696 126132cb1c19 index: Removed duplication of mail_index_open_flags from struct mail_index. Timo Sirainen <tss@iki.fi> parents: 9609 diff changeset	110 strmap->dotlock_settings.use_excl_lock =
126132cb1c19 index: Removed duplication of mail_index_open_flags from struct mail_index. Timo Sirainen <tss@iki.fi> parents: 9609 diff changeset	111 (index->flags & MAIL_INDEX_OPEN_FLAG_DOTLOCK_USE_EXCL) != 0;
126132cb1c19 index: Removed duplication of mail_index_open_flags from struct mail_index. Timo Sirainen <tss@iki.fi> parents: 9609 diff changeset	112 strmap->dotlock_settings.nfs_flush =
126132cb1c19 index: Removed duplication of mail_index_open_flags from struct mail_index. Timo Sirainen <tss@iki.fi> parents: 9609 diff changeset	113 (index->flags & MAIL_INDEX_OPEN_FLAG_NFS_FLUSH) != 0;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	114 return strmap;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	115 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	116
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	117 static bool
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	118 mail_index_strmap_read_rec_next(struct mail_index_strmap_read_context *ctx,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	119 uint32_t *crc32_r);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	120
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	121 static void
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	122 mail_index_strmap_set_syscall_error(struct mail_index_strmap *strmap,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	123 const char *function)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	124 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	125 i_assert(function != NULL);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	126
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	127 if (ENOSPACE(errno)) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	128 strmap->index->nodiskspace = TRUE;
9696 126132cb1c19 index: Removed duplication of mail_index_open_flags from struct mail_index. Timo Sirainen <tss@iki.fi> parents: 9609 diff changeset	129 if ((strmap->index->flags &
126132cb1c19 index: Removed duplication of mail_index_open_flags from struct mail_index. Timo Sirainen <tss@iki.fi> parents: 9609 diff changeset	130 MAIL_INDEX_OPEN_FLAG_NEVER_IN_MEMORY) == 0)
8973 f295678cef1e dbox: Make sure indexes are never moved to memory (in case of out-of-disk space). Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	131 return;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	132 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	133
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	134 mail_index_set_error(strmap->index,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	135 "%s failed with strmap index file %s: %m",
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	136 function, strmap->path);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	137 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	138
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	139 static void mail_index_strmap_close(struct mail_index_strmap *strmap)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	140 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	141 if (strmap->file_lock != NULL)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	142 file_lock_free(&strmap->file_lock);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	143 else if (strmap->dotlock != NULL)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	144 file_dotlock_delete(&strmap->dotlock);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	145
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	146 if (strmap->fd != -1) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	147 if (close(strmap->fd) < 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	148 mail_index_strmap_set_syscall_error(strmap, "close()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	149 strmap->fd = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	150 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	151 if (strmap->input != NULL)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	152 i_stream_unref(&strmap->input);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	153 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	154
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	155 void mail_index_strmap_deinit(struct mail_index_strmap **_strmap)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	156 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	157 struct mail_index_strmap strmap = _strmap;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	158
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	159 *_strmap = NULL;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	160 mail_index_strmap_close(strmap);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	161 i_free(strmap->path);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	162 i_free(strmap);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	163 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	164
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	165 static unsigned int mail_index_strmap_hash_key(const void *_key)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	166 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	167 const struct mail_index_strmap_hash_key *key = _key;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	168
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	169 return key->crc32;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	170 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	171
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	172 static bool
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	173 mail_index_strmap_hash_cmp(const void _key, const void _value, void *context)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	174 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	175 const struct mail_index_strmap_hash_key *key = _key;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	176 const struct mail_index_strmap_rec *rec = _value;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	177 struct mail_index_strmap_view *view = context;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	178
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	179 return view->key_compare(key->str, rec, view->cb_context);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	180 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	181
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	182 struct mail_index_strmap_view *
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	183 mail_index_strmap_view_open(struct mail_index_strmap *strmap,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	184 struct mail_index_view *idx_view,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	185 mail_index_strmap_key_cmp_t *key_compare_cb,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	186 mail_index_strmap_rec_cmp_t *rec_compare_cb,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	187 mail_index_strmap_remap_t *remap_cb,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	188 void *context,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	189 const ARRAY_TYPE(mail_index_strmap_rec) **recs_r,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	190 const struct hash2_table **hash_r)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	191 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	192 struct mail_index_strmap_view *view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	193
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	194 view = i_new(struct mail_index_strmap_view, 1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	195 view->strmap = strmap;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	196 view->view = idx_view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	197 view->key_compare = key_compare_cb;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	198 view->rec_compare = rec_compare_cb;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	199 view->remap_cb = remap_cb;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	200 view->cb_context = context;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	201 view->next_str_idx = 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	202
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	203 i_array_init(&view->recs, 64);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	204 i_array_init(&view->recs_crc32, 64);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	205 view->hash = hash2_create(0, sizeof(struct mail_index_strmap_rec),
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	206 mail_index_strmap_hash_key,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	207 mail_index_strmap_hash_cmp, view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	208 *recs_r = &view->recs;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	209 *hash_r = view->hash;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	210 return view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	211 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	212
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	213 void mail_index_strmap_view_close(struct mail_index_strmap_view **_view)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	214 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	215 struct mail_index_strmap_view view = _view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	216
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	217 *_view = NULL;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	218 array_free(&view->recs);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	219 array_free(&view->recs_crc32);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	220 hash2_destroy(&view->hash);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	221 i_free(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	222 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	223
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	224 uint32_t mail_index_strmap_view_get_highest_idx(struct mail_index_strmap_view *view)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	225 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	226 return view->next_str_idx-1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	227 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	228
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	229 static void mail_index_strmap_view_reset(struct mail_index_strmap_view *view)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	230 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	231 view->remap_cb(NULL, 0, 0, view->cb_context);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	232 array_clear(&view->recs);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	233 array_clear(&view->recs_crc32);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	234 hash2_clear(view->hash);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	235
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	236 view->last_added_uid = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	237 view->lost_expunged_uid = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	238 view->desynced = FALSE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	239 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	240
16689 e6c2a1344f86 lib-storage: If dovecot.index.thread corruption is noticed, delete the file. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	241 void mail_index_strmap_view_set_corrupted(struct mail_index_strmap_view *view)
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	242 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	243 mail_index_set_error(view->strmap->index,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	244 "Corrupted strmap index file: %s",
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	245 view->strmap->path);
19136 fefaa6d09a81 Replaced unlink() calls with i_unlink() wherever possible. Timo Sirainen <tss@iki.fi>* parents: 18137 diff changeset	246 i_unlink(view->strmap->path);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	247 mail_index_strmap_close(view->strmap);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	248 mail_index_strmap_view_reset(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	249 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	250
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	251 static int mail_index_strmap_open(struct mail_index_strmap_view *view)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	252 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	253 struct mail_index_strmap *strmap = view->strmap;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	254 const struct mail_index_header *idx_hdr;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	255 struct mail_index_strmap_header hdr;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	256 const unsigned char *data;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	257 size_t size;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	258 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	259
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	260 i_assert(strmap->fd == -1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	261
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	262 strmap->fd = open(strmap->path, O_RDWR);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	263 if (strmap->fd == -1) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	264 if (errno == ENOENT)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	265 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	266 mail_index_strmap_set_syscall_error(strmap, "open()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	267 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	268 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	269 strmap->input = i_stream_create_fd(strmap->fd, (size_t)-1, FALSE);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	270 ret = i_stream_read_data(strmap->input, &data, &size, sizeof(hdr)-1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	271 if (ret <= 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	272 if (ret < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	273 mail_index_strmap_set_syscall_error(strmap, "read()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	274 mail_index_strmap_close(strmap);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	275 } else {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	276 i_assert(ret == 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	277 mail_index_strmap_view_set_corrupted(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	278 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	279 return ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	280 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	281 memcpy(&hdr, data, sizeof(hdr));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	282
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	283 idx_hdr = mail_index_get_header(view->view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	284 if (hdr.version != MAIL_INDEX_STRMAP_VERSION \|\|
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	285 hdr.uid_validity != idx_hdr->uid_validity) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	286 /* need to rebuild. if we already had something in the strmap,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	287 we can keep it. */
19136 fefaa6d09a81 Replaced unlink() calls with i_unlink() wherever possible. Timo Sirainen <tss@iki.fi>* parents: 18137 diff changeset	288 i_unlink(strmap->path);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	289 mail_index_strmap_close(strmap);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	290 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	291 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	292
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	293 /* we'll read the entire file from the beginning */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	294 view->last_added_uid = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	295 view->last_read_uid = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	296 view->total_ref_count = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	297 view->last_read_block_offset = sizeof(struct mail_index_strmap_header);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	298 view->next_str_idx = 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	299
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	300 mail_index_strmap_view_reset(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	301 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	302 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	303
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	304 static bool mail_index_strmap_need_reopen(struct mail_index_strmap *strmap)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	305 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	306 struct stat st1, st2;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	307
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	308 /* FIXME: nfs flush */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	309 if (fstat(strmap->fd, &st1) < 0) {
12936 d14b0fd0a423 Linux NFS: fstat() may return ENOENT instead of ESTALE in some kernel versions. Timo Sirainen <tss@iki.fi> parents: 12782 diff changeset	310 if (!ESTALE_FSTAT(errno))
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	311 mail_index_strmap_set_syscall_error(strmap, "fstat()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	312 return TRUE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	313 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	314 if (stat(strmap->path, &st2) < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	315 mail_index_strmap_set_syscall_error(strmap, "stat()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	316 return TRUE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	317 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	318 return st1.st_ino != st2.st_ino \|\| !CMP_DEV_T(st1.st_dev, st2.st_dev);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	319 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	320
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	321 static int mail_index_strmap_refresh(struct mail_index_strmap_view *view)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	322 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	323 uint32_t seq;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	324
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	325 if (MAIL_INDEX_IS_IN_MEMORY(view->strmap->index))
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	326 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	327
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	328 if (view->strmap->fd != -1) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	329 if (!mail_index_strmap_need_reopen(view->strmap)) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	330 if (view->lost_expunged_uid != 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	331 /* last read failed because view had a message
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	332 that didn't exist in the strmap (because it
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	333 was expunged by another session). if the
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	334 message still isn't expunged in this view,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	335 just continue using the current strmap. */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	336 if (mail_index_lookup_seq(view->view,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	337 view->lost_expunged_uid, &seq))
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	338 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	339 } else if (view->desynced) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	340 /* our view isn't synced with the disk, we
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	341 can't read strmap without first resetting
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	342 the view */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	343 } else {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	344 i_stream_sync(view->strmap->input);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	345 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	346 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	347 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	348 mail_index_strmap_close(view->strmap);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	349 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	350
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	351 return mail_index_strmap_open(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	352 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	353
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	354 static int
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	355 mail_index_strmap_read_packed(struct mail_index_strmap_read_context *ctx,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	356 uint32_t *num_r)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	357 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	358 const unsigned char *data;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	359 const uint8_t bytes, p, *end;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	360 size_t size;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	361 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	362
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	363 ret = i_stream_read_data(ctx->input, &data, &size, sizeof(*num_r) - 1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	364 if (ret <= 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	365 return ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	366
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	367 if (ctx->input->v_offset + size > ctx->end_offset)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	368 size = ctx->end_offset - ctx->input->v_offset;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	369 bytes = p = (const uint8_t *)data;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	370 end = bytes + size;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	371
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	372 if (mail_index_unpack_num(&p, end, num_r) < 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	373 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	374 i_stream_skip(ctx->input, p - bytes);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	375 return 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	376 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	377
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	378 static int
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	379 mail_index_strmap_uid_exists(struct mail_index_strmap_read_context *ctx,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	380 uint32_t uid)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	381 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	382 const struct mail_index_record *rec;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	383
17568 d25443778208 lib-index: Recent idx->seq change in strmap forgot to initialize uid_lookup_seq in one place. Timo Sirainen <tss@iki.fi> parents: 17433 diff changeset	384 i_assert(ctx->uid_lookup_seq > 0);
d25443778208 lib-index: Recent idx->seq change in strmap forgot to initialize uid_lookup_seq in one place. Timo Sirainen <tss@iki.fi> parents: 17433 diff changeset	385
17433 2d5c66e8d77c mail-index: make uid_lookup_idx hold a sequence number, not a 0-based C array index Phil Carmody <phil@dovecot.fi> parents: 17351 diff changeset	386 if (ctx->uid_lookup_seq > ctx->view->view->map->hdr.messages_count) {
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	387 if (uid >= ctx->view->view->map->hdr.next_uid) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	388 /* thread index has larger UIDs than what we've seen
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	389 in our view. we'll have to read them again later
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	390 when we know about them */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	391 ctx->too_large_uids = TRUE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	392 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	393 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	394 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	395
17433 2d5c66e8d77c mail-index: make uid_lookup_idx hold a sequence number, not a 0-based C array index Phil Carmody <phil@dovecot.fi> parents: 17351 diff changeset	396 rec = MAIL_INDEX_REC_AT_SEQ(ctx->view->view->map, ctx->uid_lookup_seq);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	397 if (rec->uid == uid) {
17433 2d5c66e8d77c mail-index: make uid_lookup_idx hold a sequence number, not a 0-based C array index Phil Carmody <phil@dovecot.fi> parents: 17351 diff changeset	398 ctx->uid_lookup_seq++;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	399 return 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	400 } else if (rec->uid > uid) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	401 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	402 } else {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	403 /* record that exists in index is missing from strmap.
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	404 see if it's because the strmap is corrupted or because
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	405 our current view is a bit stale and the message has already
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	406 been expunged. */
14688 128c598d2870 Avoid using (void)s by adding ATTR_NOWARN_UNUSED_RESULT attributes and other ways. Timo Sirainen <tss@iki.fi> parents: 14686 diff changeset	407 mail_index_refresh(ctx->view->view->index);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	408 if (mail_index_is_expunged(ctx->view->view,
17433 2d5c66e8d77c mail-index: make uid_lookup_idx hold a sequence number, not a 0-based C array index Phil Carmody <phil@dovecot.fi> parents: 17351 diff changeset	409 ctx->uid_lookup_seq))
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	410 ctx->lost_expunged_uid = rec->uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	411 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	412 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	413 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	414
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	415 static int
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	416 mail_index_strmap_read_rec_first(struct mail_index_strmap_read_context *ctx,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	417 uint32_t *crc32_r)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	418 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	419 size_t size;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	420 uint32_t n, i, count, str_idx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	421 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	422
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	423 /* <uid> <n> <crc32>count <str_idx>count
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	424 where
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	425 n = 0 -> count=1 (only Message-ID:)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	426 n = 1 -> count=2 (Message-ID: + In-Reply-To:)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	427 n = 2+ -> count=n (Message-ID: + References:)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	428 */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	429 if (mail_index_strmap_read_packed(ctx, &n) <= 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	430 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	431 count = n < 2 ? n + 1 : n;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	432 ctx->view->total_ref_count += count;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	433
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	434 ctx->rec_size = count * (sizeof(ctx->rec.str_idx) + sizeof(*crc32_r));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	435 ret = mail_index_strmap_uid_exists(ctx, ctx->rec.uid);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	436 if (ret < 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	437 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	438 if (i_stream_read_data(ctx->view->strmap->input, &ctx->data, &size,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	439 ctx->rec_size - 1) <= 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	440 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	441 ctx->str_idx_base = ctx->data + count * sizeof(uint32_t);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	442
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	443 if (ret == 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	444 /* this message has already been expunged, ignore it.
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	445 update highest string indexes anyway. */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	446 for (i = 0; i < count; i++) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	447 memcpy(&str_idx, ctx->str_idx_base, sizeof(str_idx));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	448 if (ctx->highest_str_idx < str_idx)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	449 ctx->highest_str_idx = str_idx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	450 ctx->str_idx_base += sizeof(str_idx);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	451 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	452 i_stream_skip(ctx->view->strmap->input, ctx->rec_size);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	453 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	454 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	455
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	456 /* everything exists. save it. FIXME: these ref_index values
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	457 are thread index specific, perhaps something more generic
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	458 should be used some day */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	459 ctx->end = ctx->data + count * sizeof(*crc32_r);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	460
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	461 ctx->next_ref_index = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	462 if (!mail_index_strmap_read_rec_next(ctx, crc32_r))
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	463 i_unreached();
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	464 ctx->next_ref_index = n == 1 ? 1 : 2;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	465 return 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	466 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	467
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	468 static bool
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	469 mail_index_strmap_read_rec_next(struct mail_index_strmap_read_context *ctx,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	470 uint32_t *crc32_r)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	471 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	472 if (ctx->data == ctx->end) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	473 i_stream_skip(ctx->view->strmap->input, ctx->rec_size);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	474 return FALSE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	475 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	476
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	477 /* FIXME: str_idx could be stored as packed relative values
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	478 (first relative to highest_idx, the rest relative to the
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	479 previous str_idx) */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	480
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	481 /* read the record contents */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	482 memcpy(&ctx->rec.str_idx, ctx->str_idx_base, sizeof(ctx->rec.str_idx));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	483 memcpy(crc32_r, ctx->data, sizeof(*crc32_r));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	484
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	485 ctx->rec.ref_index = ctx->next_ref_index++;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	486
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	487 if (ctx->highest_str_idx < ctx->rec.str_idx)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	488 ctx->highest_str_idx = ctx->rec.str_idx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	489
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	490 /* get to the next record */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	491 ctx->data += sizeof(*crc32_r);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	492 ctx->str_idx_base += sizeof(ctx->rec.str_idx);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	493 return TRUE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	494 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	495
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	496 static int
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	497 strmap_read_block_init(struct mail_index_strmap_view *view,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	498 struct mail_index_strmap_read_context *ctx)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	499 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	500 struct mail_index_strmap *strmap = view->strmap;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	501 const unsigned char *data;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	502 size_t size;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	503 uint32_t block_size, seq1, seq2;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	504 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	505
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	506 if (view->last_read_uid + 1 >= view->view->map->hdr.next_uid) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	507 /* come back later when we know about the new UIDs */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	508 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	509 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	510
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20264 diff changeset	511 i_zero(ctx);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	512 ret = i_stream_read_data(strmap->input, &data, &size,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	513 sizeof(block_size)-1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	514 if (ret <= 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	515 if (strmap->input->stream_errno == 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	516 /* no new data */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	517 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	518 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	519 mail_index_strmap_set_syscall_error(strmap, "read()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	520 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	521 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	522 memcpy(&block_size, data, sizeof(block_size));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	523 block_size = mail_index_offset_to_uint32(block_size) >> 2;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	524 if (block_size == 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	525 /* the rest of the file is either not written, or the previous
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	526 write didn't finish */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	527 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	528 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	529 i_stream_skip(strmap->input, sizeof(block_size));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	530
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	531 ctx->view = view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	532 ctx->input = strmap->input;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	533 ctx->end_offset = strmap->input->v_offset + block_size;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	534 if (ctx->end_offset < strmap->input->v_offset) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	535 /* block size too large */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	536 mail_index_strmap_view_set_corrupted(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	537 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	538 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	539 ctx->rec.uid = view->last_read_uid + 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	540
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	541 /* FIXME: when reading multiple blocks we shouldn't have to calculate
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	542 this every time */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	543 if (!mail_index_lookup_seq_range(view->view, ctx->rec.uid, (uint32_t)-1,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	544 &seq1, &seq2))
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	545 seq1 = mail_index_view_get_messages_count(view->view) + 1;
17433 2d5c66e8d77c mail-index: make uid_lookup_idx hold a sequence number, not a 0-based C array index Phil Carmody <phil@dovecot.fi> parents: 17351 diff changeset	546 ctx->uid_lookup_seq = seq1;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	547 return 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	548 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	549
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	550 static int
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	551 strmap_read_block_next(struct mail_index_strmap_read_context *ctx,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	552 uint32_t *crc32_r)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	553 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	554 uint32_t uid_diff;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	555 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	556
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	557 if (mail_index_strmap_read_rec_next(ctx, crc32_r))
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	558 return 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	559
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	560 /* get next UID */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	561 do {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	562 if (ctx->input->v_offset == ctx->end_offset) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	563 /* this block is done */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	564 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	565 }
17340 0e8ddc1bfe8d lib-index: Fixed detecting broken strmap index file (thread index). Teemu Huovila <teemu.huovila@dovecot.fi> parents: 17130 diff changeset	566 if (mail_index_strmap_read_packed(ctx, &uid_diff) <= 0)
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	567 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	568
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	569 ctx->rec.uid += uid_diff;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	570 ret = mail_index_strmap_read_rec_first(ctx, crc32_r);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	571 } while (ret == 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	572 return ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	573 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	574
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	575 static int
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	576 strmap_read_block_deinit(struct mail_index_strmap_read_context *ctx, int ret,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	577 bool update_block_offset)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	578 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	579 struct mail_index_strmap_view *view = ctx->view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	580 struct mail_index_strmap *strmap = view->strmap;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	581
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	582 if (ctx->highest_str_idx > view->total_ref_count) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	583 /* if all string indexes are unique, highest_str_index equals
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	584 total_ref_count. otherwise it's always lower. */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	585 mail_index_set_error(strmap->index,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	586 "Corrupted strmap index file %s: "
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	587 "String indexes too high "
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	588 "(highest=%u max=%u)",
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	589 strmap->path, ctx->highest_str_idx,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	590 view->total_ref_count);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	591 mail_index_strmap_view_set_corrupted(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	592 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	593 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	594 if (ctx->lost_expunged_uid != 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	595 /* our view contained a message that had since been expunged. */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	596 i_assert(ret < 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	597 view->lost_expunged_uid = ctx->lost_expunged_uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	598 } else if (ret < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	599 if (strmap->input->stream_errno != 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	600 mail_index_strmap_set_syscall_error(strmap, "read()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	601 else
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	602 mail_index_strmap_view_set_corrupted(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	603 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	604 } else if (update_block_offset && !ctx->too_large_uids) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	605 view->last_read_block_offset = strmap->input->v_offset;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	606 view->last_read_uid = ctx->rec.uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	607 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	608 if (view->next_str_idx <= ctx->highest_str_idx)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	609 view->next_str_idx = ctx->highest_str_idx + 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	610 return ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	611 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	612
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	613 static bool
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	614 strmap_view_sync_handle_conflict(struct mail_index_strmap_read_context *ctx,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	615 const struct mail_index_strmap_rec *hash_rec,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	616 struct hash2_iter *iter)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	617 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	618 uint32_t seq;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	619
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	620 /* hopefully it's a message that has since been expunged */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	621 if (!mail_index_lookup_seq(ctx->view->view, hash_rec->uid, &seq)) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	622 /* message is no longer in our view. remove it completely. */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	623 hash2_remove_iter(ctx->view->hash, iter);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	624 return TRUE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	625 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	626 if (mail_index_is_expunged(ctx->view->view, seq)) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	627 /* it's quite likely a conflict. we may not be able to verify
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	628 it, so just assume it is. nothing breaks even if we guess
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	629 wrong, the performance just suffers a bit. */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	630 return FALSE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	631 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	632
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	633 /* 0 means "doesn't match", which is the only acceptable case */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	634 return ctx->view->rec_compare(&ctx->rec, hash_rec,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	635 ctx->view->cb_context) == 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	636 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	637
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	638 static int
8160 e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	639 strmap_view_sync_block_check_conflicts(struct mail_index_strmap_read_context *ctx,
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	640 uint32_t crc32)
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	641 {
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	642 struct mail_index_strmap_rec *hash_rec;
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	643 struct hash2_iter iter;
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	644
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	645 if (crc32 == 0) {
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	646 /* unique string - there are no conflicts */
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	647 return 0;
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	648 }
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	649
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	650 /* check for conflicting string indexes. they may happen if
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	651
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	652 1) msgid exists only for a message X that has been expunged
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	653 2) another process doesn't see X, but sees msgid for another
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	654 message and writes it using a new string index
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	655 3) if we still see X, we now see the same msgid with two
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	656 string indexes.
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	657
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	658 if we detect such a conflict, we can't continue using the
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	659 strmap index until X has been expunged. */
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20264 diff changeset	660 i_zero(&iter);
8160 e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	661 while ((hash_rec = hash2_iterate(ctx->view->hash,
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	662 crc32, &iter)) != NULL &&
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	663 hash_rec->str_idx != ctx->rec.str_idx) {
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	664 /* CRC32 matches, but string index doesn't */
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	665 if (!strmap_view_sync_handle_conflict(ctx, hash_rec, &iter)) {
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	666 ctx->lost_expunged_uid = hash_rec->uid;
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	667 return -1;
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	668 }
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	669 }
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	670 return 0;
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	671 }
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	672
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	673 static int
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	674 mail_index_strmap_view_sync_block(struct mail_index_strmap_read_context *ctx)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	675 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	676 struct mail_index_strmap_rec *hash_rec;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	677 uint32_t crc32, prev_uid = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	678 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	679
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	680 while ((ret = strmap_read_block_next(ctx, &crc32)) > 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	681 if (ctx->rec.uid <= ctx->view->last_added_uid) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	682 if (ctx->rec.uid < ctx->view->last_added_uid \|\|
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	683 prev_uid != ctx->rec.uid) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	684 /* we've already added this */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	685 continue;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	686 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	687 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	688 prev_uid = ctx->rec.uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	689
8160 e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	690 if (strmap_view_sync_block_check_conflicts(ctx, crc32) < 0) {
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	691 ret = -1;
e76d8777d20c strmap: Don't check conflicts for strings known to be unique. Timo Sirainen <tss@iki.fi> parents: 8155 diff changeset	692 break;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	693 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	694 ctx->view->last_added_uid = ctx->rec.uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	695
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	696 /* add the record to records array */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	697 array_append(&ctx->view->recs, &ctx->rec, 1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	698 array_append(&ctx->view->recs_crc32, &crc32, 1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	699
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	700 /* add a separate copy of the record to hash */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	701 hash_rec = hash2_insert_hash(ctx->view->hash, crc32);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	702 memcpy(hash_rec, &ctx->rec, sizeof(*hash_rec));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	703 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	704 return strmap_read_block_deinit(ctx, ret, TRUE);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	705 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	706
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	707 struct mail_index_strmap_view_sync *
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	708 mail_index_strmap_view_sync_init(struct mail_index_strmap_view *view,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	709 uint32_t *last_uid_r)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	710 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	711 struct mail_index_strmap_view_sync *sync;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	712 struct mail_index_strmap_read_context ctx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	713 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	714
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	715 sync = i_new(struct mail_index_strmap_view_sync, 1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	716 sync->view = view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	717
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	718 if (mail_index_strmap_refresh(view) < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	719 /* reading the strmap failed - just ignore and do
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	720 this in-memory based on whatever we knew last */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	721 } else if (view->strmap->input != NULL) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	722 i_stream_seek(view->strmap->input,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	723 view->last_read_block_offset);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	724 while ((ret = strmap_read_block_init(view, &ctx)) > 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	725 if (mail_index_strmap_view_sync_block(&ctx) < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	726 ret = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	727 break;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	728 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	729 if (ctx.too_large_uids)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	730 break;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	731 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	732
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	733 if (ret < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	734 /* something failed - we can still use the strmap as far
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	735 as we managed to read it, but our view is now out
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	736 of sync */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	737 view->desynced = TRUE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	738 } else {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	739 i_assert(view->lost_expunged_uid == 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	740 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	741 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	742 *last_uid_r = view->last_added_uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	743 return sync;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	744 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	745
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	746 static inline uint32_t crc32_str_nonzero(const char *str)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	747 {
10640 b87a15fc462a lib-index: Changed strmap's crc32 field to be backwards compatible. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	748 /* we'll flip the bits because of a bug in our old crc32 code.
b87a15fc462a lib-index: Changed strmap's crc32 field to be backwards compatible. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	749 this keeps the index format backwards compatible with the new fixed
b87a15fc462a lib-index: Changed strmap's crc32 field to be backwards compatible. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	750 crc32 code. */
b87a15fc462a lib-index: Changed strmap's crc32 field to be backwards compatible. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	751 uint32_t value = crc32_str(str) ^ 0xffffffffU;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	752 return value == 0 ? 1 : value;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	753 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	754
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	755 void mail_index_strmap_view_sync_add(struct mail_index_strmap_view_sync *sync,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	756 uint32_t uid, uint32_t ref_index,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	757 const char *key)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	758 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	759 struct mail_index_strmap_view *view = sync->view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	760 struct mail_index_strmap_rec rec, old_rec;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	761 struct mail_index_strmap_hash_key hash_key;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	762 uint32_t str_idx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	763
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	764 i_assert(uid > view->last_added_uid \|\|
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	765 (uid == view->last_added_uid &&
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	766 ref_index > view->last_ref_index));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	767
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	768 hash_key.str = key;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	769 hash_key.crc32 = crc32_str_nonzero(key);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	770
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	771 old_rec = hash2_lookup(view->hash, &hash_key);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	772 if (old_rec != NULL) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	773 /* The string already exists, use the same unique idx */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	774 str_idx = old_rec->str_idx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	775 } else {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	776 /* Newly seen string, assign a new unique idx to it */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	777 str_idx = view->next_str_idx++;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	778 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	779 i_assert(str_idx != 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	780
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	781 rec = hash2_insert(view->hash, &hash_key);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	782 rec->uid = uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	783 rec->ref_index = ref_index;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	784 rec->str_idx = str_idx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	785 array_append(&view->recs, rec, 1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	786 array_append(&view->recs_crc32, &hash_key.crc32, 1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	787
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	788 view->last_added_uid = uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	789 view->last_ref_index = ref_index;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	790 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	791
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	792 void mail_index_strmap_view_sync_add_unique(struct mail_index_strmap_view_sync *sync,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	793 uint32_t uid, uint32_t ref_index)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	794 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	795 struct mail_index_strmap_view *view = sync->view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	796 struct mail_index_strmap_rec rec;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	797
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	798 i_assert(uid > view->last_added_uid \|\|
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	799 (uid == view->last_added_uid &&
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	800 ref_index > view->last_ref_index));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	801
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20264 diff changeset	802 i_zero(&rec);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	803 rec.uid = uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	804 rec.ref_index = ref_index;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	805 rec.str_idx = view->next_str_idx++;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	806 array_append(&view->recs, &rec, 1);
14686 9ff19c1d5f69 Added array_append_zero() to write a zero-filled record to an array. Timo Sirainen <tss@iki.fi> parents: 14682 diff changeset	807 array_append_zero(&view->recs_crc32);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	808
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	809 view->last_added_uid = uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	810 view->last_ref_index = ref_index;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	811 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	812
8155 b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	813 static void
b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	814 mail_index_strmap_zero_terminate(struct mail_index_strmap_view *view)
b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	815 {
b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	816 /* zero-terminate the records array */
14686 9ff19c1d5f69 Added array_append_zero() to write a zero-filled record to an array. Timo Sirainen <tss@iki.fi> parents: 14682 diff changeset	817 array_append_zero(&view->recs);
8155 b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	818 array_delete(&view->recs, array_count(&view->recs)-1, 1);
b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	819 }
b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	820
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	821 static void mail_index_strmap_view_renumber(struct mail_index_strmap_view *view)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	822 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	823 struct mail_index_strmap_read_context ctx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	824 struct mail_index_strmap_rec recs, hash_rec;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	825 uint32_t prev_uid, str_idx, recs_crc32, renumber_map;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	826 unsigned int i, dest, count, count2;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	827 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	828
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20264 diff changeset	829 i_zero(&ctx);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	830 ctx.view = view;
17568 d25443778208 lib-index: Recent idx->seq change in strmap forgot to initialize uid_lookup_seq in one place. Timo Sirainen <tss@iki.fi> parents: 17433 diff changeset	831 ctx.uid_lookup_seq = 1;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	832
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	833 /* create a map of old -> new index and remove records of
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	834 expunged messages */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	835 renumber_map = i_new(uint32_t, view->next_str_idx);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	836 str_idx = 0; prev_uid = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	837 recs = array_get_modifiable(&view->recs, &count);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	838 recs_crc32 = array_get_modifiable(&view->recs_crc32, &count2);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	839 i_assert(count == count2);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	840
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	841 for (i = dest = 0; i < count; ) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	842 if (prev_uid != recs[i].uid) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	843 /* see if this record should be removed */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	844 prev_uid = recs[i].uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	845 ret = mail_index_strmap_uid_exists(&ctx, prev_uid);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	846 i_assert(ret >= 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	847 if (ret == 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	848 /* message expunged */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	849 do {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	850 i++;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	851 } while (i < count && recs[i].uid == prev_uid);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	852 continue;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	853 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	854 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	855
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	856 i_assert(recs[i].str_idx < view->next_str_idx);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	857 if (renumber_map[recs[i].str_idx] == 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	858 renumber_map[recs[i].str_idx] = ++str_idx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	859 if (i != dest) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	860 recs[dest] = recs[i];
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	861 recs_crc32[dest] = recs_crc32[i];
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	862 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	863 i++; dest++;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	864 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	865 i_assert(renumber_map[0] == 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	866 array_delete(&view->recs, dest, i-dest);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	867 array_delete(&view->recs_crc32, dest, i-dest);
8155 b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	868 mail_index_strmap_zero_terminate(view);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	869
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	870 /* notify caller of the renumbering */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	871 i_assert(str_idx <= view->next_str_idx);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	872 view->remap_cb(renumber_map, view->next_str_idx, str_idx + 1,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	873 view->cb_context);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	874
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	875 /* renumber the indexes in-place and recreate the hash */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	876 recs = array_get_modifiable(&view->recs, &count);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	877 hash2_clear(view->hash);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	878 for (i = 0; i < count; i++) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	879 recs[i].str_idx = renumber_map[recs[i].str_idx];
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	880 hash_rec = hash2_insert_hash(view->hash, recs_crc32[i]);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	881 memcpy(hash_rec, &recs[i], sizeof(*hash_rec));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	882 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	883
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	884 /* update the new next_str_idx only after remapping */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	885 view->next_str_idx = str_idx + 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	886 i_free(renumber_map);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	887 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	888
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	889 static void mail_index_strmap_write_block(struct mail_index_strmap_view *view,
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	890 struct ostream *output,
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	891 unsigned int i, uint32_t base_uid)
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	892 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	893 const struct mail_index_strmap_rec *recs;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	894 const uint32_t *crc32;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	895 unsigned int j, n, count, count2, uid_rec_count;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	896 uint32_t block_size;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	897 uint8_t p, packed[MAIL_INDEX_PACK_MAX_SIZE2];
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	898 uoff_t block_offset, end_offset;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	899
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	900 /* skip over the block size for now, we don't know it yet */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	901 block_offset = output->offset;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	902 block_size = 0;
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	903 o_stream_nsend(output, &block_size, sizeof(block_size));
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	904
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	905 /* write records */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	906 recs = array_get(&view->recs, &count);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	907 crc32 = array_get(&view->recs_crc32, &count2);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	908 i_assert(count == count2);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	909 while (i < count) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	910 /* @UNSAFE: <uid diff> */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	911 p = packed;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	912 mail_index_pack_num(&p, recs[i].uid - base_uid);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	913 base_uid = recs[i].uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	914
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	915 /* find how many records belong to this UID */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	916 uid_rec_count = 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	917 for (j = i + 1; j < count; j++) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	918 if (recs[j].uid != base_uid)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	919 break;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	920 uid_rec_count++;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	921 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	922 view->total_ref_count += uid_rec_count;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	923
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	924 /* <n> <crc32>count <str_idx>count -
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	925 FIXME: thread index specific code */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	926 i_assert(recs[i].ref_index == 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	927 if (uid_rec_count == 1) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	928 /* Only Message-ID: header */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	929 n = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	930 } else if (recs[i+1].ref_index == 1) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	931 /* In-Reply-To: header */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	932 n = 1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	933 i_assert(uid_rec_count == 2);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	934 } else {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	935 /* References: header */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	936 n = uid_rec_count;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	937 i_assert(recs[i+1].ref_index == 2);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	938 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	939
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	940 mail_index_pack_num(&p, n);
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	941 o_stream_nsend(output, packed, p-packed);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	942 for (j = 0; j < uid_rec_count; j++)
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	943 o_stream_nsend(output, &crc32[i+j], sizeof(crc32[i+j]));
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	944 for (j = 0; j < uid_rec_count; j++) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	945 i_assert(j < 2 \|\| recs[i+j].ref_index == j+1);
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	946 o_stream_nsend(output, &recs[i+j].str_idx,
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	947 sizeof(recs[i+j].str_idx));
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	948 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	949 i += uid_rec_count;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	950 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	951
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	952 /* we know the block size now - write it */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	953 block_size = output->offset - (block_offset + sizeof(block_size));
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	954 block_size = mail_index_uint32_to_offset(block_size << 2);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	955 i_assert(block_size != 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	956
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	957 end_offset = output->offset;
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14681 diff changeset	958 (void)o_stream_seek(output, block_offset);
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	959 o_stream_nsend(output, &block_size, sizeof(block_size));
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14681 diff changeset	960 (void)o_stream_seek(output, end_offset);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	961
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	962 if (output->last_failed_errno != 0)
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	963 return;
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	964
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	965 i_assert(view->last_added_uid == recs[count-1].uid);
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	966 view->last_read_uid = recs[count-1].uid;
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	967 view->last_read_block_offset = output->offset;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	968 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	969
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	970 static void
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	971 mail_index_strmap_recreate_write(struct mail_index_strmap_view *view,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	972 struct ostream *output)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	973 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	974 const struct mail_index_header *idx_hdr;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	975 struct mail_index_strmap_header hdr;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	976
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	977 idx_hdr = mail_index_get_header(view->view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	978
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	979 /* write header */
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20264 diff changeset	980 i_zero(&hdr);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	981 hdr.version = MAIL_INDEX_STRMAP_VERSION;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	982 hdr.uid_validity = idx_hdr->uid_validity;
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	983 o_stream_nsend(output, &hdr, sizeof(hdr));
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	984
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	985 view->total_ref_count = 0;
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	986 mail_index_strmap_write_block(view, output, 0, 1);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	987 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	988
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	989 static int mail_index_strmap_recreate(struct mail_index_strmap_view *view)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	990 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	991 struct mail_index_strmap *strmap = view->strmap;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	992 string_t *str;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	993 struct ostream *output;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	994 const char *temp_path;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	995 int fd, ret = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	996
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	997 if (array_count(&view->recs) == 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	998 /* everything expunged - just unlink the existing index */
8863 b61c15279b65 If unlinking strmap fails because it didn't exist to begin with, don't log an error. Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	999 if (unlink(strmap->path) < 0 && errno != ENOENT)
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1000 mail_index_strmap_set_syscall_error(strmap, "unlink()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1001 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1002 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1003
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1004 str = t_str_new(256);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1005 str_append(str, strmap->path);
9537 b373de4973cd Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9001 diff changeset	1006 fd = safe_mkstemp_hostpid_group(str, view->view->index->mode,
b373de4973cd Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9001 diff changeset	1007 view->view->index->gid,
b373de4973cd Whenever file's group changing fails, show the group origin in the error message. Timo Sirainen <tss@iki.fi> parents: 9001 diff changeset	1008 view->view->index->gid_origin);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1009 temp_path = str_c(str);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1010
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1011 if (fd == -1) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1012 mail_index_set_error(strmap->index,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1013 "safe_mkstemp_hostpid(%s) failed: %m",
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1014 temp_path);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1015 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1016 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1017 output = o_stream_create_fd(fd, 0, FALSE);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1018 o_stream_cork(output);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1019 mail_index_strmap_recreate_write(view, output);
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	1020 if (o_stream_nfinish(output) < 0) {
20264 0fd0fb05e2b3 global: Use [io]_stream_get_error() insted of %m Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	1021 mail_index_set_error(strmap->index, "write(%s) failed: %s",
0fd0fb05e2b3 global: Use [io]_stream_get_error() insted of %m Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	1022 temp_path, o_stream_get_error(output));
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1023 ret = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1024 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1025 o_stream_destroy(&output);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1026 if (close(fd) < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1027 mail_index_set_error(strmap->index,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1028 "close(%s) failed: %m", temp_path);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1029 ret = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1030 } else if (ret == 0 && rename(temp_path, strmap->path) < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1031 mail_index_set_error(strmap->index,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1032 "rename(%s, %s) failed: %m",
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1033 temp_path, strmap->path);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1034 ret = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1035 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1036 if (ret < 0)
19136 fefaa6d09a81 Replaced unlink() calls with i_unlink() wherever possible. Timo Sirainen <tss@iki.fi>* parents: 18137 diff changeset	1037 i_unlink(temp_path);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1038 return ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1039 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1040
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1041 static int mail_index_strmap_lock(struct mail_index_strmap *strmap)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1042 {
11623 c3ee4421e86a Added mail_max_lock_timeout setting. Timo Sirainen <tss@iki.fi> parents: 10781 diff changeset	1043 unsigned int timeout_secs;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1044 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1045
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1046 i_assert(strmap->fd != -1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1047
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1048 if (strmap->index->lock_method != FILE_LOCK_METHOD_DOTLOCK) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1049 i_assert(strmap->file_lock == NULL);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1050
11623 c3ee4421e86a Added mail_max_lock_timeout setting. Timo Sirainen <tss@iki.fi> parents: 10781 diff changeset	1051 timeout_secs = I_MIN(MAIL_INDEX_STRMAP_TIMEOUT_SECS,
c3ee4421e86a Added mail_max_lock_timeout setting. Timo Sirainen <tss@iki.fi> parents: 10781 diff changeset	1052 strmap->index->max_lock_timeout_secs);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1053 ret = file_wait_lock(strmap->fd, strmap->path, F_WRLCK,
11623 c3ee4421e86a Added mail_max_lock_timeout setting. Timo Sirainen <tss@iki.fi> parents: 10781 diff changeset	1054 strmap->index->lock_method, timeout_secs,
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1055 &strmap->file_lock);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1056 if (ret <= 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1057 mail_index_strmap_set_syscall_error(strmap,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1058 "file_wait_lock()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1059 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1060 } else {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1061 i_assert(strmap->dotlock == NULL);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1062
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1063 ret = file_dotlock_create(&strmap->dotlock_settings,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1064 strmap->path, 0, &strmap->dotlock);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1065 if (ret <= 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1066 mail_index_strmap_set_syscall_error(strmap,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1067 "file_dotlock_create()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1068 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1069 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1070 return ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1071 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1072
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1073 static void mail_index_strmap_unlock(struct mail_index_strmap *strmap)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1074 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1075 if (strmap->file_lock != NULL)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1076 file_unlock(&strmap->file_lock);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1077 else if (strmap->dotlock != NULL)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1078 file_dotlock_delete(&strmap->dotlock);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1079 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1080
9609 d404d34097f2 Use array_bsearch(). Timo Sirainen <tss@iki.fi>* parents: 9537 diff changeset	1081 static int
d404d34097f2 Use array_bsearch(). Timo Sirainen <tss@iki.fi>* parents: 9537 diff changeset	1082 strmap_rec_cmp(const uint32_t uid, const struct mail_index_strmap_rec rec)
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1083 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1084 return *uid < rec->uid ? -1 :
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1085 (*uid > rec->uid ? 1 : 0);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1086 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1087
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1088 static int
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1089 mail_index_strmap_write_append(struct mail_index_strmap_view *view)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1090 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1091 struct mail_index_strmap_read_context ctx;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1092 const struct mail_index_strmap_rec *old_recs;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1093 unsigned int i, old_count;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1094 struct ostream *output;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1095 uint32_t crc32, next_uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1096 bool full_block;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1097 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1098
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1099 /* Check first if another process had written new records to the file.
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1100 If there are any, hopefully they're the same as what we would be
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1101 writing. There are two problematic cases when messages have been
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1102 expunged recently:
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1103
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1104 1) The file contains UIDs that we don't have. This means the string
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1105 indexes won't be compatible anymore, so we'll have to renumber ours
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1106 to match the ones in the strmap file.
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1107
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1108 Currently we don't bother handling 1) case. If indexes don't match
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1109 what we have, we just don't write anything.
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1110
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1111 2) We have UIDs that don't exist in the file. We can't simply skip
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1112 those records, because other records may have pointers to them using
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1113 different string indexes than we have. Even if we renumbered those,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1114 future appends by other processes might cause the same problem (they
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1115 see the string for the first time and assign it a new index, but we
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1116 already have internally given it another index). So the only
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1117 sensible choice is to write nothing and hope that the message goes
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1118 away soon. */
9609 d404d34097f2 Use array_bsearch(). Timo Sirainen <tss@iki.fi>* parents: 9537 diff changeset	1119 next_uid = view->last_read_uid + 1;
d404d34097f2 Use array_bsearch(). Timo Sirainen <tss@iki.fi>* parents: 9537 diff changeset	1120 (void)array_bsearch_insert_pos(&view->recs, &next_uid,
d404d34097f2 Use array_bsearch(). Timo Sirainen <tss@iki.fi>* parents: 9537 diff changeset	1121 strmap_rec_cmp, &i);
d404d34097f2 Use array_bsearch(). Timo Sirainen <tss@iki.fi>* parents: 9537 diff changeset	1122
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1123 old_recs = array_get(&view->recs, &old_count);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1124 if (i < old_count) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1125 while (i > 0 && old_recs[i-1].uid == old_recs[i].uid)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1126 i--;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1127 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1128
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1129 i_stream_sync(view->strmap->input);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1130 i_stream_seek(view->strmap->input, view->last_read_block_offset);
8283 1e0b1888a876 uninitialized variable fix Timo Sirainen <tss@iki.fi> parents: 8160 diff changeset	1131 full_block = TRUE; ret = 0;
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1132 while (i < old_count &&
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1133 (ret = strmap_read_block_init(view, &ctx)) > 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1134 while ((ret = strmap_read_block_next(&ctx, &crc32)) > 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1135 if (ctx.rec.uid != old_recs[i].uid \|\|
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1136 ctx.rec.str_idx != old_recs[i].str_idx) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1137 /* mismatch */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1138 if (ctx.rec.uid > old_recs[i].uid) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1139 /* 1) case */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1140 ctx.lost_expunged_uid = ctx.rec.uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1141 } else if (ctx.rec.uid < old_recs[i].uid) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1142 /* 2) case */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1143 ctx.lost_expunged_uid = old_recs[i].uid;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1144 } else {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1145 /* string index mismatch,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1146 shouldn't happen */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1147 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1148 ret = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1149 break;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1150 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1151 if (++i == old_count) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1152 full_block = FALSE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1153 break;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1154 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1155 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1156 if (strmap_read_block_deinit(&ctx, ret, full_block) < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1157 ret = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1158 break;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1159 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1160 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1161 if (ret < 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1162 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1163 if (i == old_count) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1164 /* nothing new to write */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1165 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1166 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1167 i_assert(full_block);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1168 i_assert(old_recs[i].uid > view->last_read_uid);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1169
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1170 /* write the new records */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1171 output = o_stream_create_fd(view->strmap->fd, 0, FALSE);
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14681 diff changeset	1172 (void)o_stream_seek(output, view->last_read_block_offset);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1173 o_stream_cork(output);
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	1174 mail_index_strmap_write_block(view, output, i,
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	1175 view->last_read_uid + 1);
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14133 diff changeset	1176 if (o_stream_nfinish(output) < 0) {
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1177 mail_index_strmap_set_syscall_error(view->strmap, "write()");
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1178 ret = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1179 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1180 o_stream_destroy(&output);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1181 return ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1182 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1183
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1184 static int mail_index_strmap_write(struct mail_index_strmap_view *view)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1185 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1186 int ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1187
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1188 /* FIXME: this renumbering doesn't work well when running for a long
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1189 time since records aren't removed from hash often enough */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1190 if (STRIDX_MUST_RENUMBER(view->next_str_idx - 1,
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1191 hash2_count(view->hash))) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1192 mail_index_strmap_view_renumber(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1193 if (!MAIL_INDEX_IS_IN_MEMORY(view->strmap->index)) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1194 if (mail_index_strmap_recreate(view) < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1195 view->desynced = TRUE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1196 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1197 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1198 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1199 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1200 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1201
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1202 if (MAIL_INDEX_IS_IN_MEMORY(view->strmap->index) \|\| view->desynced)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1203 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1204
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1205 if (view->strmap->fd == -1) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1206 /* initial file creation */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1207 if (mail_index_strmap_recreate(view) < 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1208 view->desynced = TRUE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1209 return -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1210 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1211 return 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1212 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1213
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1214 /* append the new records to the strmap file */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1215 if (mail_index_strmap_lock(view->strmap) <= 0) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1216 /* timeout / error */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1217 ret = -1;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1218 } else if (mail_index_strmap_need_reopen(view->strmap)) {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1219 /* the file was already recreated - leave the syncing as it is
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1220 for now and let the next sync re-read the file. */
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1221 ret = 0;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1222 } else {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1223 ret = mail_index_strmap_write_append(view);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1224 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1225 mail_index_strmap_unlock(view->strmap);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1226 if (ret < 0)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1227 view->desynced = TRUE;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1228 return ret;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1229 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1230
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1231 void mail_index_strmap_view_sync_commit(struct mail_index_strmap_view_sync **_sync)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1232 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1233 struct mail_index_strmap_view_sync sync = _sync;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1234 struct mail_index_strmap_view *view = sync->view;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1235
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1236 *_sync = NULL;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1237 i_free(sync);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1238
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1239 (void)mail_index_strmap_write(view);
8155 b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	1240 mail_index_strmap_zero_terminate(view);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1241
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1242 /* zero-terminate the records array */
14686 9ff19c1d5f69 Added array_append_zero() to write a zero-filled record to an array. Timo Sirainen <tss@iki.fi> parents: 14682 diff changeset	1243 array_append_zero(&view->recs);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1244 array_delete(&view->recs, array_count(&view->recs)-1, 1);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1245 }
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1246
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1247 void mail_index_strmap_view_sync_rollback(struct mail_index_strmap_view_sync **_sync)
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1248 {
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1249 struct mail_index_strmap_view_sync sync = _sync;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1250
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1251 *_sync = NULL;
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1252
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1253 mail_index_strmap_view_reset(sync->view);
8155 b8feb139cf92 strmap: Make sure the records array is always zero-terminated. Timo Sirainen <tss@iki.fi> parents: 8146 diff changeset	1254 mail_index_strmap_zero_terminate(sync->view);
8146 70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1255 i_free(sync);
70b53e9b232e Rewrote thread indexing code. It's a lot simpler and takes less disk space. Timo Sirainen <tss@iki.fi> parents: diff changeset	1256 }

Mercurial > dovecot > core-2.2

annotate src/lib-index/mail-index-strmap.c @ 22576:707ae9de3812