dovecot/original-hg/dovecot-1.2: src/lib-index/mailbox-list-index-sync.c annotate

annotate src/lib-index/mailbox-list-index-sync.c @ 6075:813c976ed476 HEAD

Handle desync errors without marking mailbox list index corrupted, the problem is with mail index.

author	Timo Sirainen <tss@iki.fi>
date	Wed, 18 Jul 2007 08:37:30 +0300
parents	461496644c67
children	1f0e71c833c3

rev	line source
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	1 /* Copyright (C) 2006 Timo Sirainen */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	2
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	3 #include "lib.h"
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	4 #include "array.h"
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	5 #include "bsearch-insert-pos.h"
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	6 #include "crc32.h"
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	7 #include "file-cache.h"
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	8 #include "file-set-size.h"
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	9 #include "mmap-util.h"
5983 272796cfee66 compiler warning fixes Timo Sirainen <tss@iki.fi> parents: 5972 diff changeset	10 #include "nfs-workarounds.h"
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	11 #include "ostream.h"
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	12 #include "mail-index-private.h"
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	13 #include "mailbox-list-index-private.h"
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	14
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	15 #include <stddef.h>
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	16
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	17 #define ROOT_INIT_COUNT 128
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	18 #define DIR_ALLOC_MORE_COUNT 4
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	19 #define MAILBOX_LIST_INDEX_GROW_PERCENTAGE 10
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	20 #define MAILBOX_LIST_INDEX_MIN_SIZE 512
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	21
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	22 struct mailbox_list_sync_record {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	23 uint32_t name_hash;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	24 uint32_t seq;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	25 uint32_t uid;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	26 const char *name;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	27
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	28 /* dir is used if it's non-NULL, otherwise dir_offset is used */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	29 struct mailbox_list_sync_dir *dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	30 uint32_t dir_offset;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	31
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	32 uint32_t created:1;
6057 4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	33 /* This record was seen while syncing, either as parent or as leaf */
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	34 uint32_t seen:1;
6057 4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	35 /* This record was seen as leaf while syncing, so it exists */
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	36 uint32_t exists:1;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	37 };
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	38
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	39 struct mailbox_list_sync_dir {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	40 /* The records are sorted by their name_hash */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	41 ARRAY_DEFINE(records, struct mailbox_list_sync_record);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	42
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	43 /* Offset to the original location in the index, or 0 for new dirs */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	44 uint32_t offset;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	45 unsigned int seen_records_count;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	46 unsigned int new_records_count;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	47 };
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	48
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	49 struct mailbox_list_index_sync_ctx {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	50 struct mailbox_list_index *index;
5324 4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	51 struct mailbox_list_index_view *view;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	52 pool_t pool;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	53
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	54 enum mailbox_list_sync_flags flags;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	55 const char *sync_path;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	56 struct mail_index_sync_ctx *mail_sync_ctx;
5324 4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	57 struct mail_index_view *mail_view;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	58 struct mail_index_transaction *trans;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	59
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	60 struct mailbox_list_index_header hdr;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	61 struct mailbox_list_sync_dir root, sync_root;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	62
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	63 struct ostream *output;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	64 buffer_t *output_buf;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	65
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	66 unsigned int failed:1;
6075 813c976ed476 Handle desync errors without marking mailbox list index corrupted, the Timo Sirainen <tss@iki.fi> parents: 6072 diff changeset	67 unsigned int restart:1;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	68 unsigned int partial:1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	69 unsigned int seen_sync_root:1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	70 };
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	71
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	72 struct mailbox_list_sync_lookup_key {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	73 uint32_t name_hash;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	74 const char *name;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	75 };
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	76
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	77 static bool mailbox_list_index_need_compress(struct mailbox_list_index *index);
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	78 static int mailbox_list_index_compress(struct mailbox_list_index_sync_ctx *ctx);
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	79
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	80 static struct mailbox_list_sync_dir *
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	81 mailbox_list_alloc_sync_dir(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	82 unsigned int initial_count)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	83 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	84 struct mailbox_list_sync_dir *sync_dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	85
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	86 sync_dir = p_new(ctx->pool, struct mailbox_list_sync_dir, 1);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	87 p_array_init(&sync_dir->records, ctx->pool, initial_count);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	88 return sync_dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	89 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	90
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	91 static int
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	92 mailbox_list_copy_sync_dir(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	93 uint32_t offset,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	94 struct mailbox_list_sync_dir **sync_dir_r)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	95 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	96 const struct mailbox_list_dir_record *dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	97 const struct mailbox_list_record *recs;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	98 struct mailbox_list_sync_dir *sync_dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	99 struct mailbox_list_sync_record *sync_rec;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	100 const char *name;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	101 size_t max_len;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	102 unsigned int i;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	103
5324 4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	104 if (mailbox_list_index_get_dir(ctx->view, &offset, &dir) < 0)
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	105 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	106
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	107 sync_dir = mailbox_list_alloc_sync_dir(ctx, dir->count +
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	108 DIR_ALLOC_MORE_COUNT);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	109 sync_dir->offset = offset;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	110
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	111 recs = MAILBOX_LIST_RECORDS(dir);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	112 for (i = 0; i < dir->count; i++) {
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	113 if (recs[i].deleted)
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	114 continue;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	115
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	116 if (recs[i].uid == 0) {
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	117 return mailbox_list_index_set_corrupted(ctx->index,
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	118 "Record with UID=0");
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	119 }
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	120
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	121 sync_rec = array_append_space(&sync_dir->records);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	122 sync_rec->name_hash = recs[i].name_hash;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	123 sync_rec->uid = recs[i].uid;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	124 sync_rec->dir_offset =
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	125 mail_index_offset_to_uint32(recs[i].dir_offset);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	126
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	127 max_len = ctx->index->mmap_size - recs[i].name_offset;
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	128 name = CONST_PTR_OFFSET(ctx->index->const_mmap_base,
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	129 recs[i].name_offset);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	130
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	131 sync_rec->name = p_strndup(ctx->pool, name, max_len);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	132 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	133
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	134 *sync_dir_r = sync_dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	135 return 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	136 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	137
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	138 static int mailbox_list_sync_record_cmp(const void _key, const void _rec)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	139 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	140 const struct mailbox_list_sync_lookup_key *key = _key;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	141 const struct mailbox_list_sync_record *rec = _rec;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	142
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	143 if (key->name_hash < rec->name_hash)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	144 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	145 if (key->name_hash > rec->name_hash)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	146 return 1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	147
5336 109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	148 return strcmp(key->name, rec->name);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	149 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	150
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	151 static struct mailbox_list_sync_record *
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	152 mailbox_list_sync_dir_lookup(struct mailbox_list_sync_dir *dir,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	153 const char name, unsigned int idx_r)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	154 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	155 struct mailbox_list_sync_lookup_key key;
5336 109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	156 struct mailbox_list_sync_record *recs;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	157 unsigned int count;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	158 bool match;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	159
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	160 /* binary search the current hierarchy level name. the values are
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	161 sorted primarily by their hash value and secondarily by the actual
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	162 name */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	163 key.name = name;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	164 key.name_hash = crc32_str(name);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	165
5336 109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	166 recs = array_get_modifiable(&dir->records, &count);
109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	167 match = bsearch_insert_pos(&key, recs, count, sizeof(*recs),
109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	168 mailbox_list_sync_record_cmp,
109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	169 idx_r);
109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	170 if (!match)
109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	171 return NULL;
109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	172
109ca861405f bsearch_insert_pos() API changed. Patch by Max Kellermann Timo Sirainen <tss@iki.fi> parents: 5324 diff changeset	173 return &recs[*idx_r];
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	174 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	175
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	176 static struct mailbox_list_sync_record *
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	177 mailbox_list_alloc_add_record(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	178 struct mailbox_list_sync_dir *dir,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	179 const char *name, unsigned int idx)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	180 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	181 struct mailbox_list_sync_record *rec;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	182
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	183 rec = array_insert_space(&dir->records, idx);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	184 rec->name_hash = crc32_str(name);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	185 rec->name = p_strdup(ctx->pool, name);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	186 rec->uid = ctx->hdr.next_uid++;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	187 rec->created = TRUE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	188 mail_index_append(ctx->trans, rec->uid, &rec->seq);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	189
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	190 dir->new_records_count++;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	191 return rec;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	192 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	193
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	194 static int
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	195 mailbox_list_index_sync_get_seq(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	196 struct mailbox_list_sync_record *rec)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	197 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	198 if (rec->uid == 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	199 return mailbox_list_index_set_corrupted(ctx->index,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	200 "Record with UID=0");
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	201 }
5324 4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	202 if (mail_index_lookup_uid_range(ctx->mail_view, rec->uid, rec->uid,
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	203 &rec->seq, &rec->seq) < 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	204 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	205
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	206 if (rec->seq == 0) {
6075 813c976ed476 Handle desync errors without marking mailbox list index corrupted, the Timo Sirainen <tss@iki.fi> parents: 6072 diff changeset	207 i_warning("%s: Desync: Record uid=%u expunged from mail index",
813c976ed476 Handle desync errors without marking mailbox list index corrupted, the Timo Sirainen <tss@iki.fi> parents: 6072 diff changeset	208 ctx->index->mail_index->filepath, rec->uid);
813c976ed476 Handle desync errors without marking mailbox list index corrupted, the Timo Sirainen <tss@iki.fi> parents: 6072 diff changeset	209 ctx->restart = TRUE;
813c976ed476 Handle desync errors without marking mailbox list index corrupted, the Timo Sirainen <tss@iki.fi> parents: 6072 diff changeset	210 return -1;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	211 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	212 return 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	213 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	214
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	215 static int
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	216 mailbox_list_index_sync_int(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	217 const char *name,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	218 struct mailbox_list_sync_dir **dir_r,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	219 uint32_t *seq_r)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	220 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	221 const char p, hier_name;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	222 struct mailbox_list_sync_dir *dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	223 struct mailbox_list_sync_record *rec = NULL;
5560 3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	224 unsigned int idx, child_flags;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	225
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	226 if (ctx->failed)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	227 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	228
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	229 dir = ctx->sync_root;
5560 3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	230 child_flags = MAILBOX_LIST_INDEX_FLAG_CHILDREN;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	231
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	232 t_push();
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	233 for (;;) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	234 p = strchr(name, ctx->index->separator);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	235 hier_name = p == NULL ? name : t_strdup_until(name, p);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	236
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	237 if (*hier_name == '\0') {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	238 if (p == NULL) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	239 /* name ended with a separator */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	240 break;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	241 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	242 /* two separators adjacently, skip this */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	243 name = p + 1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	244 continue;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	245 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	246
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	247 if (rec != NULL) {
5560 3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	248 /* add CHILDREN flag to the parent and remove
3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	249 NOCHILDREN flag. if we happened to create the
3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	250 parent ourself, also add NONEXISTENT flag. */
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	251 mail_index_update_flags(ctx->trans, rec->seq,
5560 3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	252 MODIFY_ADD, child_flags);
3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	253 mail_index_update_flags(ctx->trans, rec->seq,
3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	254 MODIFY_REMOVE,
3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	255 MAILBOX_LIST_INDEX_FLAG_NOCHILDREN);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	256 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	257
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	258 rec = mailbox_list_sync_dir_lookup(dir, hier_name, &idx);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	259 if (rec == NULL) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	260 /* new record */
5560 3a3051d0e79b Mailbox list index fixes Timo Sirainen <tss@iki.fi> parents: 5345 diff changeset	261 child_flags \|= MAILBOX_LIST_INDEX_FLAG_NONEXISTENT;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	262 rec = mailbox_list_alloc_add_record(ctx, dir,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	263 hier_name, idx);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	264 } else if (rec->seq == 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	265 /* this record was copied from existing index.
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	266 the uid is known, but the sequence isn't. */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	267 if (mailbox_list_index_sync_get_seq(ctx, rec) < 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	268 ctx->failed = TRUE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	269 break;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	270 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	271 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	272 *seq_r = rec->seq;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	273
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	274 /* remember that we've seen this record */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	275 if (!rec->seen) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	276 rec->seen = TRUE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	277 dir->seen_records_count++;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	278 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	279
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	280 if (p == NULL) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	281 /* leaf */
6057 4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	282 rec->exists = TRUE;
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	283 mail_index_update_flags(ctx->trans, rec->seq,
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	284 MODIFY_REMOVE,
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	285 MAILBOX_LIST_INDEX_FLAG_NONEXISTENT);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	286 break;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	287 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	288
6057 4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	289 /* we were handling a parent, continue with its child */
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	290 if (!rec->exists) {
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	291 /* Mark this mailbox non-existent for now. If it
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	292 exists, this flag is removed later. */
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	293 mail_index_update_flags(ctx->trans, rec->seq,
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	294 MODIFY_ADD,
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	295 MAILBOX_LIST_INDEX_FLAG_NONEXISTENT);
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	296 }
4d14254bc40a Handle setting deleted mailboxes to nonexistent. Timo Sirainen <tss@iki.fi> parents: 5983 diff changeset	297
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	298 if (rec->dir == NULL) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	299 if (rec->dir_offset != 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	300 if (mailbox_list_copy_sync_dir(ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	301 rec->dir_offset,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	302 &rec->dir) < 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	303 ctx->failed = TRUE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	304 break;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	305 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	306 } else {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	307 rec->dir = mailbox_list_alloc_sync_dir(ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	308 1 + DIR_ALLOC_MORE_COUNT);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	309 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	310 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	311
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	312 name = p + 1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	313 dir = rec->dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	314 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	315 t_pop();
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	316
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	317 i_assert(dir != NULL);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	318 *dir_r = dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	319 return ctx->failed ? -1 : 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	320 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	321
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	322 static int mailbox_list_index_get_root(struct mailbox_list_index_sync_ctx *ctx)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	323 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	324 uint32_t seq;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	325
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	326 i_assert(ctx->index->mmap_size > 0);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	327
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	328 if (ctx->index->mmap_size == sizeof(*ctx->index->hdr)) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	329 /* root doesn't exist in the file yet */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	330 ctx->root = mailbox_list_alloc_sync_dir(ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	331 ROOT_INIT_COUNT);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	332 } else {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	333 if (mailbox_list_copy_sync_dir(ctx, sizeof(*ctx->index->hdr),
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	334 &ctx->root) < 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	335 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	336 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	337
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	338 /* keep sync_root=root until we've built the sync_root path. */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	339 ctx->sync_root = ctx->root;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	340
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	341 if (*ctx->sync_path != '\0') {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	342 if (mailbox_list_index_sync_more(ctx, ctx->sync_path, &seq) < 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	343 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	344 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	345
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	346 return mailbox_list_index_sync_int(ctx, ctx->sync_path,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	347 &ctx->sync_root, &seq);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	348 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	349
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	350 static int sync_mail_sync_init(struct mailbox_list_index_sync_ctx *ctx)
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	351 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	352 struct mail_index_sync_rec sync_rec;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	353
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	354 if (mail_index_sync_begin(ctx->index->mail_index, &ctx->mail_sync_ctx,
5643 453128e12b11 mail_index_sync_begin() returns now transaction directly so the syncing code Timo Sirainen <tss@iki.fi> parents: 5560 diff changeset	355 &ctx->mail_view, &ctx->trans, (uint32_t)-1, 0,
5779 b25c9ca2142b mail_index_sync_begin() takes now flags parameter instead of two booleans. Timo Sirainen <tss@iki.fi> parents: 5643 diff changeset	356 0) < 0)
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	357 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	358
5972 9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	359 if (ctx->index->mail_index->nfs_flush) {
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	360 nfs_flush_read_cache(ctx->index->filepath, ctx->index->fd,
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	361 F_UNLCK, FALSE);
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	362 }
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	363
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	364 /* we should have only external transactions in here, for which we
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	365 don't need to do anything but write them to the index */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	366 while (mail_index_sync_next(ctx->mail_sync_ctx, &sync_rec) > 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	367 ;
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	368 return 0;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	369 }
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	370
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	371 static int sync_mail_sync_init2(struct mailbox_list_index_sync_ctx *ctx)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	372 {
6072 461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	373 const struct mail_index_header *mail_hdr;
461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	374 uint32_t uid_validity;
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	375
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	376 ctx->hdr = *ctx->index->hdr;
6072 461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	377 mail_hdr = mail_index_get_header(ctx->mail_view);
461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	378 uid_validity = mail_hdr->uid_validity;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	379
6072 461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	380 if (uid_validity != 0 \|\| mail_hdr->next_uid != 1) {
461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	381 if (uid_validity != ctx->hdr.uid_validity) {
461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	382 i_warning("%s: Desync: uid_validity changed %u -> %u",
461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	383 ctx->index->mail_index->filepath,
461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	384 uid_validity, ctx->hdr.uid_validity);
461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	385 uid_validity = 0;
461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	386 mail_index_reset(ctx->trans);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	387 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	388 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	389
6072 461496644c67 Handle uid validity changes a bit better. Timo Sirainen <tss@iki.fi> parents: 6069 diff changeset	390 if (uid_validity != ctx->hdr.uid_validity ) {
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	391 mail_index_update_header(ctx->trans,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	392 offsetof(struct mail_index_header, uid_validity),
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	393 &ctx->hdr.uid_validity, sizeof(ctx->hdr.uid_validity),
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	394 TRUE);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	395 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	396
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	397 return 0;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	398 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	399
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	400 int mailbox_list_index_sync_init(struct mailbox_list_index *index,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	401 const char *path,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	402 enum mailbox_list_sync_flags flags,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	403 struct mailbox_list_index_sync_ctx **ctx_r)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	404 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	405 struct mailbox_list_index_sync_ctx *ctx;
6069 8576ce5d7fb4 Changed mailbox_list_index_view_init() to return -1 if uid_validity doesn't Timo Sirainen <tss@iki.fi> parents: 6057 diff changeset	406 struct mailbox_list_index_view *view;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	407 pool_t pool;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	408 size_t len;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	409
6069 8576ce5d7fb4 Changed mailbox_list_index_view_init() to return -1 if uid_validity doesn't Timo Sirainen <tss@iki.fi> parents: 6057 diff changeset	410 if (mailbox_list_index_view_init(index, NULL, &view) < 0)
8576ce5d7fb4 Changed mailbox_list_index_view_init() to return -1 if uid_validity doesn't Timo Sirainen <tss@iki.fi> parents: 6057 diff changeset	411 return -1;
8576ce5d7fb4 Changed mailbox_list_index_view_init() to return -1 if uid_validity doesn't Timo Sirainen <tss@iki.fi> parents: 6057 diff changeset	412
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	413 /* add separator to end of path if it isn't there */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	414 len = strlen(path);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	415 if (len > 0 && path[len-1] != index->separator)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	416 path = t_strdup_printf("%s%c", path, index->separator);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	417
5315 7d0469858e6d Marked memory pools growing Timo Sirainen <tss@iki.fi> parents: 5313 diff changeset	418 pool = pool_alloconly_create(MEMPOOL_GROWING"mailbox list index sync",
7d0469858e6d Marked memory pools growing Timo Sirainen <tss@iki.fi> parents: 5313 diff changeset	419 1024*32);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	420
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	421 ctx = p_new(pool, struct mailbox_list_index_sync_ctx, 1);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	422 ctx->pool = pool;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	423 ctx->index = index;
6069 8576ce5d7fb4 Changed mailbox_list_index_view_init() to return -1 if uid_validity doesn't Timo Sirainen <tss@iki.fi> parents: 6057 diff changeset	424 ctx->view = view;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	425 ctx->sync_path = p_strdup(pool, path);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	426 ctx->flags = flags;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	427
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	428 /* mail index syncing acts as the only locking for us */
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	429 if (sync_mail_sync_init(ctx) < 0 \|\|
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	430 mailbox_list_index_refresh(index) < 0 \|\|
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	431 sync_mail_sync_init2(ctx) < 0 \|\|
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	432 mailbox_list_index_get_root(ctx) < 0) {
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	433 mailbox_list_index_sync_rollback(&ctx);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	434 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	435 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	436
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	437 *ctx_r = ctx;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	438 return 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	439 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	440
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	441 struct mail_index_view *
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	442 mailbox_list_index_sync_get_view(struct mailbox_list_index_sync_ctx *ctx)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	443 {
5324 4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	444 return ctx->mail_view;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	445 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	446
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	447 struct mail_index_transaction *
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	448 mailbox_list_index_sync_get_transaction(struct mailbox_list_index_sync_ctx *ctx)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	449 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	450 return ctx->trans;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	451 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	452
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	453 int mailbox_list_index_sync_more(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	454 const char name, uint32_t seq_r)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	455 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	456 struct mailbox_list_sync_dir *dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	457
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	458 return mailbox_list_index_sync_int(ctx, name, &dir, seq_r);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	459 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	460
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	461 static int
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	462 mailbox_list_index_sync_grow(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	463 uint32_t size)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	464 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	465 struct mailbox_list_index *index = ctx->index;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	466 uoff_t new_fsize, grow_size;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	467
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	468 new_fsize = ctx->hdr.used_space + size;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	469 grow_size = new_fsize / 100 * MAILBOX_LIST_INDEX_GROW_PERCENTAGE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	470 if (grow_size < MAILBOX_LIST_INDEX_MIN_SIZE)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	471 grow_size = MAILBOX_LIST_INDEX_MIN_SIZE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	472 new_fsize += grow_size;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	473 new_fsize &= ~(512-1);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	474
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	475 i_assert(new_fsize >= ctx->hdr.used_space + size);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	476
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	477 if (file_set_size(index->fd, (off_t)new_fsize) < 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	478 mailbox_list_index_set_syscall_error(index, "file_set_size()");
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	479 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	480 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	481
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	482 return mailbox_list_index_map(index);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	483 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	484
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	485 static int
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	486 mailbox_list_index_sync_alloc_space(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	487 uint32_t size, void **base_r,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	488 uint32_t *base_offset_r)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	489 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	490 size_t pos = ctx->hdr.used_space;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	491
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	492 /* all allocations must be 32bit aligned */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	493 pos = (pos + 3) & ~3;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	494
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	495 if (ctx->index->mmap_disable) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	496 /* write the data into temporary buffer first */
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	497 buffer_set_used_size(ctx->output_buf, 0);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	498
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	499 if (ctx->output->offset != pos) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	500 i_assert(pos > ctx->output->offset &&
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	501 pos - ctx->output->offset <= 3);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	502
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	503 buffer_append_zero(ctx->output_buf,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	504 pos - ctx->output->offset);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	505 }
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	506
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	507 *base_r = buffer_append_space_unsafe(ctx->output_buf, size);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	508 } else {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	509 if (pos + size > ctx->index->mmap_size) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	510 if (mailbox_list_index_sync_grow(ctx, size + 3) < 0)
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	511 return -1;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	512
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	513 i_assert(pos + size < ctx->index->mmap_size);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	514 }
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	515 *base_r = PTR_OFFSET(ctx->index->mmap_base, pos);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	516 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	517
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	518 *base_offset_r = pos;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	519 ctx->hdr.used_space = pos + size;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	520 return 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	521 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	522
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	523 static int
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	524 mailbox_list_index_sync_recreate_dir(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	525 struct mailbox_list_sync_dir *sync_dir,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	526 uint32_t offset_pos, bool partial)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	527 {
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	528 struct mailbox_list_index *index = ctx->index;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	529 const struct mailbox_list_dir_record *dir;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	530 const struct mailbox_list_record *recs;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	531 struct mailbox_list_dir_record *new_dir;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	532 struct mailbox_list_record *new_recs;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	533 struct mailbox_list_sync_record *sync_recs;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	534 unsigned int src, dest, orig, count, nondeleted_count;
5343 cdf39128dd91 More desyncing fixes Timo Sirainen <tss@iki.fi> parents: 5336 diff changeset	535 unsigned int space_needed, deleted_space;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	536 uint32_t base_offset, name_pos, size;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	537 void *base;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	538
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	539 i_assert((offset_pos % sizeof(uint32_t)) == 0);
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	540 i_assert(offset_pos < index->mmap_size);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	541
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	542 /* count how much space we need and how much we wasted for deleted
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	543 records */
5345 063c892d0439 deleted_space wasn't being calculated, so the index was never compressed. Timo Sirainen <tss@iki.fi> parents: 5344 diff changeset	544 nondeleted_count = 0; space_needed = 0;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	545 sync_recs = array_get_modifiable(&sync_dir->records, &count);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	546 for (src = 0; src < count; src++) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	547 if (sync_recs[src].seen \|\| partial) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	548 nondeleted_count++;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	549 if (sync_recs[src].created) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	550 /* new record */
5343 cdf39128dd91 More desyncing fixes Timo Sirainen <tss@iki.fi> parents: 5336 diff changeset	551 space_needed += strlen(sync_recs[src].name) + 1;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	552 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	553 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	554 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	555
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	556 /* @UNSAFE */
5343 cdf39128dd91 More desyncing fixes Timo Sirainen <tss@iki.fi> parents: 5336 diff changeset	557 space_needed += sizeof(dir) + nondeleted_count sizeof(*new_recs);
cdf39128dd91 More desyncing fixes Timo Sirainen <tss@iki.fi> parents: 5336 diff changeset	558 if (mailbox_list_index_sync_alloc_space(ctx, space_needed,
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	559 &base, &base_offset) < 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	560 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	561 /* NOTE: any pointers to the index file may have been invalidated
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	562 as a result of growing the the memory area */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	563
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	564 if (sync_dir->offset == 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	565 dir = NULL;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	566 recs = NULL;
5345 063c892d0439 deleted_space wasn't being calculated, so the index was never compressed. Timo Sirainen <tss@iki.fi> parents: 5344 diff changeset	567 deleted_space = 0;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	568 } else {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	569 /* the offset should have been verified already to be valid */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	570 i_assert(sync_dir->offset == offset_pos);
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	571 i_assert(sync_dir->offset < index->mmap_size);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	572 dir = CONST_PTR_OFFSET(index->const_mmap_base,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	573 sync_dir->offset);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	574 recs = MAILBOX_LIST_RECORDS(dir);
5345 063c892d0439 deleted_space wasn't being calculated, so the index was never compressed. Timo Sirainen <tss@iki.fi> parents: 5344 diff changeset	575
063c892d0439 deleted_space wasn't being calculated, so the index was never compressed. Timo Sirainen <tss@iki.fi> parents: 5344 diff changeset	576 /* approximate deleted_space. some of the mailbox names will be
063c892d0439 deleted_space wasn't being calculated, so the index was never compressed. Timo Sirainen <tss@iki.fi> parents: 5344 diff changeset	577 reused, but don't bother calculating them. */
063c892d0439 deleted_space wasn't being calculated, so the index was never compressed. Timo Sirainen <tss@iki.fi> parents: 5344 diff changeset	578 deleted_space = sizeof(*dir) + dir->dir_size;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	579 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	580
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	581 new_dir = base;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	582 new_dir->count = nondeleted_count;
5343 cdf39128dd91 More desyncing fixes Timo Sirainen <tss@iki.fi> parents: 5336 diff changeset	583 new_dir->dir_size = space_needed;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	584
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	585 new_recs = MAILBOX_LIST_RECORDS_MODIFIABLE(new_dir);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	586 name_pos = (const char *)(new_recs + nondeleted_count) -
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	587 (const char *)base;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	588 for (src = dest = 0; src < count; src++) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	589 if (!sync_recs[src].seen && !partial) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	590 /* expunge from mail index */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	591 uint32_t seq;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	592
5324 4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	593 if (mail_index_lookup_uid_range(ctx->mail_view,
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	594 sync_recs[src].uid,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	595 sync_recs[src].uid,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	596 &seq, &seq) < 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	597 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	598
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	599 if (seq != 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	600 mail_index_expunge(ctx->trans, seq);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	601 // FIXME: expunge also NONEXISTENT parents
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	602 continue;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	603 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	604
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	605 new_recs[dest].name_hash = sync_recs[src].name_hash;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	606 new_recs[dest].dir_offset =
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	607 mail_index_uint32_to_offset(sync_recs[src].dir_offset);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	608 if (sync_recs[src].created) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	609 /* new record */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	610 new_recs[dest].uid = sync_recs[src].uid;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	611 new_recs[dest].name_offset = base_offset + name_pos;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	612 size = strlen(sync_recs[src].name) + 1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	613 memcpy(PTR_OFFSET(base, name_pos), sync_recs[src].name,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	614 size);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	615 name_pos += size;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	616 } else {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	617 /* existing record. need to find its name_offset */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	618 for (orig = 0; orig < dir->count; orig++) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	619 if (recs[orig].uid == sync_recs[src].uid)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	620 break;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	621 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	622 i_assert(orig < dir->count);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	623
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	624 new_recs[dest].uid = sync_recs[src].uid;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	625 new_recs[dest].name_offset = recs[orig].name_offset;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	626 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	627 dest++;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	628 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	629 i_assert(dest == nondeleted_count);
5343 cdf39128dd91 More desyncing fixes Timo Sirainen <tss@iki.fi> parents: 5336 diff changeset	630 i_assert(name_pos == space_needed);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	631
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	632 if (index->mmap_disable) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	633 file_cache_write(index->file_cache, ctx->output_buf->data,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	634 ctx->output_buf->used, ctx->output->offset);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	635 o_stream_send(ctx->output, ctx->output_buf->data,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	636 ctx->output_buf->used);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	637 }
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	638
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	639 if (offset_pos == 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	640 /* we're writing the root directory */
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	641 i_assert(base_offset == sizeof(*index->hdr));
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	642 } else {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	643 /* add a link to this newly created directory. */
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	644 uint32_t data = mail_index_uint32_to_offset(base_offset);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	645
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	646 if (!index->mmap_disable) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	647 uint32_t *pos;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	648
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	649 pos = PTR_OFFSET(index->mmap_base, offset_pos);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	650 i_assert(mail_index_offset_to_uint32(*pos) == 0);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	651 *pos = data;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	652 } else {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	653 uoff_t old_offset = ctx->output->offset;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	654
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	655 file_cache_write(index->file_cache,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	656 &data, sizeof(data), offset_pos);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	657
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	658 o_stream_seek(ctx->output, offset_pos);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	659 o_stream_send(ctx->output, &data, sizeof(data));
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	660 o_stream_seek(ctx->output, old_offset);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	661 }
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	662 }
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	663
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	664 if (index->mmap_disable) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	665 /* file_cache_write() calls may have moved mmaping */
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	666 index->const_mmap_base = file_cache_get_map(index->file_cache,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	667 &index->mmap_size);
5313 f17cd7735f8d fix Timo Sirainen <tss@iki.fi> parents: 5043 diff changeset	668 index->hdr = index->const_mmap_base;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	669 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	670
5345 063c892d0439 deleted_space wasn't being calculated, so the index was never compressed. Timo Sirainen <tss@iki.fi> parents: 5344 diff changeset	671 ctx->hdr.deleted_space += deleted_space;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	672 sync_dir->offset = base_offset;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	673 return 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	674 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	675
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	676 static int
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	677 mailbox_list_index_sync_update_dir(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	678 struct mailbox_list_sync_dir *sync_dir)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	679 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	680 const struct mailbox_list_dir_record *dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	681 struct mailbox_list_record *recs;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	682 const struct mailbox_list_sync_record *sync_recs;
5043 6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	683 unsigned int i, j, count;
5344 da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	684 uint32_t seq;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	685
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	686 i_assert(sync_dir->offset != 0);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	687
5324 4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	688 if (mailbox_list_index_get_dir(ctx->view, &sync_dir->offset, &dir) < 0)
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	689 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	690
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	691 sync_recs = array_get(&sync_dir->records, &count);
5043 6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	692 i_assert(count <= dir->count);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	693 i_assert(sync_dir->seen_records_count < count);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	694
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	695 if (!ctx->index->mmap_disable)
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	696 recs = MAILBOX_LIST_RECORDS_MODIFIABLE(dir);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	697 else {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	698 /* @UNSAFE: copy the records into a temporary buffer that
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	699 we modify and then write back to disk */
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	700 t_push();
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	701 recs = t_new(struct mailbox_list_record, dir->count);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	702 memcpy(recs, MAILBOX_LIST_RECORDS(dir),
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	703 sizeof(struct mailbox_list_record) * dir->count);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	704 }
5043 6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	705
6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	706 /* records marked with deleted have been removed from sync_recs, so
6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	707 we need to skip those */
6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	708 for (i = j = 0; i < count; i++) {
6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	709 while (recs[j].uid != sync_recs[i].uid) {
6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	710 j++;
6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	711 i_assert(j < dir->count);
6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	712 }
6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	713
5344 da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	714 if (!sync_recs[i].seen) {
5043 6eb476100897 Fixed index updating when mailboxes were deleted. Timo Sirainen <tss@iki.fi> parents: 4929 diff changeset	715 recs[j].deleted = TRUE;
5344 da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	716
da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	717 /* expunge from mail index */
da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	718 if (mail_index_lookup_uid_range(ctx->mail_view,
da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	719 sync_recs[i].uid,
da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	720 sync_recs[i].uid,
da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	721 &seq, &seq) == 0 &&
da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	722 seq != 0)
da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	723 mail_index_expunge(ctx->trans, seq);
da77f5dd7854 We didn't always expunge deleted mailboxes from mail index. Timo Sirainen <tss@iki.fi> parents: 5343 diff changeset	724 }
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	725 }
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	726 if (ctx->index->mmap_disable) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	727 uoff_t offset, old_offset;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	728 size_t size = sizeof(struct mailbox_list_record) * dir->count;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	729
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	730 offset = sync_dir->offset +
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	731 sizeof(struct mailbox_list_dir_record);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	732 file_cache_write(ctx->index->file_cache, recs, size, offset);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	733
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	734 old_offset = ctx->output->offset;
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	735 o_stream_seek(ctx->output, offset);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	736 o_stream_send(ctx->output, recs, size);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	737 o_stream_seek(ctx->output, old_offset);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	738 t_pop();
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	739 }
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	740 return 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	741 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	742
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	743 static int
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	744 mailbox_list_index_sync_write_dir(struct mailbox_list_index_sync_ctx *ctx,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	745 struct mailbox_list_sync_dir *sync_dir,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	746 uint32_t offset_pos, bool partial)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	747 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	748 const struct mailbox_list_dir_record *dir;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	749 const struct mailbox_list_record *recs;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	750 const struct mailbox_list_sync_record *sync_recs;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	751 uint32_t child_offset_pos;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	752 unsigned int i, j, count;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	753
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	754 if (!ctx->seen_sync_root && ctx->sync_root == sync_dir) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	755 i_assert(partial);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	756 ctx->seen_sync_root = TRUE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	757 partial = (ctx->flags & MAILBOX_LIST_SYNC_FLAG_PARTIAL) != 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	758 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	759
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	760 if (sync_dir->offset != 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	761 /* point to latest dir entry's next_offset */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	762 offset_pos = sync_dir->offset +
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	763 offsetof(struct mailbox_list_dir_record, next_offset);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	764 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	765
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	766 if (sync_dir->new_records_count > 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	767 /* need to recreate the dir record */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	768 if (mailbox_list_index_sync_recreate_dir(ctx, sync_dir,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	769 offset_pos,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	770 partial) < 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	771 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	772 /* NOTE: index may have been remaped here */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	773 } else if (sync_dir->seen_records_count !=
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	774 array_count(&sync_dir->records) && !partial) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	775 /* just mark the records deleted */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	776 if (mailbox_list_index_sync_update_dir(ctx, sync_dir) < 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	777 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	778 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	779
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	780 if (!partial && (ctx->flags & MAILBOX_LIST_SYNC_FLAG_RECURSIVE) == 0) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	781 /* we're doing a full sync only for the root */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	782 partial = TRUE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	783 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	784
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	785 /* update child mailboxes */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	786 sync_recs = array_get(&sync_dir->records, &count);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	787 if (count == 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	788 return 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	789
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	790 i_assert(sync_dir->offset != 0 &&
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	791 sync_dir->offset < ctx->index->mmap_size);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	792 for (i = j = 0; i < count; i++) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	793 if (sync_recs[i].dir == NULL)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	794 continue;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	795
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	796 /* these may change after each sync_write_dir() call */
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	797 dir = CONST_PTR_OFFSET(ctx->index->const_mmap_base,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	798 sync_dir->offset);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	799 recs = MAILBOX_LIST_RECORDS(dir);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	800
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	801 /* child_offset_pos needs to point to record's dir_offset */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	802 for (; j < dir->count; j++) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	803 if (recs[j].uid == sync_recs[i].uid)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	804 break;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	805 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	806 i_assert(j < dir->count);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	807
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	808 child_offset_pos = (const char *)&recs[j].dir_offset -
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	809 (const char *)ctx->index->const_mmap_base;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	810 if (mailbox_list_index_sync_write_dir(ctx, sync_recs[i].dir,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	811 child_offset_pos,
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	812 partial) < 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	813 return -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	814 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	815 return 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	816 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	817
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	818 static int
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	819 mailbox_list_index_sync_write(struct mailbox_list_index_sync_ctx *ctx)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	820 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	821 struct mailbox_list_index_header *hdr;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	822 bool partial;
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	823 int ret = 0;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	824
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	825 if (ctx->index->mmap_disable) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	826 ctx->output = o_stream_create_file(ctx->index->fd, default_pool,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	827 0, FALSE);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	828 ctx->output_buf = buffer_create_dynamic(default_pool, 4096);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	829 o_stream_seek(ctx->output, ctx->hdr.used_space);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	830 }
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	831
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	832 if (ctx->sync_root == ctx->root) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	833 ctx->seen_sync_root = TRUE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	834 partial = (ctx->flags & MAILBOX_LIST_SYNC_FLAG_PARTIAL) != 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	835 } else {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	836 /* until we've seen the sync root, we're doing only partial
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	837 syncing */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	838 partial = TRUE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	839 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	840
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	841 if (mailbox_list_index_sync_write_dir(ctx, ctx->root, 0, partial) < 0)
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	842 ret = -1;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	843
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	844 if (!ctx->index->mmap_disable) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	845 /* update header */
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	846 hdr = ctx->index->mmap_base;
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	847 if (ret == 0)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	848 memcpy(hdr, &ctx->hdr, sizeof(*hdr));
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	849
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	850 if (msync(ctx->index->mmap_base,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	851 hdr->used_space, MS_SYNC) < 0) {
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	852 mailbox_list_index_set_syscall_error(ctx->index,
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	853 "msync()");
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	854 ret = -1;
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	855 }
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	856 } else {
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	857 if (ret == 0) {
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	858 o_stream_seek(ctx->output, 0);
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	859 o_stream_send(ctx->output, &ctx->hdr, sizeof(ctx->hdr));
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	860 }
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	861
5972 9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	862 if (o_stream_flush(ctx->output) < 0) {
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	863 mailbox_list_index_set_syscall_error(ctx->index,
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	864 "o_stream_flush()");
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	865 ret = -1;
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	866 }
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	867 if (ret == 0 && ctx->index->mail_index->nfs_flush &&
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	868 fdatasync(ctx->index->fd) < 0) {
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	869 mailbox_list_index_set_syscall_error(ctx->index,
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	870 "fdatasync()");
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	871 ret = -1;
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	872 }
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5779 diff changeset	873
4896 3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	874 o_stream_destroy(&ctx->output);
3d00a9636393 Mailbox list index works now with mmap_disable=yes. Timo Sirainen <tss@iki.fi> parents: 4848 diff changeset	875 buffer_free(ctx->output_buf);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	876 }
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	877 return ret;
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	878 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	879
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	880 int mailbox_list_index_sync_commit(struct mailbox_list_index_sync_ctx **_ctx)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	881 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	882 struct mailbox_list_index_sync_ctx ctx = _ctx;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	883 int ret = ctx->failed ? -1 : 0;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	884
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	885 *_ctx = NULL;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	886
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	887 if (!ctx->failed) {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	888 /* write all the changes to the index */
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	889 ret = mailbox_list_index_sync_write(ctx);
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	890 if (ret == 0 && mailbox_list_index_need_compress(ctx->index))
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	891 ret = mailbox_list_index_compress(ctx);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	892 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	893
5324 4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	894 if (ret == 0) {
4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	895 uint64_t used_space = ctx->hdr.used_space;
4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	896
4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	897 /* FIXME: we should use some extension header instead of
4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	898 reusing sync_size */
4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	899 mail_index_update_header(ctx->trans,
4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	900 offsetof(struct mail_index_header, sync_size),
4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	901 &used_space, sizeof(used_space), FALSE);
4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	902 }
4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	903
5323 5c8f84bc07b1 Error handling fixes Timo Sirainen <tss@iki.fi> parents: 5315 diff changeset	904 if (ctx->mail_sync_ctx != NULL) {
6075 813c976ed476 Handle desync errors without marking mailbox list index corrupted, the Timo Sirainen <tss@iki.fi> parents: 6072 diff changeset	905 if (ret < 0 && !ctx->restart)
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	906 mail_index_sync_rollback(&ctx->mail_sync_ctx);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	907 else {
6075 813c976ed476 Handle desync errors without marking mailbox list index corrupted, the Timo Sirainen <tss@iki.fi> parents: 6072 diff changeset	908 if (ctx->restart)
813c976ed476 Handle desync errors without marking mailbox list index corrupted, the Timo Sirainen <tss@iki.fi> parents: 6072 diff changeset	909 mail_index_reset(ctx->trans);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	910 if (mail_index_sync_commit(&ctx->mail_sync_ctx) < 0)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	911 ret = -1;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	912 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	913 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	914
5324 4d3294c59939 Fixed desyncing problems with mail index and mailbox list index. Timo Sirainen <tss@iki.fi> parents: 5323 diff changeset	915 mailbox_list_index_view_deinit(&ctx->view);
4848 967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	916 pool_unref(ctx->pool);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	917 return ret;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	918 }
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	919
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	920 void mailbox_list_index_sync_rollback(struct mailbox_list_index_sync_ctx **ctx)
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	921 {
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	922 (*ctx)->failed = TRUE;
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	923 (void)mailbox_list_index_sync_commit(ctx);
967de900c73a Mailbox list indexing and related changes. Currently works only with Timo Sirainen <tss@iki.fi> parents: diff changeset	924 }
4929 791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	925
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	926 static bool mailbox_list_index_need_compress(struct mailbox_list_index *index)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	927 {
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	928 uoff_t max_del_space;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	929
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	930 max_del_space = index->hdr->used_space / 100 *
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	931 MAILBOX_LIST_COMPRESS_PERCENTAGE;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	932 if (index->hdr->deleted_space >= max_del_space &&
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	933 index->hdr->used_space >= MAILBOX_LIST_COMPRESS_MIN_SIZE)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	934 return TRUE;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	935
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	936 return FALSE;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	937 }
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	938
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	939 static int mailbox_list_copy_to_mem_all(struct mailbox_list_index_sync_ctx *ctx,
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	940 struct mailbox_list_sync_dir *dir)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	941 {
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	942 struct mailbox_list_sync_record *recs;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	943 unsigned int i, count;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	944
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	945 /* mark the directories as new */
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	946 dir->offset = 0;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	947 dir->new_records_count = 1;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	948
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	949 recs = array_get_modifiable(&dir->records, &count);
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	950 for (i = 0; i < count; i++) {
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	951 recs[i].created = TRUE;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	952 recs[i].seen = TRUE;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	953
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	954 if (recs[i].dir == NULL) {
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	955 if (recs[i].dir_offset == 0)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	956 continue;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	957
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	958 if (mailbox_list_copy_sync_dir(ctx, recs[i].dir_offset,
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	959 &recs[i].dir) < 0)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	960 return -1;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	961 }
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	962 recs[i].dir_offset = 0;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	963
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	964 if (mailbox_list_copy_to_mem_all(ctx, recs[i].dir) < 0)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	965 return -1;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	966 }
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	967 return 0;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	968 }
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	969
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	970 static int mailbox_list_index_compress(struct mailbox_list_index_sync_ctx *ctx)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	971 {
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	972 /* first read everything to memory */
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	973 if (mailbox_list_copy_to_mem_all(ctx, ctx->root) < 0)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	974 return -1;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	975
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	976 /* truncate the index file */
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	977 mailbox_list_index_file_close(ctx->index);
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	978 if (mailbox_list_index_file_create(ctx->index,
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	979 ctx->hdr.uid_validity) < 0)
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	980 return -1;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	981
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	982 /* reset header */
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	983 ctx->hdr.used_space = sizeof(ctx->hdr);
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	984 ctx->hdr.deleted_space = 0;
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	985
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	986 /* and write everything back */
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	987 return mailbox_list_index_sync_write(ctx);
791c9c8d3a02 Compress list index when its deleted space amount gets too high. Also fixed Timo Sirainen <tss@iki.fi> parents: 4896 diff changeset	988 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-index/mailbox-list-index-sync.c @ 6075:813c976ed476 HEAD