dovecot/original-hg/dovecot-1.2: src/plugins/fts-squat/squat-uidlist.c annotate

annotate src/plugins/fts-squat/squat-uidlist.c @ 9575:0a00dcc4f0ea HEAD

lib-storage: Allow shared namespace prefix to use %variable modifiers.

author	Timo Sirainen <tss@iki.fi>
date	Wed, 26 May 2010 17:07:51 +0100
parents	00cd9aacd03c
children

rev	line source
9532 00cd9aacd03c Updated copyright notices to include year 2010. Timo Sirainen <tss@iki.fi> parents: 8850 diff changeset	1 /* Copyright (c) 2007-2010 Dovecot authors, see the included COPYING file */
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	2
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	3 #include "lib.h"
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	4 #include "array.h"
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	5 #include "bsearch-insert-pos.h"
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	6 #include "file-cache.h"
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	7 #include "file-lock.h"
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	8 #include "read-full.h"
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	9 #include "write-full.h"
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	10 #include "ostream.h"
7373 4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	11 #include "mmap-util.h"
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	12 #include "squat-trie-private.h"
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	13 #include "squat-uidlist.h"
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	14
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	15 #include <stdio.h>
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	16 #include <sys/stat.h>
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	17
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	18 #define UIDLIST_LIST_SIZE 31
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	19 #define UIDLIST_BLOCK_LIST_COUNT 100
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	20 #define UID_LIST_MASK_RANGE 0x80000000
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	21
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	22 /* set = points to uidlist index number, unset = points to uidlist offset */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	23 #define UID_LIST_POINTER_MASK_LIST_IDX 0x80000000
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	24
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	25 #define UIDLIST_PACKED_FLAG_BITMASK 1
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	26 #define UIDLIST_PACKED_FLAG_BEGINS_WITH_POINTER 2
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	27
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	28 struct uidlist_list {
7370 d51565e31ceb Minor optimization and possible compile fix. Timo Sirainen <tss@iki.fi> parents: 7369 diff changeset	29 unsigned int uid_count:31;
d51565e31ceb Minor optimization and possible compile fix. Timo Sirainen <tss@iki.fi> parents: 7369 diff changeset	30 unsigned int uid_begins_with_pointer:1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	31 uint32_t uid_list[UIDLIST_LIST_SIZE];
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	32 };
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	33
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	34 struct squat_uidlist {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	35 struct squat_trie *trie;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	36
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	37 char *path;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	38 int fd;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	39 struct file_cache *file_cache;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	40
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	41 struct file_lock *file_lock;
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	42 struct dotlock *dotlock;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	43 uoff_t locked_file_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	44
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	45 void *mmap_base;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	46 size_t mmap_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	47 struct squat_uidlist_file_header hdr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	48
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	49 const void *data;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	50 size_t data_size;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	51
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	52 unsigned int cur_block_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	53 const uint32_t *cur_block_offsets;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	54 const uint32_t *cur_block_end_indexes;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	55
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	56 size_t max_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	57 unsigned int corrupted:1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	58 unsigned int building:1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	59 };
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	60
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	61 struct squat_uidlist_build_context {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	62 struct squat_uidlist *uidlist;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	63 struct ostream *output;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	64
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	65 ARRAY_TYPE(uint32_t) block_offsets;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	66 ARRAY_TYPE(uint32_t) block_end_indexes;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	67
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	68 ARRAY_DEFINE(lists, struct uidlist_list);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	69 uint32_t list_start_idx;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	70
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	71 struct squat_uidlist_file_header build_hdr;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	72 unsigned int need_reopen:1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	73 };
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	74
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	75 struct squat_uidlist_rebuild_context {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	76 struct squat_uidlist *uidlist;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	77 struct squat_uidlist_build_context *build_ctx;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	78
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	79 int fd;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	80 struct ostream *output;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	81
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	82 ARRAY_TYPE(uint32_t) new_block_offsets, new_block_end_indexes;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	83 uoff_t cur_block_start_offset;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	84
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	85 uint32_t list_sizes[UIDLIST_BLOCK_LIST_COUNT];
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	86 uint32_t next_uid_list_idx;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	87 unsigned int list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	88 unsigned int new_count;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	89 };
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	90
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	91 static void squat_uidlist_close(struct squat_uidlist *uidlist);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	92
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	93 void squat_uidlist_delete(struct squat_uidlist *uidlist)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	94 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	95 if (unlink(uidlist->path) < 0 && errno != ENOENT)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	96 i_error("unlink(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	97 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	98
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	99 static void squat_uidlist_set_corrupted(struct squat_uidlist *uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	100 const char *reason)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	101 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	102 if (uidlist->corrupted)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	103 return;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	104 uidlist->corrupted = TRUE;
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	105
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	106 i_error("Corrupted squat uidlist file %s: %s", uidlist->path, reason);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	107 squat_trie_delete(uidlist->trie);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	108 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	109
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	110 static int
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	111 uidlist_write_array(struct ostream output, const uint32_t uid_list,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	112 unsigned int uid_count, uint32_t packed_flags,
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	113 uint32_t offset, bool write_size, uint32_t *size_r)
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	114 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	115 uint8_t uidbuf, bufp, sizebuf[SQUAT_PACK_MAX_SIZE], *sizebufp;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	116 uint8_t listbuf[SQUAT_PACK_MAX_SIZE], *listbufp = listbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	117 uint32_t uid, uid2, prev, base_uid, size_value;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	118 unsigned int i, bitmask_len, uid_list_len;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	119 unsigned int idx, max_idx, mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	120 bool datastack;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	121 int num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	122
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	123 if ((packed_flags & UIDLIST_PACKED_FLAG_BEGINS_WITH_POINTER) != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	124 squat_pack_num(&listbufp, offset);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	125
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	126 /* @UNSAFE */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	127 base_uid = uid_list[0] & ~UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	128 datastack = uid_count < 1024*8/SQUAT_PACK_MAX_SIZE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	129 if (datastack)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	130 uidbuf = t_malloc(SQUAT_PACK_MAX_SIZE * uid_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	131 else
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	132 uidbuf = i_malloc(SQUAT_PACK_MAX_SIZE * uid_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	133 bufp = uidbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	134 squat_pack_num(&bufp, base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	135
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	136 bitmask_len = (uid_list[uid_count-1] - base_uid + 7) / 8 +
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	137 (bufp - uidbuf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	138 if (bitmask_len < uid_count) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	139 bitmask_build:
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	140 i_assert(bitmask_len < SQUAT_PACK_MAX_SIZE*uid_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	141
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	142 memset(bufp, 0, bitmask_len - (bufp - uidbuf));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	143 if ((uid_list[0] & UID_LIST_MASK_RANGE) == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	144 i = 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	145 uid = i == uid_count ? 0 : uid_list[i];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	146 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	147 i = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	148 uid = uid_list[0] + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	149 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	150 base_uid++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	151
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	152 for (; i < uid_count; i++) {
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	153 i_assert((uid & ~UID_LIST_MASK_RANGE) >= base_uid);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	154 if ((uid & UID_LIST_MASK_RANGE) == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	155 uid -= base_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	156 uid2 = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	157 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	158 uid &= ~UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	159 uid -= base_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	160 uid2 = uid_list[i+1] - base_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	161 i++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	162 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	163
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	164 if (uid2 - uid < 3*8) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	165 for (; uid <= uid2; uid++)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	166 bufp[uid / 8] \|= 1 << (uid % 8);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	167 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	168 /* first byte */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	169 idx = uid / 8;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	170 num = uid % 8;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	171 if (num != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	172 uid += 8 - num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	173 for (mask = 0; num < 8; num++)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	174 mask \|= 1 << num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	175 bufp[idx++] \|= mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	176 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	177
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	178 /* middle bytes */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	179 num = uid2 % 8;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	180 max_idx = idx + (uid2 - num - uid)/8;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	181 for (; idx < max_idx; idx++, uid += 8)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	182 bufp[idx] = 0xff;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	183
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	184 /* last byte */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	185 for (mask = 0; num >= 0; num--)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	186 mask \|= 1 << num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	187 bufp[idx] \|= mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	188 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	189 uid = i+1 == uid_count ? 0 : uid_list[i+1];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	190 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	191 uid_list_len = bitmask_len;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	192 packed_flags \|= UIDLIST_PACKED_FLAG_BITMASK;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	193 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	194 bufp = uidbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	195 prev = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	196 for (i = 0; i < uid_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	197 uid = uid_list[i];
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	198 if (unlikely((uid & ~UID_LIST_MASK_RANGE) < prev))
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	199 return -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	200 if ((uid & UID_LIST_MASK_RANGE) == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	201 squat_pack_num(&bufp, (uid - prev) << 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	202 prev = uid + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	203 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	204 uid &= ~UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	205 squat_pack_num(&bufp, 1 \| (uid - prev) << 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	206 squat_pack_num(&bufp, uid_list[i+1] - uid - 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	207 prev = uid_list[i+1] + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	208 i++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	209 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	210 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	211 uid_list_len = bufp - uidbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	212 if (uid_list_len > bitmask_len) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	213 bufp = uidbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	214 squat_pack_num(&bufp, base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	215 goto bitmask_build;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	216 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	217 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	218
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	219 size_value = ((uid_list_len +
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	220 (listbufp - listbuf)) << 2) \| packed_flags;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	221 if (write_size) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	222 sizebufp = sizebuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	223 squat_pack_num(&sizebufp, size_value);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	224 o_stream_send(output, sizebuf, sizebufp - sizebuf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	225 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	226 o_stream_send(output, listbuf, listbufp - listbuf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	227 o_stream_send(output, uidbuf, uid_list_len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	228 if (!datastack)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	229 i_free(uidbuf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	230
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	231 *size_r = size_value;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	232 return 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	233 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	234
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	235 static int
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	236 uidlist_write(struct ostream output, const struct uidlist_list list,
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	237 bool write_size, uint32_t *size_r)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	238 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	239 const uint32_t *uid_list = list->uid_list;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	240 uint8_t buf[SQUAT_PACK_MAX_SIZE], *bufp;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	241 uint32_t uid_count = list->uid_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	242 uint32_t packed_flags = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	243 uint32_t offset = 0;
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	244 int ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	245
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	246 if (list->uid_begins_with_pointer) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	247 /* continued UID list */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	248 packed_flags \|= UIDLIST_PACKED_FLAG_BEGINS_WITH_POINTER;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	249 if ((uid_list[0] & UID_LIST_POINTER_MASK_LIST_IDX) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	250 offset = ((uid_list[0] & ~UID_LIST_POINTER_MASK_LIST_IDX) << 1) \| 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	251 if (list->uid_count == 1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	252 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	253 squat_pack_num(&bufp, offset);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	254 o_stream_send(output, buf, bufp - buf);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	255 *size_r = (bufp - buf) << 2 \| packed_flags;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	256 return 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	257 }
7369 fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	258 } else if (unlikely(output->offset <= uid_list[0])) {
fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	259 i_assert(output->closed);
fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	260 return -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	261 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	262 i_assert(list->uid_count > 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	263 offset = (output->offset - uid_list[0]) << 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	264 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	265 uid_list++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	266 uid_count--;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	267 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	268
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	269 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	270 ret = uidlist_write_array(output, uid_list, uid_count,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	271 packed_flags, offset,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	272 write_size, size_r);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	273 } T_END;
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	274 return ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	275 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	276
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	277 static void squat_uidlist_map_blocks_set_pointers(struct squat_uidlist *uidlist)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	278 {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	279 const void *base;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	280 size_t end_index_size, end_size;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	281
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	282 base = CONST_PTR_OFFSET(uidlist->data, uidlist->hdr.block_list_offset +
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	283 sizeof(uint32_t));
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	284
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	285 end_index_size = uidlist->cur_block_count * sizeof(uint32_t);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	286 end_size = end_index_size + uidlist->cur_block_count * sizeof(uint32_t);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	287 if (end_size <= uidlist->data_size) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	288 uidlist->cur_block_end_indexes = base;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	289 uidlist->cur_block_offsets =
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	290 CONST_PTR_OFFSET(base, end_index_size);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	291 } else {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	292 uidlist->cur_block_end_indexes = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	293 uidlist->cur_block_offsets = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	294 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	295 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	296
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	297 static int uidlist_file_cache_read(struct squat_uidlist *uidlist,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	298 size_t offset, size_t size)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	299 {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	300 if (uidlist->file_cache == NULL)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	301 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	302
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	303 if (file_cache_read(uidlist->file_cache, offset, size) < 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	304 i_error("read(%s) failed: %m", uidlist->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	305 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	306 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	307 uidlist->data = file_cache_get_map(uidlist->file_cache,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	308 &uidlist->data_size);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	309 squat_uidlist_map_blocks_set_pointers(uidlist);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	310 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	311 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	312
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	313 static int squat_uidlist_map_blocks(struct squat_uidlist *uidlist)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	314 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	315 const struct squat_uidlist_file_header *hdr = &uidlist->hdr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	316 const void *base;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	317 uint32_t block_count, blocks_offset, blocks_size, i, verify_count;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	318
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	319 if (hdr->block_list_offset == 0) {
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	320 /* empty file */
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	321 uidlist->cur_block_count = 0;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	322 return 1;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	323 }
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	324
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	325 /* get number of blocks */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	326 if (uidlist_file_cache_read(uidlist, hdr->block_list_offset,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	327 sizeof(block_count)) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	328 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	329 blocks_offset = hdr->block_list_offset + sizeof(block_count);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	330 if (blocks_offset > uidlist->data_size) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	331 squat_uidlist_set_corrupted(uidlist, "block list outside file");
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	332 return 0;
5414 3ec03a4c0f26 fixes Timo Sirainen <tss@iki.fi> parents: 5010 diff changeset	333 }
3ec03a4c0f26 fixes Timo Sirainen <tss@iki.fi> parents: 5010 diff changeset	334
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	335 base = CONST_PTR_OFFSET(uidlist->data, hdr->block_list_offset);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	336 memcpy(&block_count, base, sizeof(block_count));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	337
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	338 /* map the blocks */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	339 blocks_size = block_count * sizeof(uint32_t)*2;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	340 if (uidlist_file_cache_read(uidlist, blocks_offset, blocks_size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	341 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	342 if (blocks_offset + blocks_size > uidlist->data_size) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	343 squat_uidlist_set_corrupted(uidlist, "block list outside file");
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	344 return 0;
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	345 }
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	346
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	347 uidlist->cur_block_count = block_count;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	348 squat_uidlist_map_blocks_set_pointers(uidlist);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	349
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	350 /* verify just a couple of the end indexes to make sure they
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	351 look correct */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	352 verify_count = I_MIN(block_count, 8);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	353 for (i = 1; i < verify_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	354 if (unlikely(uidlist->cur_block_end_indexes[i-1] >=
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	355 uidlist->cur_block_end_indexes[i])) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	356 squat_uidlist_set_corrupted(uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	357 "block list corrupted");
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	358 return 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	359 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	360 }
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	361 return 1;
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	362 }
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	363
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	364 static int squat_uidlist_map_header(struct squat_uidlist *uidlist)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	365 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	366 if (uidlist->hdr.indexid == 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	367 /* still being built */
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	368 return 1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	369 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	370 if (uidlist->hdr.indexid != uidlist->trie->hdr.indexid) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	371 /* see if trie was recreated */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	372 squat_trie_refresh(uidlist->trie);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	373 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	374 if (uidlist->hdr.indexid != uidlist->trie->hdr.indexid) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	375 squat_uidlist_set_corrupted(uidlist, "wrong indexid");
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	376 return 0;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	377 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	378 if (uidlist->hdr.used_file_size < sizeof(uidlist->hdr) \|\|
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	379 (uidlist->hdr.used_file_size > uidlist->mmap_size &&
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	380 uidlist->mmap_base != NULL)) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	381 squat_uidlist_set_corrupted(uidlist, "broken used_file_size");
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	382 return 0;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	383 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	384 return squat_uidlist_map_blocks(uidlist);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	385 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	386
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	387 static void squat_uidlist_unmap(struct squat_uidlist *uidlist)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	388 {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	389 if (uidlist->mmap_size != 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	390 if (munmap(uidlist->mmap_base, uidlist->mmap_size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	391 i_error("munmap(%s) failed: %m", uidlist->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	392 uidlist->mmap_base = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	393 uidlist->mmap_size = 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	394 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	395 uidlist->cur_block_count = 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	396 uidlist->cur_block_end_indexes = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	397 uidlist->cur_block_offsets = NULL;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	398 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	399
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	400 static int squat_uidlist_mmap(struct squat_uidlist *uidlist)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	401 {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	402 struct stat st;
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	403
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	404 if (fstat(uidlist->fd, &st) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	405 i_error("fstat(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	406 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	407 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	408 if (st.st_size < (off_t)sizeof(uidlist->hdr)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	409 squat_uidlist_set_corrupted(uidlist, "File too small");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	410 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	411 }
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	412
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	413 squat_uidlist_unmap(uidlist);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	414 uidlist->mmap_size = st.st_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	415 uidlist->mmap_base = mmap(NULL, uidlist->mmap_size,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	416 PROT_READ \| PROT_WRITE,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	417 MAP_SHARED, uidlist->fd, 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	418 if (uidlist->mmap_base == MAP_FAILED) {
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	419 uidlist->data = uidlist->mmap_base = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	420 uidlist->data_size = uidlist->mmap_size = 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	421 i_error("mmap(%s) failed: %m", uidlist->path);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	422 return -1;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	423 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	424 uidlist->data = uidlist->mmap_base;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	425 uidlist->data_size = uidlist->mmap_size;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	426 return 0;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	427 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	428
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	429 static int squat_uidlist_map(struct squat_uidlist *uidlist)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	430 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	431 const struct squat_uidlist_file_header *mmap_hdr = uidlist->mmap_base;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	432 int ret;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	433
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	434 if (mmap_hdr != NULL && !uidlist->building &&
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	435 uidlist->hdr.block_list_offset == mmap_hdr->block_list_offset) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	436 /* file hasn't changed */
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	437 return 1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	438 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	439
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	440 if ((uidlist->trie->flags & SQUAT_INDEX_FLAG_MMAP_DISABLE) == 0) {
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	441 if (mmap_hdr == NULL \|\| uidlist->building \|\|
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	442 uidlist->mmap_size < mmap_hdr->used_file_size) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	443 if (squat_uidlist_mmap(uidlist) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	444 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	445 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	446
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	447 if (!uidlist->building) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	448 memcpy(&uidlist->hdr, uidlist->mmap_base,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	449 sizeof(uidlist->hdr));
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	450 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	451 } else if (uidlist->building) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	452 /* we want to update blocks mapping, but using the header
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	453 in memory */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	454 } else {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	455 ret = pread_full(uidlist->fd, &uidlist->hdr,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	456 sizeof(uidlist->hdr), 0);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	457 if (ret <= 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	458 if (ret < 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	459 i_error("pread(%s) failed: %m", uidlist->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	460 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	461 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	462 i_error("Corrupted %s: File too small", uidlist->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	463 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	464 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	465 uidlist->data = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	466 uidlist->data_size = 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	467 }
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	468 if (uidlist->file_cache == NULL &&
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	469 (uidlist->trie->flags & SQUAT_INDEX_FLAG_MMAP_DISABLE) != 0)
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	470 uidlist->file_cache = file_cache_new(uidlist->fd);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	471 return squat_uidlist_map_header(uidlist);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	472 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	473
7373 4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	474 static int squat_uidlist_read_to_memory(struct squat_uidlist *uidlist)
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	475 {
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	476 size_t i, page_size = mmap_get_page_size();
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	477 char x;
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	478
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	479 if (uidlist->file_cache != NULL) {
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	480 return uidlist_file_cache_read(uidlist, 0,
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	481 uidlist->hdr.used_file_size);
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	482 }
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	483 /* Tell the kernel we're going to use the uidlist data, so it loads
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	484 it into memory and keeps it there. */
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	485 (void)madvise(uidlist->mmap_base, uidlist->mmap_size, MADV_WILLNEED);
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	486 /* It also speeds up a bit for us to sequentially load everything
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	487 into memory, although at least Linux catches up quite fast even
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	488 without this code. Compiler can quite easily optimize away this
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	489 entire for loop, but volatile seems to help with gcc 4.2. */
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	490 for (i = 0; i < uidlist->mmap_size; i += page_size)
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	491 x = ((const volatile char *)uidlist->data)[i];
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	492 return 0;
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	493 }
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	494
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	495 static void squat_uidlist_free_from_memory(struct squat_uidlist *uidlist)
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	496 {
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	497 size_t page_size = mmap_get_page_size();
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	498
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	499 if (uidlist->file_cache != NULL) {
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	500 file_cache_invalidate(uidlist->file_cache,
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	501 page_size, (uoff_t)-1);
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	502 } else {
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	503 (void)madvise(uidlist->mmap_base, uidlist->mmap_size,
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	504 MADV_DONTNEED);
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	505 }
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	506 }
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	507
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	508 struct squat_uidlist squat_uidlist_init(struct squat_trie trie)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	509 {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	510 struct squat_uidlist *uidlist;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	511
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	512 uidlist = i_new(struct squat_uidlist, 1);
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	513 uidlist->trie = trie;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	514 uidlist->path = i_strconcat(trie->path, ".uids", NULL);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	515 uidlist->fd = -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	516
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	517 return uidlist;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	518 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	519
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	520 void squat_uidlist_deinit(struct squat_uidlist *uidlist)
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	521 {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	522 squat_uidlist_close(uidlist);
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	523
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	524 i_free(uidlist->path);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	525 i_free(uidlist);
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	526 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	527
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	528 static int squat_uidlist_open(struct squat_uidlist *uidlist)
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	529 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	530 squat_uidlist_close(uidlist);
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	531
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	532 uidlist->fd = open(uidlist->path, O_RDWR);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	533 if (uidlist->fd == -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	534 if (errno == ENOENT) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	535 memset(&uidlist->hdr, 0, sizeof(uidlist->hdr));
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	536 return 0;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	537 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	538 i_error("open(%s) failed: %m", uidlist->path);
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	539 return -1;
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	540 }
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	541 return squat_uidlist_map(uidlist) <= 0 ? -1 : 0;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	542 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	543
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	544 static void squat_uidlist_close(struct squat_uidlist *uidlist)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	545 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	546 i_assert(!uidlist->building);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	547
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	548 squat_uidlist_unmap(uidlist);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	549 if (uidlist->file_cache != NULL)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	550 file_cache_free(&uidlist->file_cache);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	551 if (uidlist->file_lock != NULL)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	552 file_lock_free(&uidlist->file_lock);
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	553 if (uidlist->dotlock != NULL)
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	554 file_dotlock_delete(&uidlist->dotlock);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	555 if (uidlist->fd != -1) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	556 if (close(uidlist->fd) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	557 i_error("close(%s) failed: %m", uidlist->path);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	558 uidlist->fd = -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	559 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	560 uidlist->corrupted = FALSE;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	561 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	562
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	563 int squat_uidlist_refresh(struct squat_uidlist *uidlist)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	564 {
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	565 /* we assume here that trie is locked, so that we don't need to worry
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	566 about it when reading the header */
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	567 if (uidlist->fd == -1 \|\|
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	568 uidlist->hdr.indexid != uidlist->trie->hdr.indexid) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	569 if (squat_uidlist_open(uidlist) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	570 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	571 } else {
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	572 if (squat_uidlist_map(uidlist) <= 0)
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	573 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	574 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	575 return 0;
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	576 }
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	577
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	578 static int squat_uidlist_is_file_stale(struct squat_uidlist *uidlist)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	579 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	580 struct stat st, st2;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	581
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	582 i_assert(uidlist->fd != -1);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	583
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	584 if (stat(uidlist->path, &st) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	585 if (errno == ENOENT)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	586 return 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	587
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	588 i_error("stat(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	589 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	590 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	591 if (fstat(uidlist->fd, &st2) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	592 i_error("fstat(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	593 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	594 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	595 uidlist->locked_file_size = st2.st_size;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	596
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	597 return st.st_ino == st2.st_ino &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	598 CMP_DEV_T(st.st_dev, st2.st_dev) ? 0 : 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	599 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	600
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	601 static int squat_uidlist_lock(struct squat_uidlist *uidlist)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	602 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	603 int ret;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	604
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	605 for (;;) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	606 i_assert(uidlist->fd != -1);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	607 i_assert(uidlist->file_lock == NULL);
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	608 i_assert(uidlist->dotlock == NULL);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	609
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	610 if (uidlist->trie->lock_method != FILE_LOCK_METHOD_DOTLOCK) {
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	611 ret = file_wait_lock(uidlist->fd, uidlist->path,
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	612 F_WRLCK,
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	613 uidlist->trie->lock_method,
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	614 SQUAT_TRIE_LOCK_TIMEOUT,
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	615 &uidlist->file_lock);
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	616 } else {
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	617 ret = file_dotlock_create(&uidlist->trie->dotlock_set,
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	618 uidlist->path, 0,
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	619 &uidlist->dotlock);
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	620 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	621 if (ret == 0) {
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	622 i_error("squat uidlist %s: Locking timed out",
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	623 uidlist->path);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	624 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	625 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	626 if (ret < 0)
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	627 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	628
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	629 ret = squat_uidlist_is_file_stale(uidlist);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	630 if (ret == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	631 break;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	632
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	633 if (uidlist->file_lock != NULL)
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	634 file_unlock(&uidlist->file_lock);
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	635 else
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	636 file_dotlock_delete(&uidlist->dotlock);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	637 if (ret < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	638 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	639
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	640 squat_uidlist_close(uidlist);
8205 b62042ddb79e Squat: Create files with the correct mode and group. Timo Sirainen <tss@iki.fi> parents: 7390 diff changeset	641 uidlist->fd = squat_trie_create_fd(uidlist->trie,
b62042ddb79e Squat: Create files with the correct mode and group. Timo Sirainen <tss@iki.fi> parents: 7390 diff changeset	642 uidlist->path, 0);
b62042ddb79e Squat: Create files with the correct mode and group. Timo Sirainen <tss@iki.fi> parents: 7390 diff changeset	643 if (uidlist->fd == -1)
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	644 return -1;
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	645 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	646 return 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	647 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	648
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	649 static int squat_uidlist_open_or_create(struct squat_uidlist *uidlist)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	650 {
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	651 int ret;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	652
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	653 if (uidlist->fd == -1) {
8205 b62042ddb79e Squat: Create files with the correct mode and group. Timo Sirainen <tss@iki.fi> parents: 7390 diff changeset	654 uidlist->fd = squat_trie_create_fd(uidlist->trie,
b62042ddb79e Squat: Create files with the correct mode and group. Timo Sirainen <tss@iki.fi> parents: 7390 diff changeset	655 uidlist->path, 0);
b62042ddb79e Squat: Create files with the correct mode and group. Timo Sirainen <tss@iki.fi> parents: 7390 diff changeset	656 if (uidlist->fd == -1)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	657 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	658 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	659 if (squat_uidlist_lock(uidlist) <= 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	660 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	661
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	662 if (uidlist->locked_file_size != 0) {
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	663 if ((ret = squat_uidlist_map(uidlist)) < 0)
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	664 return -1;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	665 if (ret == 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	666 /* broken file, truncate */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	667 if (ftruncate(uidlist->fd, 0) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	668 i_error("ftruncate(%s) failed: %m",
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	669 uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	670 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	671 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	672 uidlist->locked_file_size = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	673 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	674 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	675 if (uidlist->locked_file_size == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	676 /* write using 0 until we're finished */
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	677 memset(&uidlist->hdr, 0, sizeof(uidlist->hdr));
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	678 if (write_full(uidlist->fd, &uidlist->hdr,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	679 sizeof(uidlist->hdr)) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	680 i_error("write(%s) failed: %m", uidlist->path);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	681 return -1;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	682 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	683 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	684 return 0;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	685 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	686
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	687 int squat_uidlist_build_init(struct squat_uidlist *uidlist,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	688 struct squat_uidlist_build_context **ctx_r)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	689 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	690 struct squat_uidlist_build_context *ctx;
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	691 int ret;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	692
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	693 i_assert(!uidlist->building);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	694
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	695 ret = squat_uidlist_open_or_create(uidlist);
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	696 if (ret == 0 &&
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	697 lseek(uidlist->fd, uidlist->hdr.used_file_size, SEEK_SET) < 0) {
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	698 i_error("lseek(%s) failed: %m", uidlist->path);
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	699 ret = -1;
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	700 }
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	701
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	702 if (ret < 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	703 if (uidlist->file_lock != NULL)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	704 file_unlock(&uidlist->file_lock);
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	705 if (uidlist->dotlock != NULL)
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	706 file_dotlock_delete(&uidlist->dotlock);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	707 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	708 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	709
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	710 ctx = i_new(struct squat_uidlist_build_context, 1);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	711 ctx->uidlist = uidlist;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	712 ctx->output = o_stream_create_fd(uidlist->fd, 0, FALSE);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	713 if (ctx->output->offset == 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	714 struct squat_uidlist_file_header hdr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	715
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	716 memset(&hdr, 0, sizeof(hdr));
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	717 o_stream_send(ctx->output, &hdr, sizeof(hdr));
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	718 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	719 o_stream_cork(ctx->output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	720 i_array_init(&ctx->lists, 10240);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	721 i_array_init(&ctx->block_offsets, 128);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	722 i_array_init(&ctx->block_end_indexes, 128);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	723 ctx->list_start_idx = uidlist->hdr.count;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	724 ctx->build_hdr = uidlist->hdr;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	725
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	726 uidlist->building = TRUE;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	727 *ctx_r = ctx;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	728 return 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	729 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	730
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	731 static void
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	732 uidlist_write_block_list_and_header(struct squat_uidlist_build_context *ctx,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	733 struct ostream *output,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	734 ARRAY_TYPE(uint32_t) *block_offsets,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	735 ARRAY_TYPE(uint32_t) *block_end_indexes,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	736 bool write_old_blocks)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	737 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	738 struct squat_uidlist *uidlist = ctx->uidlist;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	739 unsigned int align, old_block_count, new_block_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	740 uint32_t block_offset_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	741 uoff_t block_list_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	742
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	743 i_assert(uidlist->trie->hdr.indexid != 0);
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	744 ctx->build_hdr.indexid = uidlist->trie->hdr.indexid;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	745
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	746 if (array_count(block_end_indexes) == 0) {
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	747 ctx->build_hdr.used_file_size = output->offset;
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7218 diff changeset	748 ctx->build_hdr.block_list_offset = 0;
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	749 uidlist->hdr = ctx->build_hdr;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	750 return;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	751 }
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	752
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	753 align = output->offset % sizeof(uint32_t);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	754 if (align != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	755 static char null[sizeof(uint32_t)-1] = { 0, };
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	756
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	757 o_stream_send(output, null, sizeof(uint32_t) - align);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	758 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	759 block_list_offset = output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	760
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	761 new_block_count = array_count(block_offsets);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	762 old_block_count = write_old_blocks ? uidlist->cur_block_count : 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	763
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	764 block_offset_count = new_block_count + old_block_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	765 o_stream_send(output, &block_offset_count, sizeof(block_offset_count));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	766 /* write end indexes */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	767 o_stream_send(output, uidlist->cur_block_end_indexes,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	768 old_block_count * sizeof(uint32_t));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	769 o_stream_send(output, array_idx(block_end_indexes, 0),
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	770 new_block_count * sizeof(uint32_t));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	771 /* write offsets */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	772 o_stream_send(output, uidlist->cur_block_offsets,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	773 old_block_count * sizeof(uint32_t));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	774 o_stream_send(output, array_idx(block_offsets, 0),
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	775 new_block_count * sizeof(uint32_t));
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	776 o_stream_flush(output);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	777
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	778 /* update header - it's written later when trie is locked */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	779 ctx->build_hdr.block_list_offset = block_list_offset;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	780 ctx->build_hdr.used_file_size = output->offset;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	781 uidlist->hdr = ctx->build_hdr;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	782 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	783
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	784 void squat_uidlist_build_flush(struct squat_uidlist_build_context *ctx)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	785 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	786 struct uidlist_list *lists;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	787 uint8_t buf[SQUAT_PACK_MAX_SIZE], *bufp;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	788 unsigned int i, j, count, max;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	789 uint32_t block_offset, block_end_idx, start_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	790 uint32_t list_sizes[UIDLIST_BLOCK_LIST_COUNT];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	791 size_t mem_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	792
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	793 if (ctx->uidlist->corrupted)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	794 return;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	795
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	796 lists = array_get_modifiable(&ctx->lists, &count);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	797 if (count == 0)
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	798 return;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	799
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	800 /* write the lists and save the written sizes to uid_list[0] */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	801 for (i = 0; i < count; i += UIDLIST_BLOCK_LIST_COUNT) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	802 start_offset = ctx->output->offset;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	803 max = I_MIN(count - i, UIDLIST_BLOCK_LIST_COUNT);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	804 for (j = 0; j < max; j++) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	805 if (uidlist_write(ctx->output, &lists[i+j],
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	806 FALSE, &list_sizes[j]) < 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	807 squat_uidlist_set_corrupted(ctx->uidlist,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	808 "Broken uidlists");
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	809 return;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	810 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	811 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	812
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	813 block_offset = ctx->output->offset;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	814 block_end_idx = ctx->list_start_idx + i + max;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	815 array_append(&ctx->block_offsets, &block_offset, 1);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	816 array_append(&ctx->block_end_indexes, &block_end_idx, 1);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	817
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	818 /* write the full size of the uidlists */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	819 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	820 squat_pack_num(&bufp, block_offset - start_offset);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	821 o_stream_send(ctx->output, buf, bufp - buf);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	822
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	823 /* write the sizes/flags */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	824 for (j = 0; j < max; j++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	825 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	826 squat_pack_num(&bufp, list_sizes[j]);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	827 o_stream_send(ctx->output, buf, bufp - buf);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	828 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	829 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	830
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	831 mem_size = ctx->lists.arr.buffer->used +
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	832 ctx->block_offsets.arr.buffer->used +
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	833 ctx->block_end_indexes.arr.buffer->used;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	834 if (ctx->uidlist->max_size < mem_size)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	835 ctx->uidlist->max_size = mem_size;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	836
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	837 ctx->list_start_idx += count;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	838 array_clear(&ctx->lists);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	839
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	840 uidlist_write_block_list_and_header(ctx, ctx->output,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	841 &ctx->block_offsets,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	842 &ctx->block_end_indexes, TRUE);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	843
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	844 (void)squat_uidlist_map(ctx->uidlist);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	845
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	846 array_clear(&ctx->block_offsets);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	847 array_clear(&ctx->block_end_indexes);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	848 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	849
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	850 int squat_uidlist_build_finish(struct squat_uidlist_build_context *ctx)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	851 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	852 if (ctx->uidlist->corrupted)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	853 return -1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	854
7369 fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	855 if (!ctx->output->closed) {
fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	856 o_stream_seek(ctx->output, 0);
fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	857 o_stream_send(ctx->output,
fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	858 &ctx->build_hdr, sizeof(ctx->build_hdr));
fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	859 o_stream_seek(ctx->output, ctx->build_hdr.used_file_size);
fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	860 o_stream_flush(ctx->output);
fc296dfcfd29 If uidlist writing fails, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 7226 diff changeset	861 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	862
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	863 if (ctx->output->last_failed_errno != 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	864 errno = ctx->output->last_failed_errno;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	865 i_error("write() to %s failed: %m", ctx->uidlist->path);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	866 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	867 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	868 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	869 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	870
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	871 void squat_uidlist_build_deinit(struct squat_uidlist_build_context **_ctx)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	872 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	873 struct squat_uidlist_build_context ctx = _ctx;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	874
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	875 *_ctx = NULL;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	876
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	877 i_assert(array_count(&ctx->lists) == 0 \|\| ctx->uidlist->corrupted);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	878 i_assert(ctx->uidlist->building);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	879 ctx->uidlist->building = FALSE;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	880
7371 55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	881 if (ctx->uidlist->file_lock != NULL)
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	882 file_unlock(&ctx->uidlist->file_lock);
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	883 else
55712d36224b Added support for dotlocks. Timo Sirainen <tss@iki.fi> parents: 7370 diff changeset	884 file_dotlock_delete(&ctx->uidlist->dotlock);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	885
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	886 if (ctx->need_reopen)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	887 (void)squat_uidlist_open(ctx->uidlist);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	888
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	889 array_free(&ctx->block_offsets);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	890 array_free(&ctx->block_end_indexes);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	891 array_free(&ctx->lists);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	892 o_stream_unref(&ctx->output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	893 i_free(ctx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	894 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	895
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	896 int squat_uidlist_rebuild_init(struct squat_uidlist_build_context *build_ctx,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	897 bool compress,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	898 struct squat_uidlist_rebuild_context **ctx_r)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	899 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	900 struct squat_uidlist_rebuild_context *ctx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	901 struct squat_uidlist_file_header hdr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	902 const char *temp_path;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	903 int fd;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	904
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	905 if (build_ctx->build_hdr.link_count == 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	906 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	907
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	908 if (!compress) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	909 if (build_ctx->build_hdr.link_count <
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	910 build_ctx->build_hdr.count*2/3)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	911 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	912 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	913
7373 4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	914 /* make sure the entire uidlist is in memory before beginning,
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	915 otherwise the pages are faulted to memory in random order which
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	916 takes forever. */
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	917 if (squat_uidlist_read_to_memory(build_ctx->uidlist) < 0)
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	918 return -1;
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	919
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	920 temp_path = t_strconcat(build_ctx->uidlist->path, ".tmp", NULL);
8205 b62042ddb79e Squat: Create files with the correct mode and group. Timo Sirainen <tss@iki.fi> parents: 7390 diff changeset	921 fd = squat_trie_create_fd(build_ctx->uidlist->trie, temp_path, O_TRUNC);
b62042ddb79e Squat: Create files with the correct mode and group. Timo Sirainen <tss@iki.fi> parents: 7390 diff changeset	922 if (fd == -1)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	923 return -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	924
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	925 ctx = i_new(struct squat_uidlist_rebuild_context, 1);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	926 ctx->uidlist = build_ctx->uidlist;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	927 ctx->build_ctx = build_ctx;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	928 ctx->fd = fd;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	929 ctx->output = o_stream_create_fd(ctx->fd, 0, FALSE);
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	930 ctx->next_uid_list_idx = 0x100;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	931 o_stream_cork(ctx->output);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	932
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	933 memset(&hdr, 0, sizeof(hdr));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	934 o_stream_send(ctx->output, &hdr, sizeof(hdr));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	935
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	936 ctx->cur_block_start_offset = ctx->output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	937 i_array_init(&ctx->new_block_offsets,
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	938 build_ctx->build_hdr.count / UIDLIST_BLOCK_LIST_COUNT);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	939 i_array_init(&ctx->new_block_end_indexes,
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	940 build_ctx->build_hdr.count / UIDLIST_BLOCK_LIST_COUNT);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	941 *ctx_r = ctx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	942 return 1;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	943 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	944
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	945 static void
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	946 uidlist_rebuild_flush_block(struct squat_uidlist_rebuild_context *ctx)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	947 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	948 uint8_t buf[SQUAT_PACK_MAX_SIZE], *bufp;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	949 uint32_t block_offset, block_end_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	950 unsigned int i;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	951
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	952 ctx->new_count += ctx->list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	953
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	954 block_offset = ctx->output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	955 block_end_idx = ctx->new_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	956 array_append(&ctx->new_block_offsets, &block_offset, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	957 array_append(&ctx->new_block_end_indexes, &block_end_idx, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	958
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	959 /* this block's contents started from cur_block_start_offset and
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	960 ended to current offset. write the size of this area. */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	961 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	962 squat_pack_num(&bufp, block_offset - ctx->cur_block_start_offset);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	963 o_stream_send(ctx->output, buf, bufp - buf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	964
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	965 /* write the sizes/flags */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	966 for (i = 0; i < ctx->list_idx; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	967 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	968 squat_pack_num(&bufp, ctx->list_sizes[i]);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	969 o_stream_send(ctx->output, buf, bufp - buf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	970 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	971 ctx->cur_block_start_offset = ctx->output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	972 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	973
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	974 uint32_t squat_uidlist_rebuild_next(struct squat_uidlist_rebuild_context *ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	975 const ARRAY_TYPE(uint32_t) *uids)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	976 {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	977 int ret;
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	978
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	979 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	980 ret = uidlist_write_array(ctx->output, array_idx(uids, 0),
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	981 array_count(uids), 0, 0, FALSE,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	982 &ctx->list_sizes[ctx->list_idx]);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	983 } T_END;
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	984 if (ret < 0)
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	985 squat_uidlist_set_corrupted(ctx->uidlist, "Broken uidlists");
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	986
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	987 if (++ctx->list_idx == UIDLIST_BLOCK_LIST_COUNT) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	988 uidlist_rebuild_flush_block(ctx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	989 ctx->list_idx = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	990 }
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	991 return ctx->next_uid_list_idx++ << 1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	992 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	993
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	994 uint32_t squat_uidlist_rebuild_nextu(struct squat_uidlist_rebuild_context *ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	995 const ARRAY_TYPE(seq_range) *uids)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	996 {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	997 const struct seq_range *range;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	998 ARRAY_TYPE(uint32_t) tmp_uids;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	999 uint32_t seq, uid1, ret;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1000 unsigned int i, count;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1001
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1002 range = array_get(uids, &count);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1003 if (count == 0)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1004 return 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1005
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1006 if (range[count-1].seq2 < 8) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1007 /* we can use a singleton bitmask */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1008 ret = 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1009 for (i = 0; i < count; i++) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1010 for (seq = range[i].seq1; seq <= range[i].seq2; seq++)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1011 ret \|= 1 << (seq+1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1012 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1013 return ret;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1014 }
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7218 diff changeset	1015 if (count == 1 && range[0].seq1 == range[0].seq2) {
21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7218 diff changeset	1016 /* single UID */
21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7218 diff changeset	1017 return (range[0].seq1 << 1) \| 1;
21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7218 diff changeset	1018 }
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1019
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1020 /* convert seq range to our internal representation and use the
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1021 normal _rebuild_next() to write it */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1022 i_array_init(&tmp_uids, 128);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1023 for (i = 0; i < count; i++) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1024 if (range[i].seq1 == range[i].seq2)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1025 array_append(&tmp_uids, &range[i].seq1, 1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1026 else {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1027 uid1 = range[i].seq1 \| UID_LIST_MASK_RANGE;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1028 array_append(&tmp_uids, &uid1, 1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1029 array_append(&tmp_uids, &range[i].seq2, 1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1030 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1031 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1032 ret = squat_uidlist_rebuild_next(ctx, &tmp_uids);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1033 array_free(&tmp_uids);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1034 return ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1035 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1036
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1037 int squat_uidlist_rebuild_finish(struct squat_uidlist_rebuild_context *ctx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1038 bool cancel)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1039 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1040 const char *temp_path;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1041 int ret = 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1042
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1043 if (ctx->list_idx != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1044 uidlist_rebuild_flush_block(ctx);
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1045 if (cancel \|\| ctx->uidlist->corrupted)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1046 ret = 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1047
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1048 temp_path = t_strconcat(ctx->uidlist->path, ".tmp", NULL);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1049 if (ret > 0) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1050 ctx->build_ctx->build_hdr.indexid =
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1051 ctx->uidlist->trie->hdr.indexid;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1052 ctx->build_ctx->build_hdr.count = ctx->new_count;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1053 ctx->build_ctx->build_hdr.link_count = 0;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1054 uidlist_write_block_list_and_header(ctx->build_ctx, ctx->output,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1055 &ctx->new_block_offsets,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1056 &ctx->new_block_end_indexes,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1057 FALSE);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1058 o_stream_seek(ctx->output, 0);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1059 o_stream_send(ctx->output, &ctx->build_ctx->build_hdr,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1060 sizeof(ctx->build_ctx->build_hdr));
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1061 o_stream_seek(ctx->output,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1062 ctx->build_ctx->build_hdr.used_file_size);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1063 o_stream_flush(ctx->output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1064
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1065 if (ctx->uidlist->corrupted)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1066 ret = -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1067 else if (ctx->output->last_failed_errno != 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1068 errno = ctx->output->last_failed_errno;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1069 i_error("write() to %s failed: %m", temp_path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1070 ret = -1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1071 } else if (rename(temp_path, ctx->uidlist->path) < 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1072 i_error("rename(%s, %s) failed: %m",
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1073 temp_path, ctx->uidlist->path);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1074 ret = -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1075 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1076 ctx->build_ctx->need_reopen = TRUE;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1077 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1078
7373 4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	1079 /* we no longer require the entire uidlist to be in memory,
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	1080 let it be used for something more useful. */
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	1081 squat_uidlist_free_from_memory(ctx->uidlist);
4445415da4ff When rebuilding uidlist, read the existing one entirely into memory at the Timo Sirainen <tss@iki.fi> parents: 7371 diff changeset	1082
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1083 o_stream_unref(&ctx->output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1084 if (close(ctx->fd) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1085 i_error("close(%s) failed: %m", temp_path);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1086
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1087 if (ret <= 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1088 if (unlink(temp_path) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1089 i_error("unlink(%s) failed: %m", temp_path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1090 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1091 array_free(&ctx->new_block_offsets);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1092 array_free(&ctx->new_block_end_indexes);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1093 i_free(ctx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1094 return ret < 0 ? -1 : 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1095 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1096
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1097 static void
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1098 uidlist_flush(struct squat_uidlist_build_context *ctx,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1099 struct uidlist_list *list, uint32_t uid)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1100 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1101 uint32_t size, offset = ctx->output->offset;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1102
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1103 ctx->build_hdr.link_count++;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1104 if (uidlist_write(ctx->output, list, TRUE, &size) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1105 squat_uidlist_set_corrupted(ctx->uidlist, "Broken uidlists");
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1106
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1107 list->uid_count = 2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1108 list->uid_begins_with_pointer = TRUE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1109
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1110 list->uid_list[0] = offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1111 list->uid_list[1] = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1112 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1113
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1114 static struct uidlist_list *
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1115 uidlist_add_new(struct squat_uidlist_build_context *ctx, unsigned int count,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1116 uint32_t *uid_list_idx_r)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1117 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1118 struct uidlist_list *list;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1119
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1120 i_assert(array_count(&ctx->lists) +
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1121 ctx->list_start_idx == ctx->build_hdr.count);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1122 *uid_list_idx_r = (ctx->build_hdr.count + 0x100) << 1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1123 list = array_append_space(&ctx->lists);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1124 ctx->build_hdr.count++;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1125
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1126 list->uid_count = count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1127 return list;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1128 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1129
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1130 uint32_t squat_uidlist_build_add_uid(struct squat_uidlist_build_context *ctx,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1131 uint32_t uid_list_idx, uint32_t uid)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1132 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1133 struct uidlist_list *list;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1134 unsigned int idx, mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1135 uint32_t *p;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1136
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1137 if ((uid_list_idx & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1138 /* adding second UID */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1139 uint32_t prev_uid = uid_list_idx >> 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1140
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1141 i_assert(prev_uid != uid);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1142 list = uidlist_add_new(ctx, 2, &uid_list_idx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1143 list->uid_list[0] = prev_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1144 if (prev_uid + 1 == uid)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1145 list->uid_list[0] \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1146 list->uid_list[1] = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1147 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1148 } else if (uid_list_idx < (0x100 << 1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1149 uint32_t old_list_idx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1150
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1151 if (uid < 8) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1152 /* UID lists containing only UIDs 0-7 are saved as
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1153 uidlist values 2..511. think of it as a bitmask. */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1154 uid_list_idx \|= 1 << (uid + 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1155 i_assert((uid_list_idx & 1) == 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1156 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1157 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1158
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1159 if (uid_list_idx == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1160 /* first UID */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1161 return (uid << 1) \| 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1162 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1163
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1164 /* create a new list */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1165 old_list_idx = uid_list_idx >> 1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1166 list = uidlist_add_new(ctx, 1, &uid_list_idx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1167 /* add the first UID ourself */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1168 idx = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1169 i_assert((old_list_idx & 0xff) != 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1170 for (mask = 1; mask <= 128; mask <<= 1, idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1171 if ((old_list_idx & mask) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1172 list->uid_list[0] = idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1173 idx++; mask <<= 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1174 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1175 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1176 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1177 for (; mask <= 128; mask <<= 1, idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1178 if ((old_list_idx & mask) != 0) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1179 squat_uidlist_build_add_uid(ctx,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1180 uid_list_idx, idx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1181 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1182 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1183 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1184
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1185 /* add to existing list */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1186 idx = (uid_list_idx >> 1) - 0x100;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1187 if (idx < ctx->list_start_idx) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1188 list = uidlist_add_new(ctx, 2, &uid_list_idx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1189 list->uid_list[0] = UID_LIST_POINTER_MASK_LIST_IDX \| idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1190 list->uid_list[1] = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1191 list->uid_begins_with_pointer = TRUE;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1192 ctx->build_hdr.link_count++;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1193 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1194 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1195
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1196 idx -= ctx->list_start_idx;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1197 if (idx >= array_count(&ctx->lists)) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1198 squat_uidlist_set_corrupted(ctx->uidlist,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1199 "missing/broken uidlist");
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1200 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1201 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1202 list = array_idx_modifiable(&ctx->lists, idx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1203 i_assert(list->uid_count > 0);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1204
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1205 p = &list->uid_list[list->uid_count-1];
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1206 i_assert(uid != *p \|\| ctx->uidlist->corrupted \|\|
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1207 (list->uid_count == 1 && list->uid_begins_with_pointer));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1208 if (uid == *p + 1 &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1209 (list->uid_count > 1 \|\| !list->uid_begins_with_pointer)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1210 /* use a range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1211 if (list->uid_count > 1 && (p[-1] & UID_LIST_MASK_RANGE) != 0 &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1212 (list->uid_count > 2 \|\| !list->uid_begins_with_pointer)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1213 /* increase the existing range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1214 *p += 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1215 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1216 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1217
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1218 if (list->uid_count == UIDLIST_LIST_SIZE) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1219 uidlist_flush(ctx, list, uid);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1220 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1221 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1222 /* create a new range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1223 *p \|= UID_LIST_MASK_RANGE;
7370 d51565e31ceb Minor optimization and possible compile fix. Timo Sirainen <tss@iki.fi> parents: 7369 diff changeset	1224 } else {
d51565e31ceb Minor optimization and possible compile fix. Timo Sirainen <tss@iki.fi> parents: 7369 diff changeset	1225 if (list->uid_count == UIDLIST_LIST_SIZE) {
d51565e31ceb Minor optimization and possible compile fix. Timo Sirainen <tss@iki.fi> parents: 7369 diff changeset	1226 uidlist_flush(ctx, list, uid);
d51565e31ceb Minor optimization and possible compile fix. Timo Sirainen <tss@iki.fi> parents: 7369 diff changeset	1227 return uid_list_idx;
d51565e31ceb Minor optimization and possible compile fix. Timo Sirainen <tss@iki.fi> parents: 7369 diff changeset	1228 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1229 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1230
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1231 p++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1232 list->uid_count++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1233
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1234 *p = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1235 return uid_list_idx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1236 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1237
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1238 static void uidlist_array_append(ARRAY_TYPE(uint32_t) *uids, uint32_t uid)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1239 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1240 uint32_t *uidlist;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1241 unsigned int count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1242
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1243 uidlist = array_get_modifiable(uids, &count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1244 if (count == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1245 array_append(uids, &uid, 1);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1246 return;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1247 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1248 if (uidlist[count-1] + 1 == uid) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1249 if (count > 1 && (uidlist[count-2] &
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1250 UID_LIST_MASK_RANGE) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1251 uidlist[count-1]++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1252 return;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1253 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1254 uidlist[count-1] \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1255 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1256 array_append(uids, &uid, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1257 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1258
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1259 static void uidlist_array_append_range(ARRAY_TYPE(uint32_t) *uids,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1260 uint32_t uid1, uint32_t uid2)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1261 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1262 uint32_t *uidlist;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1263 unsigned int count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1264
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1265 i_assert(uid1 < uid2);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1266
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1267 uidlist = array_get_modifiable(uids, &count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1268 if (count == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1269 uid1 \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1270 array_append(uids, &uid1, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1271 array_append(uids, &uid2, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1272 return;
4933 44a0e7606f7a Use seq_range_array_remove_range() instead of slowly iterating ourself. Timo Sirainen <tss@iki.fi> parents: 4931 diff changeset	1273 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1274 if (uidlist[count-1] + 1 == uid1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1275 if (count > 1 && (uidlist[count-2] &
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1276 UID_LIST_MASK_RANGE) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1277 uidlist[count-1] = uid2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1278 return;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1279 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1280 uidlist[count-1] \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1281 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1282 uid1 \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1283 array_append(uids, &uid1, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1284 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1285 array_append(uids, &uid2, 1);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1286 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1287
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1288 static int
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1289 squat_uidlist_get_at_offset(struct squat_uidlist *uidlist, uoff_t offset,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1290 uint32_t num, ARRAY_TYPE(uint32_t) *uids)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1291 {
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1292 const uint32_t *uid_list;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1293 const uint8_t p, end;
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1294 uint32_t size, base_uid, next_uid, flags, prev;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1295 uoff_t uidlist_data_offset;
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1296 unsigned int i, j, count;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1297
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1298 if (num != 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1299 uidlist_data_offset = offset;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1300 else {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1301 /* not given, read it */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1302 if (uidlist_file_cache_read(uidlist, offset,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1303 SQUAT_PACK_MAX_SIZE) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1304 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1305
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1306 p = CONST_PTR_OFFSET(uidlist->data, offset);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1307 end = CONST_PTR_OFFSET(uidlist->data, uidlist->data_size);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1308 num = squat_unpack_num(&p, end);
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1309 uidlist_data_offset = p - (const uint8_t *)uidlist->data;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1310 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1311 size = num >> 2;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1312
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1313 if (uidlist_file_cache_read(uidlist, uidlist_data_offset, size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1314 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1315 if (uidlist_data_offset + size > uidlist->data_size) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1316 squat_uidlist_set_corrupted(uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1317 "size points outside file");
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1318 return -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1319 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1320
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1321 p = CONST_PTR_OFFSET(uidlist->data, uidlist_data_offset);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1322 end = p + size;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1323
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1324 flags = num;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1325 if ((flags & UIDLIST_PACKED_FLAG_BEGINS_WITH_POINTER) != 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1326 /* link to the file */
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1327 prev = squat_unpack_num(&p, end);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1328
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1329 if ((prev & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1330 /* pointer to uidlist */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1331 prev = ((prev >> 1) + 0x100) << 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1332 if (squat_uidlist_get(uidlist, prev, uids) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1333 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1334 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1335 prev = offset - (prev >> 1);
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1336 if (squat_uidlist_get_at_offset(uidlist, prev,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1337 0, uids) < 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1338 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1339 }
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1340 uid_list = array_get(uids, &count);
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1341 next_uid = count == 0 ? 0 : uid_list[count-1] + 1;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1342 } else {
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1343 next_uid = 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1344 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1345
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1346 num = base_uid = squat_unpack_num(&p, end);
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1347 if ((flags & UIDLIST_PACKED_FLAG_BITMASK) == 0)
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1348 base_uid >>= 1;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1349 if (base_uid < next_uid) {
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1350 squat_uidlist_set_corrupted(uidlist,
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1351 "broken continued uidlist");
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1352 return -1;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1353 }
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1354
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1355 if ((flags & UIDLIST_PACKED_FLAG_BITMASK) != 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1356 /* bitmask */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1357 size = end - p;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1358
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1359 uidlist_array_append(uids, base_uid++);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1360 for (i = 0; i < size; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1361 for (j = 0; j < 8; j++, base_uid++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1362 if ((p[i] & (1 << j)) != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1363 uidlist_array_append(uids, base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1364 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1365 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1366 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1367 /* range */
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1368 for (;;) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1369 if ((num & 1) == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1370 uidlist_array_append(uids, base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1371 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1372 /* range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1373 uint32_t seq1 = base_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1374 base_uid += squat_unpack_num(&p, end) + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1375 uidlist_array_append_range(uids, seq1,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1376 base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1377 }
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1378 if (p == end)
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1379 break;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1380
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1381 num = squat_unpack_num(&p, end);
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1382 base_uid += (num >> 1) + 1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1383 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1384 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1385 return 0;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1386 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1387
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1388 static int uint32_cmp(const void key, const void data)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1389 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1390 const uint32_t i1 = key, i2 = data;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1391
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1392 return (int)i1 - (int)i2;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1393 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1394
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1395 static int
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1396 squat_uidlist_get_offset(struct squat_uidlist *uidlist, uint32_t uid_list_idx,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1397 uint32_t offset_r, uint32_t num_r)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1398 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1399 const uint8_t p, end;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1400 unsigned int idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1401 uint32_t num, skip_bytes, uidlists_offset;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1402 size_t max_map_size;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1403
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1404 if (uidlist->fd == -1) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1405 squat_uidlist_set_corrupted(uidlist, "no uidlists");
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1406 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1407 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1408
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1409 if (bsearch_insert_pos(&uid_list_idx, uidlist->cur_block_end_indexes,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1410 uidlist->cur_block_count,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1411 sizeof(uint32_t), uint32_cmp, &idx))
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1412 idx++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1413 if (unlikely(idx == uidlist->cur_block_count)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1414 squat_uidlist_set_corrupted(uidlist, "uidlist not found");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1415 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1416 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1417 if (unlikely(idx > 0 &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1418 uidlist->cur_block_end_indexes[idx-1] > uid_list_idx)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1419 squat_uidlist_set_corrupted(uidlist, "broken block list");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1420 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1421 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1422
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1423 /* make sure everything is mapped */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1424 uid_list_idx -= idx == 0 ? 0 : uidlist->cur_block_end_indexes[idx-1];
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1425 max_map_size = SQUAT_PACK_MAX_SIZE * (1+uid_list_idx);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1426 if (uidlist_file_cache_read(uidlist, uidlist->cur_block_offsets[idx],
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1427 max_map_size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1428 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1429
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1430 /* find the uidlist inside the block */
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1431 p = CONST_PTR_OFFSET(uidlist->data, uidlist->cur_block_offsets[idx]);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1432 end = CONST_PTR_OFFSET(uidlist->data, uidlist->data_size);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1433
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1434 uidlists_offset = uidlist->cur_block_offsets[idx] -
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1435 squat_unpack_num(&p, end);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1436 for (skip_bytes = 0; uid_list_idx > 0; uid_list_idx--) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1437 num = squat_unpack_num(&p, end);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1438 skip_bytes += num >> 2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1439 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1440 *offset_r = uidlists_offset + skip_bytes;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1441 *num_r = squat_unpack_num(&p, end);
4931 59ce1d6c0b8c Actually the code wasn't unneeded, put it back and actually execute it now. Timo Sirainen <tss@iki.fi> parents: 4930 diff changeset	1442
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1443 if (unlikely(p == end)) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1444 squat_uidlist_set_corrupted(uidlist, "broken file");
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1445 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1446 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1447 if (unlikely(*offset_r > uidlist->mmap_size &&
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1448 uidlist->mmap_base != NULL)) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1449 squat_uidlist_set_corrupted(uidlist, "broken offset");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1450 return -1;
4931 59ce1d6c0b8c Actually the code wasn't unneeded, put it back and actually execute it now. Timo Sirainen <tss@iki.fi> parents: 4930 diff changeset	1451 }
59ce1d6c0b8c Actually the code wasn't unneeded, put it back and actually execute it now. Timo Sirainen <tss@iki.fi> parents: 4930 diff changeset	1452 return 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1453 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1454
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1455 int squat_uidlist_get(struct squat_uidlist *uidlist, uint32_t uid_list_idx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1456 ARRAY_TYPE(uint32_t) *uids)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1457 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1458 unsigned int mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1459 uint32_t uid, offset, num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1460
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1461 if ((uid_list_idx & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1462 /* single UID */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1463 uid = uid_list_idx >> 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1464 uidlist_array_append(uids, uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1465 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1466 } else if (uid_list_idx < (0x100 << 1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1467 /* bitmask */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1468 for (uid = 0, mask = 2; mask <= 256; mask <<= 1, uid++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1469 if ((uid_list_idx & mask) != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1470 uidlist_array_append(uids, uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1471 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1472 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1473 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1474
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1475 uid_list_idx = (uid_list_idx >> 1) - 0x100;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1476 if (squat_uidlist_get_offset(uidlist, uid_list_idx, &offset, &num) < 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1477 return -1;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1478 return squat_uidlist_get_at_offset(uidlist, offset, num, uids);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1479 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1480
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1481 uint32_t squat_uidlist_singleton_last_uid(uint32_t uid_list_idx)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1482 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1483 unsigned int idx, mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1484
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1485 if ((uid_list_idx & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1486 /* single UID */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1487 return uid_list_idx >> 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1488 } else if (uid_list_idx < (0x100 << 1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1489 /* bitmask */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1490 if (uid_list_idx == 2) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1491 /* just a quick optimization */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1492 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1493 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1494 for (idx = 7, mask = 256; mask > 2; mask >>= 1, idx--) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1495 if ((uid_list_idx & mask) != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1496 return idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1497 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1498 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1499
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1500 i_unreached();
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1501 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1502 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1503
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1504 int squat_uidlist_get_seqrange(struct squat_uidlist *uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1505 uint32_t uid_list_idx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1506 ARRAY_TYPE(seq_range) *seq_range_arr)
4886 ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	1507 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1508 ARRAY_TYPE(uint32_t) tmp_uid_arr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1509 struct seq_range range;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1510 const uint32_t *tmp_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1511 unsigned int i, count;
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1512 int ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1513
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1514 i_array_init(&tmp_uid_arr, 128);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1515 ret = squat_uidlist_get(uidlist, uid_list_idx, &tmp_uid_arr);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1516 if (ret == 0) {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1517 tmp_uids = array_get(&tmp_uid_arr, &count);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1518 for (i = 0; i < count; i++) {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1519 if ((tmp_uids[i] & UID_LIST_MASK_RANGE) == 0)
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1520 range.seq1 = range.seq2 = tmp_uids[i];
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1521 else {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1522 range.seq1 = tmp_uids[i] & ~UID_LIST_MASK_RANGE;
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1523 range.seq2 = tmp_uids[++i];
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1524 }
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1525 array_append(seq_range_arr, &range, 1);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1526 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1527 }
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1528 array_free(&tmp_uid_arr);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1529 return ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1530 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1531
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1532 int squat_uidlist_filter(struct squat_uidlist *uidlist, uint32_t uid_list_idx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1533 ARRAY_TYPE(seq_range) *uids)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1534 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1535 const struct seq_range *parent_range;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1536 ARRAY_TYPE(seq_range) dest_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1537 ARRAY_TYPE(uint32_t) relative_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1538 const uint32_t *rel_range;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1539 unsigned int i, rel_count, parent_idx, parent_count, diff, parent_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1540 uint32_t prev_seq, seq1, seq2;
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1541 int ret = 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1542
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1543 parent_range = array_get(uids, &parent_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1544 if (parent_count == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1545 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1546
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1547 i_array_init(&relative_uids, 128);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1548 i_array_init(&dest_uids, 128);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1549 squat_uidlist_get(uidlist, uid_list_idx, &relative_uids);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1550
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1551 parent_idx = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1552 rel_range = array_get(&relative_uids, &rel_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1553 prev_seq = 0; parent_uid = parent_range[0].seq1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1554 for (i = 0; i < rel_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1555 if (unlikely(parent_uid == (uint32_t)-1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1556 i_error("broken UID ranges");
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1557 ret = -1;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1558 break;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1559 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1560 if ((rel_range[i] & UID_LIST_MASK_RANGE) == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1561 seq1 = seq2 = rel_range[i];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1562 else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1563 seq1 = (rel_range[i] & ~UID_LIST_MASK_RANGE);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1564 seq2 = rel_range[++i];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1565 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1566 i_assert(seq1 >= prev_seq);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1567 diff = seq1 - prev_seq;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1568 while (diff > 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1569 if (unlikely(parent_uid == (uint32_t)-1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1570 i_error("broken UID ranges");
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1571 ret = -1;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1572 break;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1573 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1574
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1575 for (; parent_idx < parent_count; parent_idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1576 if (parent_range[parent_idx].seq2 <= parent_uid)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1577 continue;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1578 if (parent_uid < parent_range[parent_idx].seq1)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1579 parent_uid = parent_range[parent_idx].seq1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1580 else
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1581 parent_uid++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1582 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1583 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1584 diff--;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1585 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1586 diff = seq2 - seq1 + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1587 while (diff > 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1588 if (unlikely(parent_uid == (uint32_t)-1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1589 i_error("broken UID ranges");
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1590 ret = -1;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1591 break;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1592 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1593 seq_range_array_add(&dest_uids, 0, parent_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1594 for (; parent_idx < parent_count; parent_idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1595 if (parent_range[parent_idx].seq2 <= parent_uid)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1596 continue;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1597 if (parent_uid < parent_range[parent_idx].seq1)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1598 parent_uid = parent_range[parent_idx].seq1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1599 else
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1600 parent_uid++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1601 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1602 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1603 diff--;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1604 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1605
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1606 prev_seq = seq2 + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1607 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1608
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1609 buffer_set_used_size(uids->arr.buffer, 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1610 array_append_array(uids, &dest_uids);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1611
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1612 array_free(&relative_uids);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1613 array_free(&dest_uids);
8850 79311c07f98f squat: error handling fix. Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	1614 return ret;
4886 ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	1615 }
ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	1616
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1617 size_t squat_uidlist_mem_used(struct squat_uidlist *uidlist,
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1618 unsigned int *count_r)
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1619 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1620 *count_r = uidlist->hdr.count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1621 return uidlist->max_size;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1622 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/plugins/fts-squat/squat-uidlist.c @ 9575:0a00dcc4f0ea HEAD