dovecot/original-hg/dovecot-1.2: src/plugins/fts-squat/squat-uidlist.c annotate

annotate src/plugins/fts-squat/squat-uidlist.c @ 7210:f5f77a3ae203 HEAD

Initial code to support expunging from squat indexes, probably still buggy.

author	Timo Sirainen <tss@iki.fi>
date	Sun, 03 Feb 2008 22:44:09 +0200
parents	7ed926ed7aa4
children	43d2f70fb279

rev	line source
7086 7ed926ed7aa4 Updated copyright notices to include year 2008. Timo Sirainen <tss@iki.fi> parents: 6945 diff changeset	1 /* Copyright (c) 2007-2008 Dovecot authors, see the included COPYING file */
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	2
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	3 #include "lib.h"
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	4 #include "array.h"
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	5 #include "bsearch-insert-pos.h"
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	6 #include "file-cache.h"
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	7 #include "file-lock.h"
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	8 #include "read-full.h"
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	9 #include "write-full.h"
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	10 #include "ostream.h"
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	11 #include "squat-trie-private.h"
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	12 #include "squat-uidlist.h"
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	13
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	14 #include <stdio.h>
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	15 #include <sys/stat.h>
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	16 #include <sys/mman.h>
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	17
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	18 #define UIDLIST_LIST_SIZE 31
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	19 #define UIDLIST_BLOCK_LIST_COUNT 100
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	20 #define UID_LIST_MASK_RANGE 0x80000000
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	21
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	22 /* set = points to uidlist index number, unset = points to uidlist offset */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	23 #define UID_LIST_POINTER_MASK_LIST_IDX 0x80000000
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	24
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	25 #define UIDLIST_PACKED_FLAG_BITMASK 1
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	26 #define UIDLIST_PACKED_FLAG_BEGINS_WITH_POINTER 2
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	27
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	28 struct uidlist_list {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	29 uint32_t uid_count:31;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	30 uint32_t uid_begins_with_pointer:1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	31 uint32_t uid_list[UIDLIST_LIST_SIZE];
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	32 };
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	33
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	34 struct squat_uidlist {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	35 struct squat_trie *trie;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	36
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	37 char *path;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	38 int fd;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	39 struct file_cache *file_cache;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	40
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	41 struct file_lock *file_lock;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	42 uoff_t locked_file_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	43
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	44 void *mmap_base;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	45 size_t mmap_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	46 struct squat_uidlist_file_header hdr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	47
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	48 const void *data;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	49 size_t data_size;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	50
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	51 unsigned int cur_block_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	52 const uint32_t *cur_block_offsets;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	53 const uint32_t *cur_block_end_indexes;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	54
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	55 size_t max_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	56 unsigned int corrupted:1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	57 unsigned int building:1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	58 };
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	59
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	60 struct squat_uidlist_build_context {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	61 struct squat_uidlist *uidlist;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	62 struct ostream *output;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	63
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	64 ARRAY_TYPE(uint32_t) block_offsets;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	65 ARRAY_TYPE(uint32_t) block_end_indexes;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	66
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	67 ARRAY_DEFINE(lists, struct uidlist_list);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	68 uint32_t list_start_idx;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	69
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	70 struct squat_uidlist_file_header build_hdr;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	71 unsigned int need_reopen:1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	72 };
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	73
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	74 struct squat_uidlist_rebuild_context {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	75 struct squat_uidlist *uidlist;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	76 struct squat_uidlist_build_context *build_ctx;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	77
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	78 int fd;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	79 struct ostream *output;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	80
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	81 ARRAY_TYPE(uint32_t) new_block_offsets, new_block_end_indexes;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	82 uoff_t cur_block_start_offset;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	83
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	84 uint32_t list_sizes[UIDLIST_BLOCK_LIST_COUNT];
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	85 uint32_t next_uid_list_idx;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	86 unsigned int list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	87 unsigned int new_count;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	88 };
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	89
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	90 static void squat_uidlist_close(struct squat_uidlist *uidlist);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	91
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	92 void squat_uidlist_delete(struct squat_uidlist *uidlist)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	93 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	94 if (unlink(uidlist->path) < 0 && errno != ENOENT)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	95 i_error("unlink(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	96 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	97
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	98 static void squat_uidlist_set_corrupted(struct squat_uidlist *uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	99 const char *reason)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	100 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	101 if (uidlist->corrupted)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	102 return;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	103 uidlist->corrupted = TRUE;
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	104
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	105 i_error("Corrupted squat uidlist file %s: %s", uidlist->path, reason);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	106 squat_trie_delete(uidlist->trie);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	107 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	108
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	109 static int
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	110 uidlist_write_array(struct ostream output, const uint32_t uid_list,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	111 unsigned int uid_count, uint32_t packed_flags,
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	112 uint32_t offset, bool write_size, uint32_t *size_r)
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	113 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	114 uint8_t uidbuf, bufp, sizebuf[SQUAT_PACK_MAX_SIZE], *sizebufp;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	115 uint8_t listbuf[SQUAT_PACK_MAX_SIZE], *listbufp = listbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	116 uint32_t uid, uid2, prev, base_uid, size_value;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	117 unsigned int i, bitmask_len, uid_list_len;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	118 unsigned int idx, max_idx, mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	119 bool datastack;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	120 int num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	121
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	122 if ((packed_flags & UIDLIST_PACKED_FLAG_BEGINS_WITH_POINTER) != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	123 squat_pack_num(&listbufp, offset);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	124
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	125 /* @UNSAFE */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	126 base_uid = uid_list[0] & ~UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	127 datastack = uid_count < 1024*8/SQUAT_PACK_MAX_SIZE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	128 if (datastack)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	129 uidbuf = t_malloc(SQUAT_PACK_MAX_SIZE * uid_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	130 else
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	131 uidbuf = i_malloc(SQUAT_PACK_MAX_SIZE * uid_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	132 bufp = uidbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	133 squat_pack_num(&bufp, base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	134
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	135 bitmask_len = (uid_list[uid_count-1] - base_uid + 7) / 8 +
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	136 (bufp - uidbuf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	137 if (bitmask_len < uid_count) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	138 bitmask_build:
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	139 i_assert(bitmask_len < SQUAT_PACK_MAX_SIZE*uid_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	140
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	141 memset(bufp, 0, bitmask_len - (bufp - uidbuf));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	142 if ((uid_list[0] & UID_LIST_MASK_RANGE) == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	143 i = 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	144 uid = i == uid_count ? 0 : uid_list[i];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	145 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	146 i = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	147 uid = uid_list[0] + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	148 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	149 base_uid++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	150
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	151 for (; i < uid_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	152 if ((uid & UID_LIST_MASK_RANGE) == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	153 uid -= base_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	154 uid2 = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	155 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	156 uid &= ~UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	157 uid -= base_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	158 uid2 = uid_list[i+1] - base_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	159 i++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	160 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	161
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	162 if (uid2 - uid < 3*8) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	163 for (; uid <= uid2; uid++)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	164 bufp[uid / 8] \|= 1 << (uid % 8);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	165 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	166 /* first byte */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	167 idx = uid / 8;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	168 num = uid % 8;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	169 if (num != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	170 uid += 8 - num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	171 for (mask = 0; num < 8; num++)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	172 mask \|= 1 << num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	173 bufp[idx++] \|= mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	174 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	175
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	176 /* middle bytes */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	177 num = uid2 % 8;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	178 max_idx = idx + (uid2 - num - uid)/8;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	179 for (; idx < max_idx; idx++, uid += 8)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	180 bufp[idx] = 0xff;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	181
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	182 /* last byte */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	183 for (mask = 0; num >= 0; num--)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	184 mask \|= 1 << num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	185 bufp[idx] \|= mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	186 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	187 uid = i+1 == uid_count ? 0 : uid_list[i+1];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	188 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	189 uid_list_len = bitmask_len;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	190 packed_flags \|= UIDLIST_PACKED_FLAG_BITMASK;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	191 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	192 bufp = uidbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	193 prev = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	194 for (i = 0; i < uid_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	195 uid = uid_list[i];
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	196 if (unlikely((uid & ~UID_LIST_MASK_RANGE) < prev))
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	197 return -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	198 if ((uid & UID_LIST_MASK_RANGE) == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	199 squat_pack_num(&bufp, (uid - prev) << 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	200 prev = uid + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	201 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	202 uid &= ~UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	203 squat_pack_num(&bufp, 1 \| (uid - prev) << 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	204 squat_pack_num(&bufp, uid_list[i+1] - uid - 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	205 prev = uid_list[i+1] + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	206 i++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	207 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	208 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	209 uid_list_len = bufp - uidbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	210 if (uid_list_len > bitmask_len) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	211 bufp = uidbuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	212 squat_pack_num(&bufp, base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	213 goto bitmask_build;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	214 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	215 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	216
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	217 size_value = ((uid_list_len +
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	218 (listbufp - listbuf)) << 2) \| packed_flags;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	219 if (write_size) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	220 sizebufp = sizebuf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	221 squat_pack_num(&sizebufp, size_value);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	222 o_stream_send(output, sizebuf, sizebufp - sizebuf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	223 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	224 o_stream_send(output, listbuf, listbufp - listbuf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	225 o_stream_send(output, uidbuf, uid_list_len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	226 if (!datastack)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	227 i_free(uidbuf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	228
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	229 *size_r = size_value;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	230 return 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	231 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	232
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	233 static int
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	234 uidlist_write(struct ostream output, const struct uidlist_list list,
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	235 bool write_size, uint32_t *size_r)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	236 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	237 const uint32_t *uid_list = list->uid_list;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	238 uint8_t buf[SQUAT_PACK_MAX_SIZE], *bufp;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	239 uint32_t uid_count = list->uid_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	240 uint32_t packed_flags = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	241 uint32_t offset = 0;
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	242 int ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	243
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	244 if (list->uid_begins_with_pointer) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	245 /* continued UID list */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	246 packed_flags \|= UIDLIST_PACKED_FLAG_BEGINS_WITH_POINTER;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	247 if ((uid_list[0] & UID_LIST_POINTER_MASK_LIST_IDX) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	248 offset = ((uid_list[0] & ~UID_LIST_POINTER_MASK_LIST_IDX) << 1) \| 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	249 if (list->uid_count == 1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	250 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	251 squat_pack_num(&bufp, offset);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	252 o_stream_send(output, buf, bufp - buf);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	253 *size_r = (bufp - buf) << 2 \| packed_flags;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	254 return 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	255 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	256 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	257 i_assert(list->uid_count > 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	258 i_assert(output->offset > uid_list[0]);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	259 offset = (output->offset - uid_list[0]) << 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	260 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	261 uid_list++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	262 uid_count--;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	263 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	264
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	265 T_FRAME(
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	266 ret = uidlist_write_array(output, uid_list, uid_count,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	267 packed_flags, offset,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	268 write_size, size_r);
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	269 );
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	270 return ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	271 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	272
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	273 static void squat_uidlist_map_blocks_set_pointers(struct squat_uidlist *uidlist)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	274 {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	275 const void *base;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	276 size_t end_index_size, end_size;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	277
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	278 base = CONST_PTR_OFFSET(uidlist->data, uidlist->hdr.block_list_offset +
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	279 sizeof(uint32_t));
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	280
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	281 end_index_size = uidlist->cur_block_count * sizeof(uint32_t);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	282 end_size = end_index_size + uidlist->cur_block_count * sizeof(uint32_t);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	283 if (end_size <= uidlist->data_size) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	284 uidlist->cur_block_end_indexes = base;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	285 uidlist->cur_block_offsets =
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	286 CONST_PTR_OFFSET(base, end_index_size);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	287 } else {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	288 uidlist->cur_block_end_indexes = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	289 uidlist->cur_block_offsets = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	290 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	291 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	292
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	293 static int uidlist_file_cache_read(struct squat_uidlist *uidlist,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	294 size_t offset, size_t size)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	295 {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	296 if (uidlist->file_cache == NULL)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	297 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	298
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	299 if (file_cache_read(uidlist->file_cache, offset, size) < 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	300 i_error("read(%s) failed: %m", uidlist->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	301 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	302 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	303 uidlist->data = file_cache_get_map(uidlist->file_cache,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	304 &uidlist->data_size);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	305 squat_uidlist_map_blocks_set_pointers(uidlist);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	306 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	307 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	308
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	309 static int squat_uidlist_map_blocks(struct squat_uidlist *uidlist)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	310 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	311 const struct squat_uidlist_file_header *hdr = &uidlist->hdr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	312 const void *base;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	313 uint32_t block_count, blocks_offset, blocks_size, i, verify_count;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	314
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	315 /* get number of blocks */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	316 if (uidlist_file_cache_read(uidlist, hdr->block_list_offset,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	317 sizeof(block_count)) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	318 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	319 blocks_offset = hdr->block_list_offset + sizeof(block_count);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	320 if (blocks_offset > uidlist->data_size) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	321 squat_uidlist_set_corrupted(uidlist, "block list outside file");
5414 3ec03a4c0f26 fixes Timo Sirainen <tss@iki.fi> parents: 5010 diff changeset	322 return -1;
3ec03a4c0f26 fixes Timo Sirainen <tss@iki.fi> parents: 5010 diff changeset	323 }
3ec03a4c0f26 fixes Timo Sirainen <tss@iki.fi> parents: 5010 diff changeset	324
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	325 base = CONST_PTR_OFFSET(uidlist->data, hdr->block_list_offset);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	326 memcpy(&block_count, base, sizeof(block_count));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	327
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	328 /* map the blocks */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	329 blocks_size = block_count * sizeof(uint32_t)*2;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	330 if (uidlist_file_cache_read(uidlist, blocks_offset, blocks_size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	331 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	332 if (blocks_offset + blocks_size > uidlist->data_size) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	333 squat_uidlist_set_corrupted(uidlist, "block list outside file");
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	334 return -1;
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	335 }
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	336
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	337 uidlist->cur_block_count = block_count;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	338 squat_uidlist_map_blocks_set_pointers(uidlist);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	339
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	340 /* verify just a couple of the end indexes to make sure they
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	341 look correct */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	342 verify_count = I_MIN(block_count, 8);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	343 for (i = 1; i < verify_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	344 if (unlikely(uidlist->cur_block_end_indexes[i-1] >=
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	345 uidlist->cur_block_end_indexes[i])) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	346 squat_uidlist_set_corrupted(uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	347 "block list corrupted");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	348 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	349 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	350 }
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	351 return 0;
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	352 }
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	353
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	354 static int squat_uidlist_map_header(struct squat_uidlist *uidlist)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	355 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	356 if (uidlist->hdr.indexid == 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	357 /* still being built */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	358 return 0;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	359 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	360 if (uidlist->hdr.indexid != uidlist->trie->hdr.indexid) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	361 /* see if trie was recreated */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	362 squat_trie_refresh(uidlist->trie);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	363 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	364 if (uidlist->hdr.indexid != uidlist->trie->hdr.indexid) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	365 squat_uidlist_set_corrupted(uidlist, "wrong indexid");
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	366 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	367 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	368 if (uidlist->hdr.used_file_size < sizeof(uidlist->hdr) \|\|
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	369 (uidlist->hdr.used_file_size > uidlist->mmap_size &&
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	370 uidlist->mmap_base != NULL)) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	371 squat_uidlist_set_corrupted(uidlist, "broken used_file_size");
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	372 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	373 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	374 return squat_uidlist_map_blocks(uidlist);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	375 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	376
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	377 static void squat_uidlist_unmap(struct squat_uidlist *uidlist)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	378 {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	379 if (uidlist->mmap_size != 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	380 if (munmap(uidlist->mmap_base, uidlist->mmap_size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	381 i_error("munmap(%s) failed: %m", uidlist->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	382 uidlist->mmap_base = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	383 uidlist->mmap_size = 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	384 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	385 uidlist->cur_block_count = 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	386 uidlist->cur_block_end_indexes = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	387 uidlist->cur_block_offsets = NULL;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	388 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	389
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	390 static int squat_uidlist_mmap(struct squat_uidlist *uidlist)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	391 {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	392 struct stat st;
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	393
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	394 if (fstat(uidlist->fd, &st) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	395 i_error("fstat(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	396 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	397 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	398 if (st.st_size < (off_t)sizeof(uidlist->hdr)) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	399 if (st.st_size == 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	400 return 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	401 squat_uidlist_set_corrupted(uidlist, "File too small");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	402 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	403 }
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	404
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	405 squat_uidlist_unmap(uidlist);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	406 uidlist->mmap_size = st.st_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	407 uidlist->mmap_base = mmap(NULL, uidlist->mmap_size,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	408 PROT_READ \| PROT_WRITE,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	409 MAP_SHARED, uidlist->fd, 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	410 if (uidlist->mmap_base == MAP_FAILED) {
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	411 uidlist->data = uidlist->mmap_base = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	412 uidlist->data_size = uidlist->mmap_size = 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	413 i_error("mmap(%s) failed: %m", uidlist->path);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	414 return -1;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	415 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	416 uidlist->data = uidlist->mmap_base;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	417 uidlist->data_size = uidlist->mmap_size;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	418 return 0;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	419 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	420
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	421 static int squat_uidlist_map(struct squat_uidlist *uidlist)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	422 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	423 const struct squat_uidlist_file_header *mmap_hdr = uidlist->mmap_base;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	424 int ret;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	425
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	426 if (mmap_hdr != NULL && !uidlist->building &&
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	427 uidlist->hdr.block_list_offset == mmap_hdr->block_list_offset) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	428 /* file hasn't changed */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	429 return 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	430 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	431
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	432 if (!uidlist->trie->mmap_disable) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	433 if (mmap_hdr == NULL \|\| uidlist->building \|\|
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	434 uidlist->mmap_size < mmap_hdr->used_file_size) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	435 if (squat_uidlist_mmap(uidlist) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	436 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	437 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	438
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	439 if (!uidlist->building) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	440 memcpy(&uidlist->hdr, uidlist->mmap_base,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	441 sizeof(uidlist->hdr));
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	442 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	443 } else if (uidlist->building) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	444 /* we want to update blocks mapping, but using the header
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	445 in memory */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	446 } else {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	447 ret = pread_full(uidlist->fd, &uidlist->hdr,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	448 sizeof(uidlist->hdr), 0);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	449 if (ret <= 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	450 if (ret < 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	451 i_error("pread(%s) failed: %m", uidlist->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	452 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	453 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	454 i_error("Corrupted %s: File too small", uidlist->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	455 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	456 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	457 uidlist->data = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	458 uidlist->data_size = 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	459 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	460 if (uidlist->file_cache == NULL && uidlist->trie->mmap_disable)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	461 uidlist->file_cache = file_cache_new(uidlist->fd);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	462 return squat_uidlist_map_header(uidlist);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	463 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	464
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	465 struct squat_uidlist squat_uidlist_init(struct squat_trie trie)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	466 {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	467 struct squat_uidlist *uidlist;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	468
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	469 uidlist = i_new(struct squat_uidlist, 1);
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	470 uidlist->trie = trie;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	471 uidlist->path = i_strconcat(trie->path, ".uids", NULL);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	472 uidlist->fd = -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	473
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	474 return uidlist;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	475 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	476
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	477 void squat_uidlist_deinit(struct squat_uidlist *uidlist)
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	478 {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	479 squat_uidlist_close(uidlist);
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	480
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	481 i_free(uidlist->path);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	482 i_free(uidlist);
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	483 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	484
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	485 static int squat_uidlist_open(struct squat_uidlist *uidlist)
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	486 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	487 squat_uidlist_close(uidlist);
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	488
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	489 uidlist->fd = open(uidlist->path, O_RDWR);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	490 if (uidlist->fd == -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	491 if (errno == ENOENT) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	492 memset(&uidlist->hdr, 0, sizeof(uidlist->hdr));
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	493 return 0;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	494 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	495 i_error("open(%s) failed: %m", uidlist->path);
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	496 return -1;
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	497 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	498 return squat_uidlist_map(uidlist);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	499 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	500
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	501 static void squat_uidlist_close(struct squat_uidlist *uidlist)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	502 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	503 i_assert(!uidlist->building);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	504
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	505 squat_uidlist_unmap(uidlist);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	506 if (uidlist->file_cache != NULL)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	507 file_cache_free(&uidlist->file_cache);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	508 if (uidlist->file_lock != NULL)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	509 file_lock_free(&uidlist->file_lock);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	510 if (uidlist->fd != -1) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	511 if (close(uidlist->fd) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	512 i_error("close(%s) failed: %m", uidlist->path);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	513 uidlist->fd = -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	514 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	515 uidlist->corrupted = FALSE;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	516 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	517
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	518 int squat_uidlist_refresh(struct squat_uidlist *uidlist)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	519 {
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	520 /* we assume here that trie is locked, so that we don't need to worry
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	521 about it when reading the header */
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	522 if (uidlist->fd == -1 \|\|
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	523 uidlist->hdr.indexid != uidlist->trie->hdr.indexid) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	524 if (squat_uidlist_open(uidlist) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	525 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	526 } else {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	527 if (squat_uidlist_map(uidlist) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	528 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	529 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	530 return 0;
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	531 }
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	532
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	533 static int squat_uidlist_is_file_stale(struct squat_uidlist *uidlist)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	534 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	535 struct stat st, st2;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	536
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	537 i_assert(uidlist->fd != -1);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	538
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	539 if (stat(uidlist->path, &st) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	540 if (errno == ENOENT)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	541 return 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	542
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	543 i_error("stat(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	544 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	545 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	546 if (fstat(uidlist->fd, &st2) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	547 i_error("fstat(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	548 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	549 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	550 uidlist->locked_file_size = st2.st_size;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	551
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	552 return st.st_ino == st2.st_ino &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	553 CMP_DEV_T(st.st_dev, st2.st_dev) ? 0 : 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	554 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	555
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	556 static int squat_uidlist_lock(struct squat_uidlist *uidlist)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	557 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	558 int ret;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	559
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	560 for (;;) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	561 i_assert(uidlist->fd != -1);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	562 i_assert(uidlist->file_lock == NULL);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	563
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	564 ret = file_wait_lock(uidlist->fd, uidlist->path, F_WRLCK,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	565 uidlist->trie->lock_method,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	566 SQUAT_TRIE_LOCK_TIMEOUT,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	567 &uidlist->file_lock);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	568 if (ret == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	569 i_error("file_wait_lock(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	570 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	571 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	572 if (ret < 0)
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	573 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	574
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	575 ret = squat_uidlist_is_file_stale(uidlist);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	576 if (ret == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	577 break;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	578
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	579 file_unlock(&uidlist->file_lock);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	580 if (ret < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	581 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	582
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	583 squat_uidlist_close(uidlist);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	584 uidlist->fd = open(uidlist->path, O_RDWR \| O_CREAT, 0600);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	585 if (uidlist->fd == -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	586 i_error("open(%s) failed: %m", uidlist->path);
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	587 return -1;
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	588 }
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	589 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	590 return 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	591 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	592
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	593 static int squat_uidlist_open_or_create(struct squat_uidlist *uidlist)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	594 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	595 if (uidlist->fd == -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	596 uidlist->fd = open(uidlist->path, O_RDWR \| O_CREAT, 0600);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	597 if (uidlist->fd == -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	598 i_error("creat(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	599 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	600 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	601 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	602 if (squat_uidlist_lock(uidlist) <= 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	603 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	604
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	605 if (uidlist->locked_file_size != 0) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	606 if (squat_uidlist_map(uidlist) < 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	607 /* broken file, truncate */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	608 if (ftruncate(uidlist->fd, 0) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	609 i_error("ftruncate(%s) failed: %m",
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	610 uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	611 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	612 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	613 uidlist->locked_file_size = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	614 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	615 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	616 if (uidlist->locked_file_size == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	617 /* write using 0 until we're finished */
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	618 memset(&uidlist->hdr, 0, sizeof(uidlist->hdr));
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	619 if (write_full(uidlist->fd, &uidlist->hdr,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	620 sizeof(uidlist->hdr)) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	621 i_error("write(%s) failed: %m", uidlist->path);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	622 return -1;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	623 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	624 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	625 return 0;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	626 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	627
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	628 int squat_uidlist_build_init(struct squat_uidlist *uidlist,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	629 struct squat_uidlist_build_context **ctx_r)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	630 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	631 struct squat_uidlist_build_context *ctx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	632
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	633 i_assert(!uidlist->building);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	634
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	635 if (squat_uidlist_open_or_create(uidlist) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	636 if (uidlist->file_lock != NULL)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	637 file_unlock(&uidlist->file_lock);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	638 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	639 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	640 if (lseek(uidlist->fd, uidlist->hdr.used_file_size, SEEK_SET) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	641 i_error("lseek(%s) failed: %m", uidlist->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	642 if (uidlist->file_lock != NULL)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	643 file_unlock(&uidlist->file_lock);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	644 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	645 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	646
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	647 ctx = i_new(struct squat_uidlist_build_context, 1);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	648 ctx->uidlist = uidlist;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	649 ctx->output = o_stream_create_fd(uidlist->fd, 0, FALSE);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	650 if (ctx->output->offset == 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	651 struct squat_uidlist_file_header hdr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	652
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	653 memset(&hdr, 0, sizeof(hdr));
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	654 o_stream_send(ctx->output, &hdr, sizeof(hdr));
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	655 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	656 o_stream_cork(ctx->output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	657 i_array_init(&ctx->lists, 10240);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	658 i_array_init(&ctx->block_offsets, 128);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	659 i_array_init(&ctx->block_end_indexes, 128);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	660 ctx->list_start_idx = uidlist->hdr.count;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	661 ctx->build_hdr = uidlist->hdr;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	662
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	663 uidlist->building = TRUE;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	664 *ctx_r = ctx;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	665 return 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	666 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	667
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	668 static void
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	669 uidlist_write_block_list_and_header(struct squat_uidlist_build_context *ctx,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	670 struct ostream *output,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	671 ARRAY_TYPE(uint32_t) *block_offsets,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	672 ARRAY_TYPE(uint32_t) *block_end_indexes,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	673 bool write_old_blocks)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	674 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	675 struct squat_uidlist *uidlist = ctx->uidlist;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	676 unsigned int align, old_block_count, new_block_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	677 uint32_t block_offset_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	678 uoff_t block_list_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	679
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	680 align = output->offset % sizeof(uint32_t);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	681 if (align != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	682 static char null[sizeof(uint32_t)-1] = { 0, };
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	683
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	684 o_stream_send(output, null, sizeof(uint32_t) - align);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	685 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	686 block_list_offset = output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	687
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	688 new_block_count = array_count(block_offsets);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	689 old_block_count = write_old_blocks ? uidlist->cur_block_count : 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	690
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	691 block_offset_count = new_block_count + old_block_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	692 o_stream_send(output, &block_offset_count, sizeof(block_offset_count));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	693 /* write end indexes */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	694 o_stream_send(output, uidlist->cur_block_end_indexes,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	695 old_block_count * sizeof(uint32_t));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	696 o_stream_send(output, array_idx(block_end_indexes, 0),
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	697 new_block_count * sizeof(uint32_t));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	698 /* write offsets */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	699 o_stream_send(output, uidlist->cur_block_offsets,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	700 old_block_count * sizeof(uint32_t));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	701 o_stream_send(output, array_idx(block_offsets, 0),
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	702 new_block_count * sizeof(uint32_t));
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	703 o_stream_flush(output);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	704
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	705 /* update header - it's written later when trie is locked */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	706 i_assert(uidlist->trie->hdr.indexid != 0);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	707 ctx->build_hdr.indexid = uidlist->trie->hdr.indexid;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	708 ctx->build_hdr.block_list_offset = block_list_offset;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	709 ctx->build_hdr.used_file_size = output->offset;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	710 uidlist->hdr = ctx->build_hdr;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	711 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	712
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	713 void squat_uidlist_build_flush(struct squat_uidlist_build_context *ctx)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	714 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	715 struct uidlist_list *lists;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	716 uint8_t buf[SQUAT_PACK_MAX_SIZE], *bufp;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	717 unsigned int i, j, count, max;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	718 uint32_t block_offset, block_end_idx, start_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	719 uint32_t list_sizes[UIDLIST_BLOCK_LIST_COUNT];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	720 size_t mem_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	721
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	722 if (ctx->uidlist->corrupted)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	723 return;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	724
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	725 lists = array_get_modifiable(&ctx->lists, &count);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	726 if (count == 0)
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	727 return;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	728
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	729 /* write the lists and save the written sizes to uid_list[0] */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	730 for (i = 0; i < count; i += UIDLIST_BLOCK_LIST_COUNT) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	731 start_offset = ctx->output->offset;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	732 max = I_MIN(count - i, UIDLIST_BLOCK_LIST_COUNT);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	733 for (j = 0; j < max; j++) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	734 if (uidlist_write(ctx->output, &lists[i+j],
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	735 FALSE, &list_sizes[j]) < 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	736 squat_uidlist_set_corrupted(ctx->uidlist,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	737 "Broken uidlists");
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	738 return;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	739 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	740 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	741
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	742 block_offset = ctx->output->offset;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	743 block_end_idx = ctx->list_start_idx + i + max;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	744 array_append(&ctx->block_offsets, &block_offset, 1);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	745 array_append(&ctx->block_end_indexes, &block_end_idx, 1);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	746
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	747 /* write the full size of the uidlists */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	748 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	749 squat_pack_num(&bufp, block_offset - start_offset);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	750 o_stream_send(ctx->output, buf, bufp - buf);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	751
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	752 /* write the sizes/flags */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	753 for (j = 0; j < max; j++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	754 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	755 squat_pack_num(&bufp, list_sizes[j]);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	756 o_stream_send(ctx->output, buf, bufp - buf);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	757 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	758 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	759
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	760 mem_size = ctx->lists.arr.buffer->used +
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	761 ctx->block_offsets.arr.buffer->used +
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	762 ctx->block_end_indexes.arr.buffer->used;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	763 if (ctx->uidlist->max_size < mem_size)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	764 ctx->uidlist->max_size = mem_size;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	765
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	766 ctx->list_start_idx += count;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	767 array_clear(&ctx->lists);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	768
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	769 uidlist_write_block_list_and_header(ctx, ctx->output,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	770 &ctx->block_offsets,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	771 &ctx->block_end_indexes, TRUE);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	772
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	773 (void)squat_uidlist_map(ctx->uidlist);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	774
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	775 array_clear(&ctx->block_offsets);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	776 array_clear(&ctx->block_end_indexes);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	777 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	778
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	779 int squat_uidlist_build_finish(struct squat_uidlist_build_context *ctx)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	780 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	781 if (ctx->uidlist->corrupted)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	782 return -1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	783
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	784 o_stream_seek(ctx->output, 0);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	785 o_stream_send(ctx->output, &ctx->build_hdr, sizeof(ctx->build_hdr));
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	786 o_stream_seek(ctx->output, ctx->build_hdr.used_file_size);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	787 o_stream_flush(ctx->output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	788
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	789 if (ctx->output->last_failed_errno != 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	790 errno = ctx->output->last_failed_errno;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	791 i_error("write() to %s failed: %m", ctx->uidlist->path);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	792 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	793 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	794 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	795 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	796
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	797 void squat_uidlist_build_deinit(struct squat_uidlist_build_context **_ctx)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	798 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	799 struct squat_uidlist_build_context ctx = _ctx;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	800
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	801 *_ctx = NULL;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	802
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	803 i_assert(array_count(&ctx->lists) == 0 \|\| ctx->uidlist->corrupted);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	804 i_assert(ctx->uidlist->building);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	805 ctx->uidlist->building = FALSE;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	806
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	807 file_unlock(&ctx->uidlist->file_lock);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	808
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	809 if (ctx->need_reopen)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	810 (void)squat_uidlist_open(ctx->uidlist);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	811
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	812 array_free(&ctx->block_offsets);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	813 array_free(&ctx->block_end_indexes);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	814 array_free(&ctx->lists);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	815 o_stream_unref(&ctx->output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	816 i_free(ctx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	817 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	818
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	819 int squat_uidlist_rebuild_init(struct squat_uidlist_build_context *build_ctx,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	820 bool compress,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	821 struct squat_uidlist_rebuild_context **ctx_r)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	822 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	823 struct squat_uidlist_rebuild_context *ctx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	824 struct squat_uidlist_file_header hdr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	825 const char *temp_path;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	826 int fd;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	827
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	828 if (build_ctx->build_hdr.link_count == 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	829 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	830
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	831 if (!compress) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	832 if (build_ctx->build_hdr.link_count <
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	833 build_ctx->build_hdr.count*2/3)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	834 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	835 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	836
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	837 temp_path = t_strconcat(build_ctx->uidlist->path, ".tmp", NULL);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	838 fd = open(temp_path, O_RDWR \| O_TRUNC \| O_CREAT, 0600);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	839 if (fd < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	840 i_error("open(%s) failed: %m", temp_path);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	841 return -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	842 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	843
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	844 ctx = i_new(struct squat_uidlist_rebuild_context, 1);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	845 ctx->uidlist = build_ctx->uidlist;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	846 ctx->build_ctx = build_ctx;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	847 ctx->fd = fd;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	848 ctx->output = o_stream_create_fd(ctx->fd, 0, FALSE);
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	849 ctx->next_uid_list_idx = 0x100;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	850 o_stream_cork(ctx->output);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	851
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	852 memset(&hdr, 0, sizeof(hdr));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	853 o_stream_send(ctx->output, &hdr, sizeof(hdr));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	854
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	855 ctx->cur_block_start_offset = ctx->output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	856 i_array_init(&ctx->new_block_offsets,
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	857 build_ctx->build_hdr.count / UIDLIST_BLOCK_LIST_COUNT);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	858 i_array_init(&ctx->new_block_end_indexes,
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	859 build_ctx->build_hdr.count / UIDLIST_BLOCK_LIST_COUNT);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	860 *ctx_r = ctx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	861 return 1;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	862 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	863
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	864 static void
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	865 uidlist_rebuild_flush_block(struct squat_uidlist_rebuild_context *ctx)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	866 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	867 uint8_t buf[SQUAT_PACK_MAX_SIZE], *bufp;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	868 uint32_t block_offset, block_end_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	869 unsigned int i;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	870
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	871 ctx->new_count += ctx->list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	872
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	873 block_offset = ctx->output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	874 block_end_idx = ctx->new_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	875 array_append(&ctx->new_block_offsets, &block_offset, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	876 array_append(&ctx->new_block_end_indexes, &block_end_idx, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	877
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	878 /* this block's contents started from cur_block_start_offset and
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	879 ended to current offset. write the size of this area. */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	880 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	881 squat_pack_num(&bufp, block_offset - ctx->cur_block_start_offset);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	882 o_stream_send(ctx->output, buf, bufp - buf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	883
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	884 /* write the sizes/flags */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	885 for (i = 0; i < ctx->list_idx; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	886 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	887 squat_pack_num(&bufp, ctx->list_sizes[i]);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	888 o_stream_send(ctx->output, buf, bufp - buf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	889 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	890 ctx->cur_block_start_offset = ctx->output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	891 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	892
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	893 uint32_t squat_uidlist_rebuild_next(struct squat_uidlist_rebuild_context *ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	894 const ARRAY_TYPE(uint32_t) *uids)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	895 {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	896 int ret;
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	897
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	898 T_FRAME(
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	899 ret = uidlist_write_array(ctx->output, array_idx(uids, 0),
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	900 array_count(uids), 0, 0, FALSE,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	901 &ctx->list_sizes[ctx->list_idx]);
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	902 );
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	903 if (ret < 0)
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	904 squat_uidlist_set_corrupted(ctx->uidlist, "Broken uidlists");
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	905
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	906 if (++ctx->list_idx == UIDLIST_BLOCK_LIST_COUNT) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	907 uidlist_rebuild_flush_block(ctx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	908 ctx->list_idx = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	909 }
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	910 return ctx->next_uid_list_idx++ << 1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	911 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	912
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	913 uint32_t squat_uidlist_rebuild_nextu(struct squat_uidlist_rebuild_context *ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	914 const ARRAY_TYPE(seq_range) *uids)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	915 {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	916 const struct seq_range *range;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	917 ARRAY_TYPE(uint32_t) tmp_uids;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	918 uint32_t seq, uid1, ret;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	919 unsigned int i, count;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	920
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	921 range = array_get(uids, &count);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	922 if (count == 0)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	923 return 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	924
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	925 if (count == 1 && range[0].seq1 == range[0].seq2) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	926 /* single UID */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	927 return (range[0].seq1 << 1) \| 1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	928 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	929 if (range[count-1].seq2 < 8) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	930 /* we can use a singleton bitmask */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	931 ret = 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	932 for (i = 0; i < count; i++) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	933 for (seq = range[i].seq1; seq <= range[i].seq2; seq++)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	934 ret \|= 1 << (seq+1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	935 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	936 return ret;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	937 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	938
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	939 /* convert seq range to our internal representation and use the
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	940 normal _rebuild_next() to write it */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	941 i_array_init(&tmp_uids, 128);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	942 for (i = 0; i < count; i++) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	943 if (range[i].seq1 == range[i].seq2)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	944 array_append(&tmp_uids, &range[i].seq1, 1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	945 else {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	946 uid1 = range[i].seq1 \| UID_LIST_MASK_RANGE;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	947 array_append(&tmp_uids, &uid1, 1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	948 array_append(&tmp_uids, &range[i].seq2, 1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	949 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	950 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	951 ret = squat_uidlist_rebuild_next(ctx, &tmp_uids);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	952 array_free(&tmp_uids);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	953 return ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	954 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	955
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	956 int squat_uidlist_rebuild_finish(struct squat_uidlist_rebuild_context *ctx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	957 bool cancel)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	958 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	959 const char *temp_path;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	960 int ret = 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	961
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	962 if (ctx->list_idx != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	963 uidlist_rebuild_flush_block(ctx);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	964 if (array_count(&ctx->new_block_end_indexes) == 0 \|\|
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	965 cancel \|\| ctx->uidlist->corrupted)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	966 ret = 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	967
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	968 temp_path = t_strconcat(ctx->uidlist->path, ".tmp", NULL);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	969 if (ret > 0) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	970 ctx->build_ctx->build_hdr.indexid =
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	971 ctx->uidlist->trie->hdr.indexid;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	972 ctx->build_ctx->build_hdr.count = ctx->new_count;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	973 ctx->build_ctx->build_hdr.link_count = 0;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	974 uidlist_write_block_list_and_header(ctx->build_ctx, ctx->output,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	975 &ctx->new_block_offsets,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	976 &ctx->new_block_end_indexes,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	977 FALSE);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	978 o_stream_seek(ctx->output, 0);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	979 o_stream_send(ctx->output, &ctx->build_ctx->build_hdr,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	980 sizeof(ctx->build_ctx->build_hdr));
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	981 o_stream_seek(ctx->output,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	982 ctx->build_ctx->build_hdr.used_file_size);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	983 o_stream_flush(ctx->output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	984
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	985 if (ctx->uidlist->corrupted)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	986 ret = -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	987 else if (ctx->output->last_failed_errno != 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	988 errno = ctx->output->last_failed_errno;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	989 i_error("write() to %s failed: %m", temp_path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	990 ret = -1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	991 } else if (rename(temp_path, ctx->uidlist->path) < 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	992 i_error("rename(%s, %s) failed: %m",
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	993 temp_path, ctx->uidlist->path);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	994 ret = -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	995 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	996 ctx->build_ctx->need_reopen = TRUE;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	997 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	998
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	999 o_stream_unref(&ctx->output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1000 if (close(ctx->fd) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1001 i_error("close(%s) failed: %m", temp_path);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1002
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1003 if (ret <= 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1004 if (unlink(temp_path) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1005 i_error("unlink(%s) failed: %m", temp_path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1006 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1007 array_free(&ctx->new_block_offsets);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1008 array_free(&ctx->new_block_end_indexes);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1009 i_free(ctx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1010 return ret < 0 ? -1 : 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1011 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1012
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1013 static void
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1014 uidlist_flush(struct squat_uidlist_build_context *ctx,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1015 struct uidlist_list *list, uint32_t uid)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1016 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1017 uint32_t size, offset = ctx->output->offset;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1018
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1019 ctx->build_hdr.link_count++;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1020 if (uidlist_write(ctx->output, list, TRUE, &size) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1021 squat_uidlist_set_corrupted(ctx->uidlist, "Broken uidlists");
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1022
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1023 list->uid_count = 2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1024 list->uid_begins_with_pointer = TRUE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1025
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1026 list->uid_list[0] = offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1027 list->uid_list[1] = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1028 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1029
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1030 static struct uidlist_list *
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1031 uidlist_add_new(struct squat_uidlist_build_context *ctx, unsigned int count,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1032 uint32_t *uid_list_idx_r)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1033 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1034 struct uidlist_list *list;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1035
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1036 i_assert(array_count(&ctx->lists) +
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1037 ctx->list_start_idx == ctx->build_hdr.count);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1038 *uid_list_idx_r = (ctx->build_hdr.count + 0x100) << 1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1039 list = array_append_space(&ctx->lists);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1040 ctx->build_hdr.count++;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1041
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1042 list->uid_count = count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1043 return list;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1044 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1045
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1046 uint32_t squat_uidlist_build_add_uid(struct squat_uidlist_build_context *ctx,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1047 uint32_t uid_list_idx, uint32_t uid)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1048 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1049 struct uidlist_list *list;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1050 unsigned int idx, mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1051 uint32_t *p;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1052
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1053 if ((uid_list_idx & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1054 /* adding second UID */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1055 uint32_t prev_uid = uid_list_idx >> 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1056
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1057 i_assert(prev_uid != uid);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1058 list = uidlist_add_new(ctx, 2, &uid_list_idx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1059 list->uid_list[0] = prev_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1060 if (prev_uid + 1 == uid)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1061 list->uid_list[0] \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1062 list->uid_list[1] = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1063 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1064 } else if (uid_list_idx < (0x100 << 1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1065 uint32_t old_list_idx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1066
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1067 if (uid < 8) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1068 /* UID lists containing only UIDs 0-7 are saved as
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1069 uidlist values 2..511. think of it as a bitmask. */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1070 uid_list_idx \|= 1 << (uid + 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1071 i_assert((uid_list_idx & 1) == 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1072 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1073 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1074
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1075 if (uid_list_idx == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1076 /* first UID */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1077 return (uid << 1) \| 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1078 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1079
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1080 /* create a new list */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1081 old_list_idx = uid_list_idx >> 1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1082 list = uidlist_add_new(ctx, 1, &uid_list_idx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1083 /* add the first UID ourself */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1084 idx = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1085 i_assert((old_list_idx & 0xff) != 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1086 for (mask = 1; mask <= 128; mask <<= 1, idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1087 if ((old_list_idx & mask) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1088 list->uid_list[0] = idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1089 idx++; mask <<= 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1090 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1091 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1092 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1093 for (; mask <= 128; mask <<= 1, idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1094 if ((old_list_idx & mask) != 0) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1095 squat_uidlist_build_add_uid(ctx,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1096 uid_list_idx, idx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1097 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1098 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1099 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1100
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1101 /* add to existing list */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1102 idx = (uid_list_idx >> 1) - 0x100;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1103 if (idx < ctx->list_start_idx) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1104 list = uidlist_add_new(ctx, 2, &uid_list_idx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1105 list->uid_list[0] = UID_LIST_POINTER_MASK_LIST_IDX \| idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1106 list->uid_list[1] = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1107 list->uid_begins_with_pointer = TRUE;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1108 ctx->build_hdr.link_count++;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1109 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1110 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1111
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1112 idx -= ctx->list_start_idx;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1113 if (idx >= array_count(&ctx->lists)) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1114 squat_uidlist_set_corrupted(ctx->uidlist,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1115 "missing/broken uidlist");
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1116 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1117 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1118 list = array_idx_modifiable(&ctx->lists, idx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1119 i_assert(list->uid_count > 0);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1120
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1121 p = &list->uid_list[list->uid_count-1];
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1122 i_assert(uid != *p \|\| ctx->uidlist->corrupted \|\|
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1123 (list->uid_count == 1 && list->uid_begins_with_pointer));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1124 if (uid == *p + 1 &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1125 (list->uid_count > 1 \|\| !list->uid_begins_with_pointer)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1126 /* use a range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1127 if (list->uid_count > 1 && (p[-1] & UID_LIST_MASK_RANGE) != 0 &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1128 (list->uid_count > 2 \|\| !list->uid_begins_with_pointer)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1129 /* increase the existing range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1130 *p += 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1131 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1132 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1133
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1134 if (list->uid_count == UIDLIST_LIST_SIZE) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1135 uidlist_flush(ctx, list, uid);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1136 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1137 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1138 /* create a new range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1139 *p \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1140 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1141
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1142 if (list->uid_count == UIDLIST_LIST_SIZE) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1143 uidlist_flush(ctx, list, uid);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1144 return uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1145 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1146
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1147 p++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1148 list->uid_count++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1149
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1150 *p = uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1151 return uid_list_idx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1152 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1153
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1154 static void uidlist_array_append(ARRAY_TYPE(uint32_t) *uids, uint32_t uid)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1155 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1156 uint32_t *uidlist;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1157 unsigned int count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1158
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1159 uidlist = array_get_modifiable(uids, &count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1160 if (count == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1161 array_append(uids, &uid, 1);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1162 return;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1163 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1164 if (uidlist[count-1] + 1 == uid) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1165 if (count > 1 && (uidlist[count-2] &
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1166 UID_LIST_MASK_RANGE) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1167 uidlist[count-1]++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1168 return;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1169 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1170 uidlist[count-1] \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1171 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1172 array_append(uids, &uid, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1173 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1174
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1175 static void uidlist_array_append_range(ARRAY_TYPE(uint32_t) *uids,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1176 uint32_t uid1, uint32_t uid2)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1177 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1178 uint32_t *uidlist;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1179 unsigned int count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1180
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1181 i_assert(uid1 < uid2);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1182
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1183 uidlist = array_get_modifiable(uids, &count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1184 if (count == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1185 uid1 \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1186 array_append(uids, &uid1, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1187 array_append(uids, &uid2, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1188 return;
4933 44a0e7606f7a Use seq_range_array_remove_range() instead of slowly iterating ourself. Timo Sirainen <tss@iki.fi> parents: 4931 diff changeset	1189 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1190 if (uidlist[count-1] + 1 == uid1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1191 if (count > 1 && (uidlist[count-2] &
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1192 UID_LIST_MASK_RANGE) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1193 uidlist[count-1] = uid2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1194 return;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1195 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1196 uidlist[count-1] \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1197 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1198 uid1 \|= UID_LIST_MASK_RANGE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1199 array_append(uids, &uid1, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1200 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1201 array_append(uids, &uid2, 1);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1202 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1203
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1204 static int
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1205 squat_uidlist_get_at_offset(struct squat_uidlist *uidlist, uoff_t offset,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1206 uint32_t num, ARRAY_TYPE(uint32_t) *uids)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1207 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1208 const uint8_t p, end;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1209 uint32_t size, base_uid;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1210 uoff_t uidlist_data_offset;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1211 unsigned int i, j, extra = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1212
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1213 if (num != 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1214 uidlist_data_offset = offset;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1215 else {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1216 /* not given, read it */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1217 if (uidlist_file_cache_read(uidlist, offset,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1218 SQUAT_PACK_MAX_SIZE) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1219 return -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1220
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1221 p = CONST_PTR_OFFSET(uidlist->data, offset);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1222 end = CONST_PTR_OFFSET(uidlist->data, uidlist->data_size);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1223 num = squat_unpack_num(&p, end);
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1224 uidlist_data_offset = p - (const uint8_t *)uidlist->data;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1225 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1226 size = num >> 2;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1227
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1228 if (uidlist_file_cache_read(uidlist, uidlist_data_offset, size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1229 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1230 if (uidlist_data_offset + size > uidlist->data_size) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1231 squat_uidlist_set_corrupted(uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1232 "size points outside file");
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1233 return -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1234 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1235
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1236 p = CONST_PTR_OFFSET(uidlist->data, uidlist_data_offset);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1237 end = p + size;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1238
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1239 if ((num & UIDLIST_PACKED_FLAG_BEGINS_WITH_POINTER) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1240 /* link to the file */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1241 uint32_t prev = squat_unpack_num(&p, end);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1242
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1243 if ((prev & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1244 /* pointer to uidlist */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1245 prev = ((prev >> 1) + 0x100) << 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1246 if (squat_uidlist_get(uidlist, prev, uids) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1247 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1248 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1249 prev = offset - (prev >> 1);
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1250 if (squat_uidlist_get_at_offset(uidlist, prev,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1251 0, uids) < 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1252 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1253 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1254 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1255
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1256 if ((num & UIDLIST_PACKED_FLAG_BITMASK) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1257 /* bitmask */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1258 base_uid = squat_unpack_num(&p, end);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1259 size = end - p;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1260
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1261 uidlist_array_append(uids, base_uid++);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1262 for (i = 0; i < size; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1263 for (j = 0; j < 8; j++, base_uid++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1264 if ((p[i] & (1 << j)) != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1265 uidlist_array_append(uids, base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1266 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1267 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1268 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1269 /* range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1270 base_uid = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1271 while (p < end) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1272 num = squat_unpack_num(&p, end);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1273 base_uid += (num >> 1) + extra;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1274 if ((num & 1) == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1275 uidlist_array_append(uids, base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1276 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1277 /* range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1278 uint32_t seq1 = base_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1279 base_uid += squat_unpack_num(&p, end) + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1280 uidlist_array_append_range(uids, seq1,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1281 base_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1282 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1283 extra = 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1284 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1285 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1286 return 0;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1287 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1288
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1289 static int uint32_cmp(const void key, const void data)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1290 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1291 const uint32_t i1 = key, i2 = data;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1292
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1293 return (int)i1 - (int)i2;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1294 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1295
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1296 static int
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1297 squat_uidlist_get_offset(struct squat_uidlist *uidlist, uint32_t uid_list_idx,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1298 uint32_t offset_r, uint32_t num_r)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1299 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1300 const uint8_t p, end;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1301 unsigned int idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1302 uint32_t num, skip_bytes, uidlists_offset;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1303 size_t max_map_size;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1304
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1305 if (uidlist->fd == -1) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1306 squat_uidlist_set_corrupted(uidlist, "no uidlists");
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1307 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1308 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6917 diff changeset	1309
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1310 if (bsearch_insert_pos(&uid_list_idx, uidlist->cur_block_end_indexes,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1311 uidlist->cur_block_count,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1312 sizeof(uint32_t), uint32_cmp, &idx))
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1313 idx++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1314 if (unlikely(idx == uidlist->cur_block_count)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1315 squat_uidlist_set_corrupted(uidlist, "uidlist not found");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1316 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1317 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1318 if (unlikely(idx > 0 &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1319 uidlist->cur_block_end_indexes[idx-1] > uid_list_idx)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1320 squat_uidlist_set_corrupted(uidlist, "broken block list");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1321 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1322 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1323
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1324 /* make sure everything is mapped */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1325 uid_list_idx -= idx == 0 ? 0 : uidlist->cur_block_end_indexes[idx-1];
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1326 max_map_size = SQUAT_PACK_MAX_SIZE * (1+uid_list_idx);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1327 if (uidlist_file_cache_read(uidlist, uidlist->cur_block_offsets[idx],
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1328 max_map_size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1329 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1330
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1331 /* find the uidlist inside the block */
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1332 p = CONST_PTR_OFFSET(uidlist->data, uidlist->cur_block_offsets[idx]);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1333 end = CONST_PTR_OFFSET(uidlist->data, uidlist->data_size);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1334
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1335 uidlists_offset = uidlist->cur_block_offsets[idx] -
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1336 squat_unpack_num(&p, end);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1337 for (skip_bytes = 0; uid_list_idx > 0; uid_list_idx--) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1338 num = squat_unpack_num(&p, end);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1339 skip_bytes += num >> 2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1340 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1341 *offset_r = uidlists_offset + skip_bytes;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1342 *num_r = squat_unpack_num(&p, end);
4931 59ce1d6c0b8c Actually the code wasn't unneeded, put it back and actually execute it now. Timo Sirainen <tss@iki.fi> parents: 4930 diff changeset	1343
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1344 if (unlikely(p == end)) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1345 squat_uidlist_set_corrupted(uidlist, "broken file");
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1346 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1347 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1348 if (unlikely(*offset_r > uidlist->mmap_size &&
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1349 uidlist->mmap_base != NULL)) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1350 squat_uidlist_set_corrupted(uidlist, "broken offset");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1351 return -1;
4931 59ce1d6c0b8c Actually the code wasn't unneeded, put it back and actually execute it now. Timo Sirainen <tss@iki.fi> parents: 4930 diff changeset	1352 }
59ce1d6c0b8c Actually the code wasn't unneeded, put it back and actually execute it now. Timo Sirainen <tss@iki.fi> parents: 4930 diff changeset	1353 return 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1354 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1355
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1356 int squat_uidlist_get(struct squat_uidlist *uidlist, uint32_t uid_list_idx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1357 ARRAY_TYPE(uint32_t) *uids)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1358 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1359 unsigned int mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1360 uint32_t uid, offset, num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1361
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1362 if ((uid_list_idx & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1363 /* single UID */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1364 uid = uid_list_idx >> 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1365 uidlist_array_append(uids, uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1366 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1367 } else if (uid_list_idx < (0x100 << 1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1368 /* bitmask */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1369 for (uid = 0, mask = 2; mask <= 256; mask <<= 1, uid++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1370 if ((uid_list_idx & mask) != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1371 uidlist_array_append(uids, uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1372 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1373 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1374 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1375
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1376 uid_list_idx = (uid_list_idx >> 1) - 0x100;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1377 if (squat_uidlist_get_offset(uidlist, uid_list_idx, &offset, &num) < 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1378 return -1;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1379 return squat_uidlist_get_at_offset(uidlist, offset, num, uids);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1380 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1381
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1382 uint32_t squat_uidlist_singleton_last_uid(uint32_t uid_list_idx)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1383 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1384 unsigned int idx, mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1385
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1386 if ((uid_list_idx & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1387 /* single UID */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1388 return uid_list_idx >> 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1389 } else if (uid_list_idx < (0x100 << 1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1390 /* bitmask */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1391 if (uid_list_idx == 2) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1392 /* just a quick optimization */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1393 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1394 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1395 for (idx = 7, mask = 256; mask > 2; mask >>= 1, idx--) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1396 if ((uid_list_idx & mask) != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1397 return idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1398 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1399 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1400
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1401 i_unreached();
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1402 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1403 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1404
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1405 int squat_uidlist_get_seqrange(struct squat_uidlist *uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1406 uint32_t uid_list_idx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1407 ARRAY_TYPE(seq_range) *seq_range_arr)
4886 ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	1408 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1409 ARRAY_TYPE(uint32_t) tmp_uid_arr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1410 struct seq_range range;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1411 const uint32_t *tmp_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1412 unsigned int i, count;
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1413 int ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1414
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1415 i_array_init(&tmp_uid_arr, 128);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1416 ret = squat_uidlist_get(uidlist, uid_list_idx, &tmp_uid_arr);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1417 if (ret == 0) {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1418 tmp_uids = array_get(&tmp_uid_arr, &count);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1419 for (i = 0; i < count; i++) {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1420 if ((tmp_uids[i] & UID_LIST_MASK_RANGE) == 0)
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1421 range.seq1 = range.seq2 = tmp_uids[i];
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1422 else {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1423 range.seq1 = tmp_uids[i] & ~UID_LIST_MASK_RANGE;
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1424 range.seq2 = tmp_uids[++i];
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1425 }
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1426 array_append(seq_range_arr, &range, 1);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1427 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1428 }
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1429 array_free(&tmp_uid_arr);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1430 return ret;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1431 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1432
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1433 int squat_uidlist_filter(struct squat_uidlist *uidlist, uint32_t uid_list_idx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1434 ARRAY_TYPE(seq_range) *uids)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1435 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1436 const struct seq_range *parent_range;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1437 ARRAY_TYPE(seq_range) dest_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1438 ARRAY_TYPE(uint32_t) relative_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1439 const uint32_t *rel_range;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1440 unsigned int i, rel_count, parent_idx, parent_count, diff, parent_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1441 uint32_t prev_seq, seq1, seq2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1442
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1443 parent_range = array_get(uids, &parent_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1444 if (parent_count == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1445 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1446
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1447 i_array_init(&relative_uids, 128);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1448 i_array_init(&dest_uids, 128);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1449 squat_uidlist_get(uidlist, uid_list_idx, &relative_uids);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1450
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1451 parent_idx = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1452 rel_range = array_get(&relative_uids, &rel_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1453 prev_seq = 0; parent_uid = parent_range[0].seq1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1454 for (i = 0; i < rel_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1455 if (unlikely(parent_uid == (uint32_t)-1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1456 i_error("broken UID ranges");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1457 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1458 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1459 if ((rel_range[i] & UID_LIST_MASK_RANGE) == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1460 seq1 = seq2 = rel_range[i];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1461 else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1462 seq1 = (rel_range[i] & ~UID_LIST_MASK_RANGE);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1463 seq2 = rel_range[++i];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1464 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1465 i_assert(seq1 >= prev_seq);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1466 diff = seq1 - prev_seq;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1467 while (diff > 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1468 if (unlikely(parent_uid == (uint32_t)-1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1469 i_error("broken UID ranges");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1470 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1471 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1472
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1473 for (; parent_idx < parent_count; parent_idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1474 if (parent_range[parent_idx].seq2 <= parent_uid)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1475 continue;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1476 if (parent_uid < parent_range[parent_idx].seq1)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1477 parent_uid = parent_range[parent_idx].seq1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1478 else
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1479 parent_uid++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1480 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1481 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1482 diff--;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1483 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1484 diff = seq2 - seq1 + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1485 while (diff > 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1486 if (unlikely(parent_uid == (uint32_t)-1)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1487 i_error("broken UID ranges");
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1488 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1489 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1490 seq_range_array_add(&dest_uids, 0, parent_uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1491 for (; parent_idx < parent_count; parent_idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1492 if (parent_range[parent_idx].seq2 <= parent_uid)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1493 continue;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1494 if (parent_uid < parent_range[parent_idx].seq1)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1495 parent_uid = parent_range[parent_idx].seq1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1496 else
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1497 parent_uid++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1498 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1499 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1500 diff--;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1501 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1502
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1503 prev_seq = seq2 + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1504 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1505
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1506 buffer_set_used_size(uids->arr.buffer, 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1507 array_append_array(uids, &dest_uids);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1508
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1509 array_free(&relative_uids);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1510 array_free(&dest_uids);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1511 return 0;
4886 ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	1512 }
ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	1513
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1514 size_t squat_uidlist_mem_used(struct squat_uidlist *uidlist,
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1515 unsigned int *count_r)
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1516 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1517 *count_r = uidlist->hdr.count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1518 return uidlist->max_size;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1519 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/plugins/fts-squat/squat-uidlist.c @ 7210:f5f77a3ae203 HEAD