dovecot/original-hg/dovecot-1.2: src/plugins/fts-squat/squat-trie.c annotate

annotate src/plugins/fts-squat/squat-trie.c @ 7226:e6693a0ec8e1 HEAD

Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and replaced them with T_BEGIN/END calls. T_FRAME() made it difficult to debug code with gdb.

author	Timo Sirainen <tss@iki.fi>
date	Mon, 11 Feb 2008 20:17:00 +0200
parents	d5116fa2c88f
children	599a9c679ee2

rev	line source
7086 7ed926ed7aa4 Updated copyright notices to include year 2008. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	1 /* Copyright (c) 2007-2008 Dovecot authors, see the included COPYING file */
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	2
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	3 #include "lib.h"
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	4 #include "array.h"
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	5 #include "str.h"
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	6 #include "read-full.h"
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	7 #include "istream.h"
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	8 #include "ostream.h"
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	9 #include "unichar.h"
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	10 #include "file-cache.h"
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	11 #include "seq-range-array.h"
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	12 #include "squat-uidlist.h"
5068 65a4e1e61555 Some bugfixes. Moved header struct to a separate file. Timo Sirainen <tss@iki.fi> parents: 5010 diff changeset	13 #include "squat-trie-private.h"
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	14
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	15 #include <stdio.h>
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	16 #include <stdlib.h>
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	17 #include <unistd.h>
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	18 #include <sys/mman.h>
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	19
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	20 #define DEFAULT_NORMALIZE_MAP_CHARS \
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	21 "EOTIRSACDNLMVUGPHBFWYXKJQZ0123456789@.-+#$%_&"
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	22 #define DEFAULT_PARTIAL_LEN 4
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	23 #define DEFAULT_FULL_LEN 4
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	24
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	25 #define MAX_FAST_LEVEL 3
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	26 #define SEQUENTIAL_COUNT 46
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	27
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	28 #define TRIE_BYTES_LEFT(n) \
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	29 ((n) * SQUAT_PACK_MAX_SIZE)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	30 #define TRIE_READAHEAD_SIZE \
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	31 I_MAX(4096, 1 + 256 + TRIE_BYTES_LEFT(256))
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	32
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	33 struct squat_trie_build_context {
88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	34 struct squat_trie *trie;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	35 struct ostream *output;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	36 struct squat_uidlist_build_context *uidlist_build_ctx;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	37
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	38 struct file_lock *file_lock;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	39
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	40 uint32_t first_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	41 unsigned int compress_nodes:1;
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	42 };
88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	43
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	44 struct squat_trie_iterate_node {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	45 struct squat_node *node;
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	46 ARRAY_TYPE(seq_range) shifts;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	47 unsigned int idx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	48 };
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	49
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	50 struct squat_trie_iterate_context {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	51 struct squat_trie *trie;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	52 struct squat_trie_iterate_node cur;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	53 ARRAY_DEFINE(parents, struct squat_trie_iterate_node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	54 bool failed;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	55 };
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	56
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	57 static int squat_trie_map(struct squat_trie *trie, bool building);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	58
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	59 void squat_trie_delete(struct squat_trie *trie)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	60 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	61 if (unlink(trie->path) < 0 && errno != ENOENT)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	62 i_error("unlink(%s) failed: %m", trie->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	63 squat_uidlist_delete(trie->uidlist);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	64 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	65
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	66 static void squat_trie_set_corrupted(struct squat_trie *trie)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	67 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	68 trie->corrupted = TRUE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	69 i_error("Corrupted file %s", trie->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	70 squat_trie_delete(trie);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	71 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	72
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	73 static void squat_trie_normalize_map_build(struct squat_trie *trie)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	74 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	75 static unsigned char valid_chars[] =
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	76 DEFAULT_NORMALIZE_MAP_CHARS;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	77 unsigned int i, j;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	78
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	79 memset(trie->default_normalize_map, 0,
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	80 sizeof(trie->default_normalize_map));
4901 5e604d6f391e Only ASCII texts were actually indexed. Now UTF-8 input is properly Timo Sirainen <tss@iki.fi> parents: 4898 diff changeset	81
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	82 #if 1
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	83 for (i = 0, j = 1; i < sizeof(valid_chars)-1; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	84 unsigned char chr = valid_chars[i];
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	85
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	86 if (chr >= 'A' && chr <= 'Z')
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	87 trie->default_normalize_map[chr-'A'+'a'] = j;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	88 trie->default_normalize_map[chr] = j++;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	89 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	90 i_assert(j <= SEQUENTIAL_COUNT);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	91
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	92 for (i = 128; i < 256; i++)
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	93 trie->default_normalize_map[i] = j++;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	94 #else
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	95 for (i = 0; i < sizeof(valid_chars)-1; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	96 unsigned char chr = valid_chars[i];
4890 cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	97
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	98 if (chr >= 'A' && chr <= 'Z')
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	99 trie->default_normalize_map[chr-'A'+'a'] = chr;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	100 trie->default_normalize_map[chr] = chr;
4890 cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	101 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	102 for (i = 128; i < 256; i++)
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	103 trie->default_normalize_map[i] = i_toupper(i);
4890 cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	104 #endif
cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	105 }
cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	106
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	107 static void node_free(struct squat_trie trie, struct squat_node node)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	108 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	109 struct squat_node *children;
4890 cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	110 unsigned int i;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	111
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	112 if (NODE_IS_DYNAMIC_LEAF(node))
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	113 i_free(node->children.leaf_string);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	114 else if (!node->children_not_mapped && node->child_count > 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	115 children = NODE_CHILDREN_NODES(node);
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	116
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	117 trie->node_alloc_size -=
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	118 NODE_CHILDREN_ALLOC_SIZE(node->child_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	119 for (i = 0; i < node->child_count; i++)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	120 node_free(trie, &children[i]);
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	121
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	122 i_free(node->children.data);
4898 07a038b57946 Optimization. Timo Sirainen <tss@iki.fi> parents: 4897 diff changeset	123 }
07a038b57946 Optimization. Timo Sirainen <tss@iki.fi> parents: 4897 diff changeset	124 }
07a038b57946 Optimization. Timo Sirainen <tss@iki.fi> parents: 4897 diff changeset	125
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	126 struct squat_trie *
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	127 squat_trie_init(const char *path, uint32_t uidvalidity,
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	128 enum file_lock_method lock_method, bool mmap_disable)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	129 {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	130 struct squat_trie *trie;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	131
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	132 trie = i_new(struct squat_trie, 1);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	133 trie->path = i_strdup(path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	134 trie->uidlist = squat_uidlist_init(trie);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	135 trie->fd = -1;
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	136 trie->lock_method = lock_method;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	137 trie->uidvalidity = uidvalidity;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	138 trie->mmap_disable = mmap_disable;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	139 squat_trie_normalize_map_build(trie);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	140 return trie;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	141 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	142
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	143 static void squat_trie_close_fd(struct squat_trie *trie)
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	144 {
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	145 trie->data = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	146 trie->data_size = 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	147
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	148 if (trie->mmap_size != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	149 if (munmap(trie->mmap_base, trie->mmap_size) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	150 i_error("munmap(%s) failed: %m", trie->path);
7224 d5116fa2c88f Make sure mmap_* variables are reset after munmap() so we don't try to Timo Sirainen <tss@iki.fi> parents: 7223 diff changeset	151 trie->mmap_base = NULL;
d5116fa2c88f Make sure mmap_* variables are reset after munmap() so we don't try to Timo Sirainen <tss@iki.fi> parents: 7223 diff changeset	152 trie->mmap_size = 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	153 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	154 if (trie->fd != -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	155 if (close(trie->fd) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	156 i_error("close(%s) failed: %m", trie->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	157 trie->fd = -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	158 }
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	159 }
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	160
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	161 static void squat_trie_close(struct squat_trie *trie)
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	162 {
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	163 trie->corrupted = FALSE;
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	164 node_free(trie, &trie->root);
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	165 memset(&trie->root, 0, sizeof(trie->root));
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	166 memset(&trie->hdr, 0, sizeof(trie->hdr));
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	167
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	168 squat_trie_close_fd(trie);
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	169 if (trie->file_cache != NULL)
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	170 file_cache_free(&trie->file_cache);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	171 trie->locked_file_size = 0;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	172 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	173
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	174 void squat_trie_deinit(struct squat_trie **_trie)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	175 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	176 struct squat_trie trie = _trie;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	177
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	178 *_trie = NULL;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	179 squat_trie_close(trie);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	180 squat_uidlist_deinit(trie->uidlist);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	181 i_free(trie->path);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	182 i_free(trie);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	183 }
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	184
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	185 static void squat_trie_header_init(struct squat_trie *trie)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	186 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	187 memset(&trie->hdr, 0, sizeof(trie->hdr));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	188 trie->hdr.version = SQUAT_TRIE_VERSION;
7159 2de2058a5cdc Don't use ioloop_time. Fixes squat-test which doesn't create ioloop. Timo Sirainen <tss@iki.fi> parents: 7158 diff changeset	189 trie->hdr.indexid = time(NULL);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	190 trie->hdr.uidvalidity = trie->uidvalidity;
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	191 trie->hdr.partial_len = DEFAULT_PARTIAL_LEN;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	192 trie->hdr.full_len = DEFAULT_FULL_LEN;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	193
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	194 i_assert(sizeof(trie->hdr.normalize_map) ==
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	195 sizeof(trie->default_normalize_map));
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	196 memcpy(trie->hdr.normalize_map, trie->default_normalize_map,
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	197 sizeof(trie->hdr.normalize_map));
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	198 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	199
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	200 static int squat_trie_open_fd(struct squat_trie *trie)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	201 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	202 trie->fd = open(trie->path, O_RDWR);
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	203 if (trie->fd == -1) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	204 if (errno == ENOENT) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	205 squat_trie_header_init(trie);
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	206 return 0;
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	207 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	208 i_error("open(%s) failed: %m", trie->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	209 return -1;
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	210 }
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	211 if (trie->file_cache != NULL)
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	212 file_cache_set_fd(trie->file_cache, trie->fd);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	213 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	214 }
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	215
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	216 static int squat_trie_open(struct squat_trie *trie)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	217 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	218 squat_trie_close(trie);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	219
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	220 if (squat_trie_open_fd(trie) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	221 return -1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	222 return squat_trie_map(trie, FALSE);
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	223 }
88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	224
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	225 static int squat_trie_is_file_stale(struct squat_trie *trie)
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	226 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	227 struct stat st, st2;
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	228
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	229 if (stat(trie->path, &st) < 0) {
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	230 if (errno == ENOENT)
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	231 return 1;
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	232
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	233 i_error("stat(%s) failed: %m", trie->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	234 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	235 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	236 if (fstat(trie->fd, &st2) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	237 i_error("fstat(%s) failed: %m", trie->path);
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	238 return -1;
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	239 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	240 trie->locked_file_size = st2.st_size;
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	241
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	242 if (st.st_ino == st2.st_ino && CMP_DEV_T(st.st_dev, st2.st_dev)) {
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	243 i_assert(trie->locked_file_size >= trie->data_size);
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	244 return 0;
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	245 }
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	246 return 1;
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	247 }
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	248
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	249 void squat_trie_refresh(struct squat_trie *trie)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	250 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	251 if (squat_trie_is_file_stale(trie) > 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	252 (void)squat_trie_open(trie);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	253 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	254
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	255 static int squat_trie_lock(struct squat_trie *trie, int lock_type,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	256 struct file_lock **file_lock_r)
4886 ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	257 {
ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	258 int ret;
ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	259
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	260 while (trie->fd != -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	261 ret = file_wait_lock(trie->fd, trie->path, lock_type,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	262 trie->lock_method, SQUAT_TRIE_LOCK_TIMEOUT,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	263 file_lock_r);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	264 if (ret == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	265 i_error("file_wait_lock(%s) failed: %m", trie->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	266 return 0;
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	267 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	268 if (ret < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	269 return -1;
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	270
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	271 /* if the trie has been compressed, we need to reopen the
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	272 file and try to lock again */
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	273 ret = squat_trie_is_file_stale(trie);
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	274 if (ret == 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	275 return 1;
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	276
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	277 file_unlock(file_lock_r);
4879 b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	278 if (ret < 0)
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	279 return -1;
b0ada6e57b07 Fixes Timo Sirainen <tss@iki.fi> parents: 4878 diff changeset	280
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	281 squat_trie_close(trie);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	282 if (squat_trie_open_fd(trie) < 0)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	283 return -1;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	284 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	285 return 0;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	286 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	287
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	288 static void
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	289 node_make_squential(struct squat_trie trie, struct squat_node node, int level)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	290 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	291 const unsigned int alloc_size =
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	292 NODE_CHILDREN_ALLOC_SIZE(SEQUENTIAL_COUNT);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	293 struct squat_node *children;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	294 unsigned char *chars;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	295 unsigned int i;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	296
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	297 i_assert(node->child_count == 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	298
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	299 trie->node_alloc_size += alloc_size;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	300
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	301 node->want_sequential = FALSE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	302 node->have_sequential = TRUE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	303
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	304 node->child_count = SEQUENTIAL_COUNT;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	305 node->children.data = i_malloc(alloc_size);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	306
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	307 chars = NODE_CHILDREN_CHARS(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	308 for (i = 0; i < SEQUENTIAL_COUNT; i++)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	309 chars[i] = i;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	310
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	311 if (level < MAX_FAST_LEVEL) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	312 children = NODE_CHILDREN_NODES(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	313 for (i = 0; i < SEQUENTIAL_COUNT; i++)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	314 children[i].want_sequential = TRUE;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	315 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	316 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	317
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	318 static unsigned int
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	319 node_add_child(struct squat_trie trie, struct squat_node node,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	320 unsigned char chr, int level)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	321 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	322 unsigned int old_child_count = node->child_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	323 struct squat_node children, old_children;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	324 unsigned char *chars;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	325 size_t old_size, new_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	326
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	327 i_assert(node->leaf_string_length == 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	328
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	329 if (node->want_sequential) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	330 node_make_squential(trie, node, level);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	331
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	332 if (chr < SEQUENTIAL_COUNT)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	333 return chr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	334 old_child_count = SEQUENTIAL_COUNT;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	335 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	336
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	337 node->child_count++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	338 new_size = NODE_CHILDREN_ALLOC_SIZE(node->child_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	339
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	340 if (old_child_count == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	341 /* first child */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	342 node->children.data = i_malloc(new_size);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	343 trie->node_alloc_size += new_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	344 children = NODE_CHILDREN_NODES(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	345 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	346 old_size = NODE_CHILDREN_ALLOC_SIZE(old_child_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	347 if (old_size != new_size) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	348 trie->node_alloc_size += new_size - old_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	349 node->children.data = i_realloc(node->children.data,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	350 old_size, new_size);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	351 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	352
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	353 children = NODE_CHILDREN_NODES(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	354 old_children = (void *)(NODE_CHILDREN_CHARS(node) +
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	355 MEM_ALIGN(old_child_count));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	356 if (children != old_children) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	357 memmove(children, old_children,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	358 old_child_count * sizeof(struct squat_node));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	359 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	360 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	361
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	362 chars = NODE_CHILDREN_CHARS(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	363 chars[node->child_count - 1] = chr;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	364 return node->child_count - 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	365 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	366
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	367 static int
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	368 trie_file_cache_read(struct squat_trie *trie, size_t offset, size_t size)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	369 {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	370 if (trie->file_cache == NULL)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	371 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	372
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	373 if (file_cache_read(trie->file_cache, offset, size) < 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	374 i_error("read(%s) failed: %m", trie->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	375 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	376 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	377 trie->data = file_cache_get_map(trie->file_cache, &trie->data_size);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	378 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	379 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	380
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	381 static int
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	382 node_read_children(struct squat_trie trie, struct squat_node node, int level)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	383 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	384 const uint8_t data, end;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	385 const unsigned char *child_chars;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	386 struct squat_node child, children = NULL;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	387 uoff_t node_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	388 unsigned int i, child_idx, child_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	389 uoff_t base_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	390 uint32_t num;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	391
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	392 i_assert(node->children_not_mapped);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	393 i_assert(!node->have_sequential);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	394 i_assert(trie->unmapped_child_count > 0);
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	395 i_assert(trie->data_size <= trie->locked_file_size);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	396
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	397 trie->unmapped_child_count--;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	398 node_offset = node->children.offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	399 node->children_not_mapped = FALSE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	400 node->children.data = NULL;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	401
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	402 if (trie_file_cache_read(trie, node_offset, TRIE_READAHEAD_SIZE) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	403 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	404 if (unlikely(node_offset >= trie->data_size)) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	405 squat_trie_set_corrupted(trie);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	406 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	407 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	408
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	409 data = CONST_PTR_OFFSET(trie->data, node_offset);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	410 end = CONST_PTR_OFFSET(trie->data, trie->data_size);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	411 child_count = *data++;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	412 if (unlikely(node_offset + child_count >= trie->data_size)) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	413 squat_trie_set_corrupted(trie);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	414 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	415 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	416
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	417 if (child_count == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	418 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	419
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	420 child_chars = data;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	421 data += child_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	422
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	423 /* get child offsets */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	424 base_offset = node_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	425 for (i = 0; i < child_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	426 /* we always start with !have_sequential, so at i=0 this
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	427 check always goes to add the first child */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	428 if (node->have_sequential && child_chars[i] < SEQUENTIAL_COUNT)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	429 child_idx = child_chars[i];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	430 else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	431 child_idx = node_add_child(trie, node, child_chars[i],
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	432 level);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	433 children = NODE_CHILDREN_NODES(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	434 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	435 child = &children[child_idx];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	436
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	437 /* 1) child offset */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	438 num = squat_unpack_num(&data, end);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	439 if (num == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	440 /* no children */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	441 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	442 if ((num & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	443 base_offset += num >> 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	444 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	445 base_offset -= num >> 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	446 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	447 if (base_offset >= trie->locked_file_size) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	448 squat_trie_set_corrupted(trie);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	449 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	450 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	451 trie->unmapped_child_count++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	452 child->children_not_mapped = TRUE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	453 child->children.offset = base_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	454 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	455
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	456 /* 2) uidlist */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	457 child->uid_list_idx = squat_unpack_num(&data, end);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	458 if (child->uid_list_idx == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	459 /* we don't write nodes with empty uidlists */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	460 squat_trie_set_corrupted(trie);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	461 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	462 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	463 if (!UIDLIST_IS_SINGLETON(child->uid_list_idx)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	464 /* 3) next uid */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	465 child->next_uid = squat_unpack_num(&data, end) + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	466 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	467 uint32_t idx = child->uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	468
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	469 child->next_uid = 1 +
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	470 squat_uidlist_singleton_last_uid(idx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	471 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	472
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	473 /* 4) unused uids + leaf string flag */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	474 num = squat_unpack_num(&data, end);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	475 child->unused_uids = num >> 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	476 if ((num & 1) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	477 /* leaf string */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	478 unsigned int len;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	479 unsigned char *dest;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	480
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	481 /* 5) leaf string length */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	482 len = child->leaf_string_length =
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	483 squat_unpack_num(&data, end) + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	484 if (!NODE_IS_DYNAMIC_LEAF(child))
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	485 dest = child->children.static_leaf_string;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	486 else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	487 dest = child->children.leaf_string =
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	488 i_malloc(len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	489 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	490
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	491 if (trie->file_cache != NULL) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	492 /* the string may be long -
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	493 recalculate the end pos */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	494 size_t offset, size;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	495
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	496 offset = (const char *)data -
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	497 (const char *)trie->data;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	498 size = len + TRIE_BYTES_LEFT(child_count - i);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	499
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	500 if (trie_file_cache_read(trie, offset,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	501 size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	502 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	503 data = CONST_PTR_OFFSET(trie->data, offset);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	504 end = CONST_PTR_OFFSET(trie->data,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	505 trie->data_size);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	506 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	507
7043 7708a8c166d6 Compiler warning fixes on 32bit systems. Timo Sirainen <tss@iki.fi> parents: 6945 diff changeset	508 if ((size_t)(end - data) < len) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	509 squat_trie_set_corrupted(trie);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	510 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	511 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	512 memcpy(dest, data, len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	513 data += len;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	514 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	515 }
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	516 if (unlikely(data == end)) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	517 /* we should never get this far */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	518 squat_trie_set_corrupted(trie);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	519 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	520 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	521 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	522 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	523
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	524 static void
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	525 node_write_children(struct squat_trie_build_context *ctx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	526 struct squat_node node, const uoff_t node_offsets)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	527 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	528 struct squat_node *children;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	529 const unsigned char *chars;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	530 uint8_t child_count, buf[SQUAT_PACK_MAX_SIZE * 5], *bufp;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	531 uoff_t base_offset;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	532 unsigned int i;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	533
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	534 chars = NODE_CHILDREN_CHARS(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	535 children = NODE_CHILDREN_NODES(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	536
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	537 base_offset = ctx->output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	538 child_count = node->child_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	539 o_stream_send(ctx->output, &child_count, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	540 o_stream_send(ctx->output, chars, child_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	541
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	542 for (i = 0; i < child_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	543 bufp = buf;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	544 /* 1) child offset */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	545 if (node_offsets[i] == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	546 *bufp++ = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	547 else if (node_offsets[i] >= base_offset) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	548 squat_pack_num(&bufp,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	549 ((node_offsets[i] - base_offset) << 1) \| 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	550 base_offset = node_offsets[i];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	551 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	552 squat_pack_num(&bufp,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	553 (base_offset - node_offsets[i]) << 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	554 base_offset = node_offsets[i];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	555 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	556
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	557 /* 2) uidlist */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	558 squat_pack_num(&bufp, children[i].uid_list_idx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	559 if (!UIDLIST_IS_SINGLETON(children[i].uid_list_idx)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	560 /* 3) next uid */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	561 squat_pack_num(&bufp, children[i].next_uid - 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	562 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	563
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	564 if (children[i].leaf_string_length == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	565 /* 4a) unused uids */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	566 squat_pack_num(&bufp, children[i].unused_uids << 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	567 o_stream_send(ctx->output, buf, bufp - buf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	568 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	569 i_assert(node_offsets[i] == 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	570 /* 4b) unused uids + flag */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	571 squat_pack_num(&bufp, (children[i].unused_uids << 1) \| 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	572 /* 5) leaf string length */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	573 squat_pack_num(&bufp, children[i].leaf_string_length - 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	574 o_stream_send(ctx->output, buf, bufp - buf);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	575 o_stream_send(ctx->output,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	576 NODE_LEAF_STRING(&children[i]),
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	577 children[i].leaf_string_length);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	578 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	579 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	580 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	581
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	582 static inline void
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	583 node_add_uid(struct squat_trie_build_context *ctx, uint32_t uid,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	584 struct squat_node *node)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	585 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	586 if (uid < node->next_uid) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	587 /* duplicate */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	588 return;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	589 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	590 node->unused_uids += uid - node->next_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	591 node->next_uid = uid + 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	592
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	593 node->uid_list_idx =
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	594 squat_uidlist_build_add_uid(ctx->uidlist_build_ctx,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	595 node->uid_list_idx, uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	596 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	597
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	598 static void
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	599 node_split_string(struct squat_trie_build_context ctx, struct squat_node node)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	600 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	601 struct squat_node *child;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	602 unsigned char *str;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	603 unsigned int uid, idx, str_len = node->leaf_string_length;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	604
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	605 i_assert(str_len > 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	606
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	607 /* make a copy of the leaf string and convert to normal node by
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	608 removing it. */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	609 str = t_malloc(str_len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	610 if (!NODE_IS_DYNAMIC_LEAF(node))
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	611 memcpy(str, node->children.static_leaf_string, str_len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	612 else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	613 memcpy(str, node->children.leaf_string, str_len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	614 i_free(node->children.leaf_string);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	615 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	616 node->leaf_string_length = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	617
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	618 /* create a new child node for the rest of the string */
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	619 idx = node_add_child(ctx->trie, node, str[0], MAX_FAST_LEVEL);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	620 child = NODE_CHILDREN_NODES(node) + idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	621
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	622 /* update uidlist to contain all of parent's UIDs */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	623 child->next_uid = node->next_uid - node->unused_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	624 for (uid = 0; uid < child->next_uid; uid++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	625 child->uid_list_idx =
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	626 squat_uidlist_build_add_uid(ctx->uidlist_build_ctx,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	627 child->uid_list_idx, uid);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	628 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	629
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	630 i_assert(!child->have_sequential && child->children.data == NULL);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	631 if (str_len > 1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	632 /* make the child a leaf string */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	633 str_len--;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	634 child->leaf_string_length = str_len;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	635 if (!NODE_IS_DYNAMIC_LEAF(child)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	636 memcpy(child->children.static_leaf_string,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	637 str + 1, str_len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	638 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	639 child->children.leaf_string = i_malloc(str_len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	640 memcpy(child->children.leaf_string, str + 1, str_len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	641 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	642 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	643 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	644
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	645 static bool
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	646 node_leaf_string_add_or_split(struct squat_trie_build_context *ctx,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	647 struct squat_node *node,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	648 const unsigned char *data, unsigned int data_len)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	649 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	650 const unsigned char *str = NODE_LEAF_STRING(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	651 const unsigned int str_len = node->leaf_string_length;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	652 unsigned int i;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	653
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	654 if (data_len != str_len) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	655 /* different lengths, can't match */
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	656 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	657 node_split_string(ctx, node);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	658 } T_END;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	659 return FALSE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	660 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	661
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	662 for (i = 0; i < data_len; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	663 if (data[i] != str[i]) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	664 /* non-match */
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	665 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	666 node_split_string(ctx, node);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	667 } T_END;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	668 return FALSE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	669 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	670 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	671 return TRUE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	672 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	673
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	674 static int squat_build_add(struct squat_trie_build_context *ctx, uint32_t uid,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	675 const unsigned char *data, unsigned int size)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	676 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	677 struct squat_trie *trie = ctx->trie;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	678 struct squat_node *node = &trie->root;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	679 const unsigned char *end = data + size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	680 unsigned char *chars;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	681 unsigned int idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	682 int level = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	683
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	684 for (;;) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	685 if (node->children_not_mapped) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	686 if (unlikely(node_read_children(trie, node, level) < 0))
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	687 return -1;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	688 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	689
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	690 if (node->leaf_string_length != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	691 /* the whole string must match or we need to split
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	692 the node */
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	693 if (node_leaf_string_add_or_split(ctx, node, data,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	694 end - data)) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	695 node_add_uid(ctx, uid, node);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	696 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	697 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	698 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	699
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	700 node_add_uid(ctx, uid, node);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	701
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	702 if (unlikely(uid < node->unused_uids)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	703 squat_trie_set_corrupted(trie);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	704 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	705 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	706 /* child node's UIDs are relative to ours. so for example if
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	707 we're adding UID 4 and this node now has [2,4] UIDs,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	708 unused_uids=3 and so the child node will be adding
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	709 UID 4-3 = 1. */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	710 uid -= node->unused_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	711
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	712 if (data == end)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	713 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	714 level++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	715
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	716 if (node->have_sequential) {
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	717 i_assert(node->child_count >= SEQUENTIAL_COUNT);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	718 if (*data < SEQUENTIAL_COUNT) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	719 idx = *data;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	720 goto found;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	721 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	722 idx = SEQUENTIAL_COUNT;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	723 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	724 idx = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	725 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	726 chars = NODE_CHILDREN_CHARS(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	727 for (; idx < node->child_count; idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	728 if (chars[idx] == *data)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	729 goto found;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	730 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	731 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	732 found:
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	733 data++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	734 node = NODE_CHILDREN_NODES(node) + idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	735 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	736
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	737 /* create new children */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	738 i_assert(node->leaf_string_length == 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	739
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	740 for (;;) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	741 idx = node_add_child(trie, node, *data,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	742 size - (end - data) + 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	743 node = NODE_CHILDREN_NODES(node) + idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	744
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	745 node_add_uid(ctx, uid, node);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	746 uid = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	747
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	748 if (++data == end)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	749 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	750
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	751 if (!node->have_sequential) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	752 /* convert the node into a leaf string */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	753 unsigned int len = end - data;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	754
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	755 i_assert(node->children.data == NULL);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	756 node->leaf_string_length = len;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	757 if (!NODE_IS_DYNAMIC_LEAF(node)) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	758 memcpy(node->children.static_leaf_string,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	759 data, len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	760 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	761 node->children.leaf_string = i_malloc(len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	762 memcpy(node->children.leaf_string, data, len);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	763 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	764 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	765 }
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	766 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	767 return 0;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	768 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	769
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	770 static int
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	771 squat_build_word_bytes(struct squat_trie_build_context *ctx, uint32_t uid,
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	772 const unsigned char *data, unsigned int size)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	773 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	774 struct squat_trie *trie = ctx->trie;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	775 unsigned int i;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	776
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	777 if (trie->hdr.full_len <= trie->hdr.partial_len)
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	778 i = 0;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	779 else {
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	780 /* the first word is longer than others */
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	781 if (squat_build_add(ctx, uid, data,
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	782 I_MIN(size, trie->hdr.full_len)) < 0)
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	783 return -1;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	784 i = 1;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	785 }
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	786
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	787 for (; i < size; i++) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	788 if (squat_build_add(ctx, uid, data + i,
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	789 I_MIN(trie->hdr.partial_len, size-i)) < 0)
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	790 return -1;
d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	791 }
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	792 return 0;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	793 }
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	794
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	795 static int
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	796 squat_build_word(struct squat_trie_build_context *ctx, uint32_t uid,
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	797 const unsigned char data, const uint8_t char_lengths,
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	798 unsigned int size)
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	799 {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	800 struct squat_trie *trie = ctx->trie;
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	801 unsigned int i, j, bytelen;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	802
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	803 if (char_lengths == NULL) {
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	804 /* optimization path: all characters are bytes */
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	805 return squat_build_word_bytes(ctx, uid, data, size);
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	806 }
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	807
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	808 if (trie->hdr.full_len <= trie->hdr.partial_len)
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	809 i = 0;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	810 else {
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	811 /* the first word is longer than others */
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	812 bytelen = 0;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	813 for (j = 0; j < trie->hdr.full_len && bytelen < size; j++)
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	814 bytelen += char_lengths[bytelen];
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	815 i_assert(bytelen <= size);
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	816
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	817 if (squat_build_add(ctx, uid, data, bytelen) < 0)
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	818 return -1;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	819 i = char_lengths[0];
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	820 }
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	821
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	822 for (; i < size; i += char_lengths[i]) {
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	823 bytelen = 0;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	824 for (j = 0; j < trie->hdr.partial_len && i+bytelen < size; j++)
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	825 bytelen += char_lengths[i + bytelen];
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	826 i_assert(i + bytelen <= size);
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	827
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	828 if (squat_build_add(ctx, uid, data + i, bytelen) < 0)
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	829 return -1;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	830 }
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	831 return 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	832 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	833
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	834 static unsigned char *
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	835 squat_data_normalize(struct squat_trie trie, const unsigned char data,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	836 unsigned int size)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	837 {
7187 d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	838 static const unsigned char replacement_utf8[] = { 0xef, 0xbf, 0xbd };
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	839 unsigned char *dest;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	840 unsigned int i;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	841
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	842 dest = t_malloc(size);
7187 d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	843 for (i = 0; i < size; i++) {
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	844 if (data[i] == replacement_utf8[0] && i + 2 < size &&
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	845 data[i+1] == replacement_utf8[1] &&
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	846 data[i+2] == replacement_utf8[2]) {
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	847 /* Don't index replacement character */
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	848 dest[i++] = 0;
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	849 dest[i++] = 0;
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	850 dest[i] = 0;
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	851 } else {
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	852 dest[i] = trie->hdr.normalize_map[data[i]];
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	853 }
d9b87e3ce6c8 Treat replacement characters as non-indexed chars. Timo Sirainen <tss@iki.fi> parents: 7161 diff changeset	854 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	855 return dest;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	856 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	857
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	858 static int
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	859 squat_trie_build_more_real(struct squat_trie_build_context *ctx,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	860 uint32_t uid, enum squat_index_type type,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	861 const unsigned char *input, unsigned int size)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	862 {
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	863 struct squat_trie *trie = ctx->trie;
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	864 const unsigned char *data;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	865 uint8_t *char_lengths;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	866 unsigned int i, start = 0;
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	867 bool multibyte_chars = FALSE;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	868 int ret = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	869
6914 9c3f0e180751 Index message's all headers (including MIME headers) after the body. Timo Sirainen <tss@iki.fi> parents: 6912 diff changeset	870 uid = uid * 2 + (type == SQUAT_INDEX_TYPE_HEADER ? 1 : 0);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	871
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	872 char_lengths = t_malloc(size);
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	873 data = squat_data_normalize(trie, input, size);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	874 for (i = 0; i < size; i++) {
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	875 char_lengths[i] = uni_utf8_char_bytes(input[i]);
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	876 if (char_lengths[i] != 1)
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	877 multibyte_chars = TRUE;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	878 if (data[i] != '\0')
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	879 continue;
4886 ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	880
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	881 while (start < i && data[start] == '\0')
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	882 start++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	883 if (i != start) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	884 if (squat_build_word(ctx, uid, data + start,
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	885 !multibyte_chars ? NULL :
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	886 char_lengths + start,
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	887 i - start) < 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	888 ret = -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	889 start = i;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	890 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	891 }
4883 d8adbe93c969 Added support for mmap_disable=yes and several other fixes. Timo Sirainen <tss@iki.fi> parents: 4879 diff changeset	892 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	893 start = i + 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	894 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	895 while (start < i && data[start] == '\0')
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	896 start++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	897 if (i != start) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	898 if (squat_build_word(ctx, uid, data + start,
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	899 !multibyte_chars ? NULL :
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	900 char_lengths + start, i - start) < 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	901 ret = -1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	902 }
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	903 return ret;
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	904 }
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	905
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	906 int squat_trie_build_more(struct squat_trie_build_context *ctx,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	907 uint32_t uid, enum squat_index_type type,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	908 const unsigned char *input, unsigned int size)
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	909 {
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	910 int ret;
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	911
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	912 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	913 ret = squat_trie_build_more_real(ctx, uid, type, input, size);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	914 } T_END;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	915 return ret;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	916 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	917
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	918 static void node_drop_unused_children(struct squat_node *node)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	919 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	920 unsigned char *chars;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	921 struct squat_node children_src, children_dest;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	922 unsigned int i, j, orig_child_count = node->child_count;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	923
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	924 chars = NODE_CHILDREN_CHARS(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	925 children_src = NODE_CHILDREN_NODES(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	926
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	927 /* move chars */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	928 for (i = j = 0; i < orig_child_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	929 if (children_src[i].next_uid != 0)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	930 chars[j++] = chars[i];
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	931 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	932 node->child_count = j;
4890 cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	933
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	934 /* move children. note that children_dest may point to different
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	935 location than children_src, although they both point to the
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	936 same node. */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	937 children_dest = NODE_CHILDREN_NODES(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	938 for (i = j = 0; i < orig_child_count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	939 if (children_src[i].next_uid != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	940 children_dest[j++] = children_src[i];
4890 cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	941 }
cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	942 }
cb3b4153136c 64bit and big endian fixes Timo Sirainen <tss@iki.fi> parents: 4887 diff changeset	943
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	944 static int
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	945 squat_write_node(struct squat_trie_build_context ctx, struct squat_node node,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	946 uoff_t *node_offset_r, int level)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	947 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	948 struct squat_trie *trie = ctx->trie;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	949 struct squat_node *children;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	950 unsigned int i;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	951 uoff_t *node_offsets;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	952 uint8_t child_count;
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	953 int ret;
4901 5e604d6f391e Only ASCII texts were actually indexed. Now UTF-8 input is properly Timo Sirainen <tss@iki.fi> parents: 4898 diff changeset	954
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	955 i_assert(node->next_uid != 0);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	956
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	957 if (node->children_not_mapped && ctx->compress_nodes) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	958 if (node_read_children(trie, node, MAX_FAST_LEVEL) < 0)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	959 return -1;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	960 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	961
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	962 node->have_sequential = FALSE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	963 node_drop_unused_children(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	964
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	965 child_count = node->child_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	966 if (child_count == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	967 i_assert(!node->children_not_mapped \|\|
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	968 node->leaf_string_length == 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	969 *node_offset_r = !node->children_not_mapped ? 0 :
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	970 node->children.offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	971 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	972 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	973 i_assert(!node->children_not_mapped);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	974
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	975 trie->hdr.node_count++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	976
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	977 children = NODE_CHILDREN_NODES(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	978 node_offsets = t_new(uoff_t, child_count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	979 for (i = 0; i < child_count; i++) {
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	980 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	981 ret = squat_write_node(ctx, &children[i],
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	982 &node_offsets[i], level + 1);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	983 } T_END;
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	984 if (ret < 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	985 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	986 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	987
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	988 *node_offset_r = ctx->output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	989 node_write_children(ctx, node, node_offsets);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	990 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	991 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	992
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	993 static int squat_write_nodes(struct squat_trie_build_context *ctx)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	994 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	995 struct squat_trie *trie = ctx->trie;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	996 uoff_t node_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	997 int ret;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	998
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	999 if (ctx->trie->root.next_uid == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1000 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1001
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	1002 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1003 ret = squat_write_node(ctx, &ctx->trie->root, &node_offset, 0);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	1004 } T_END;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1005 if (ret < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1006 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1007
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1008 trie->hdr.root_offset = node_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1009 trie->hdr.root_unused_uids = trie->root.unused_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1010 trie->hdr.root_next_uid = trie->root.next_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1011 trie->hdr.root_uidlist_idx = trie->root.uid_list_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1012 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1013 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1014
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1015 static struct squat_trie_iterate_context *
7197 bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1016 squat_trie_iterate_init(struct squat_trie *trie)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1017 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1018 struct squat_trie_iterate_context *ctx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1019
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1020 ctx = i_new(struct squat_trie_iterate_context, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1021 ctx->trie = trie;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1022 ctx->cur.node = &trie->root;
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1023 i_array_init(&ctx->parents, trie->hdr.partial_len*2);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1024 return ctx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1025 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1026
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	1027 static int
7197 bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1028 squat_trie_iterate_deinit(struct squat_trie_iterate_context *ctx)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1029 {
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1030 struct squat_trie_iterate_node *nodes;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1031 unsigned int i, count;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1032 int ret = ctx->failed ? -1 : 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1033
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1034 if (array_is_created(&ctx->cur.shifts)) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1035 nodes = array_get_modifiable(&ctx->parents, &count);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1036 for (i = 0; i < count; i++)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1037 array_free(&nodes[i].shifts);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1038 array_free(&ctx->cur.shifts);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1039 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1040 array_free(&ctx->parents);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1041 i_free(ctx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1042 return ret;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1043 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1044
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1045 static struct squat_node *
7197 bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1046 squat_trie_iterate_first(struct squat_trie_iterate_context *ctx)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1047 {
7197 bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1048 if (ctx->cur.node->children_not_mapped) {
bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1049 if (node_read_children(ctx->trie, ctx->cur.node, 1) < 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1050 ctx->failed = TRUE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1051 return NULL;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1052 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1053 }
7197 bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1054 return ctx->cur.node;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1055 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1056
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1057 static struct squat_node *
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1058 squat_trie_iterate_next(struct squat_trie_iterate_context *ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1059 ARRAY_TYPE(seq_range) *shifts_r)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1060 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1061 struct squat_trie_iterate_node *iter_nodes;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1062 struct squat_node *children;
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1063 unsigned int count, shift_count = 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1064
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1065 while (ctx->cur.idx == ctx->cur.node->child_count \|\|
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1066 ctx->cur.node->uid_list_idx == 0)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1067 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1068 iter_nodes = array_get_modifiable(&ctx->parents, &count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1069 if (count == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1070 return NULL;
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1071
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1072 if (array_is_created(&ctx->cur.shifts))
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1073 array_free(&ctx->cur.shifts);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1074 ctx->cur = iter_nodes[count-1];
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1075 array_delete(&ctx->parents, count-1, 1);
7197 bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1076 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1077
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1078 *shifts_r = ctx->cur.shifts;
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7219 diff changeset	1079 if (array_is_created(&ctx->cur.shifts))
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1080 shift_count = array_count(&ctx->cur.shifts);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1081
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1082 children = NODE_CHILDREN_NODES(ctx->cur.node);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1083 while (children[ctx->cur.idx++].uid_list_idx == 0) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1084 if (ctx->cur.idx == ctx->cur.node->child_count) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1085 /* no more non-empty children in this node */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1086 return squat_trie_iterate_next(ctx, shifts_r);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1087 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1088 }
7197 bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1089 array_append(&ctx->parents, &ctx->cur, 1);
bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1090 ctx->cur.node = &children[ctx->cur.idx-1];
bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1091 ctx->cur.idx = 0;
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1092 if (shift_count != 0)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1093 i_array_init(&ctx->cur.shifts, shift_count);
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7219 diff changeset	1094 else
21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7219 diff changeset	1095 memset(&ctx->cur.shifts, 0, sizeof(ctx->cur.shifts));
7219 6c48b72c5096 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7218 diff changeset	1096 return squat_trie_iterate_first(ctx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1097 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1098
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1099 static void
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1100 squat_uidlist_update_expunged_uids(const ARRAY_TYPE(seq_range) *shifts_arr,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1101 ARRAY_TYPE(seq_range) *child_shifts,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1102 ARRAY_TYPE(seq_range) *uids_arr,
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1103 struct squat_trie *trie,
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1104 struct squat_node *node, bool do_shifts)
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1105 {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1106 const struct seq_range *shifts;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1107 struct seq_range *uids, shift;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1108 unsigned int i, uid_idx, uid_count, shift_count;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1109 uint32_t child_shift_seq1, child_shift_count, seq_high;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1110 unsigned int shift_sum = 0, child_sum = 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1111
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1112 if (!array_is_created(shifts_arr)) {
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1113 i_assert(node->uid_list_idx != 0 \|\| node->child_count == 0);
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7219 diff changeset	1114 return;
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1115 }
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1116
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1117 /* we'll recalculate this */
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1118 node->unused_uids = 0;
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7219 diff changeset	1119
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1120 uids = array_get_modifiable(uids_arr, &uid_count);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1121 shifts = array_get(shifts_arr, &shift_count);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1122 for (i = 0, uid_idx = 0, seq_high = 0;; ) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1123 /* skip UID ranges until we skip/overlap shifts */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1124 while (uid_idx < uid_count &&
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1125 (i == shift_count \|\|
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1126 I_MAX(shifts[i].seq1, seq_high) > uids[uid_idx].seq2))
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1127 {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1128 i_assert(uids[uid_idx].seq1 >= shift_sum);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1129 uids[uid_idx].seq1 -= shift_sum;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1130 uids[uid_idx].seq2 -= shift_sum;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1131 child_sum += uids[uid_idx].seq2 -
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1132 uids[uid_idx].seq1 + 1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1133
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1134 if (uid_idx > 0 &&
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1135 uids[uid_idx-1].seq2 >= uids[uid_idx].seq1 - 1) {
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1136 /* we can merge this and the previous range */
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1137 i_assert(uids[uid_idx-1].seq2 ==
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1138 uids[uid_idx].seq1 - 1);
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1139 uids[uid_idx-1].seq2 = uids[uid_idx].seq2;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1140 array_delete(uids_arr, uid_idx, 1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1141 uids = array_get_modifiable(uids_arr,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1142 &uid_count);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1143 } else {
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1144 if (uid_idx == 0)
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1145 node->unused_uids += uids[0].seq1;
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1146 else {
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1147 node->unused_uids +=
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1148 uids[uid_idx].seq1 -
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1149 uids[uid_idx-1].seq2 - 1;
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1150 }
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1151 uid_idx++;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1152 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1153 }
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7219 diff changeset	1154 if (uid_idx == uid_count)
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1155 break;
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7219 diff changeset	1156
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1157 shift.seq1 = I_MAX(shifts[i].seq1, seq_high);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1158 shift.seq2 = shifts[i].seq2;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1159 if (shift.seq2 < uids[uid_idx].seq1) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1160 /* shift is entirely before UID range */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1161 shift_sum += shift.seq2 - shift.seq1 + 1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1162 i++;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1163 } else {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1164 /* handle shifts before UID range */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1165 if (shift.seq1 < uids[uid_idx].seq1) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1166 shift_sum += uids[uid_idx].seq1 - shift.seq1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1167 shift.seq1 = uids[uid_idx].seq1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1168 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1169 /* update child shifts */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1170 child_shift_seq1 = child_sum +
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1171 shift.seq1 - uids[uid_idx].seq1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1172 child_shift_count =
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1173 I_MIN(shift.seq2, uids[uid_idx].seq2) -
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1174 shift.seq1 + 1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1175 seq_range_array_add_range(child_shifts,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1176 child_shift_seq1,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1177 child_shift_seq1 +
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1178 child_shift_count - 1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1179 child_sum += child_shift_count;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1180
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1181 /* if the shifts continue after the UID range,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1182 treat it in the next loop iteration */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1183 if (shift.seq2 <= uids[uid_idx].seq2)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1184 i++;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1185 else
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1186 seq_high = uids[uid_idx].seq2 + 1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1187
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1188 /* update UIDs - no matter where within the UID range
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1189 the shifts happened, the result is the same:
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1190 shift number of UIDs are removed, and the rest
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1191 are decreased by shift_sum.
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1192
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1193 123 uids child_shifts
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1194 a) s -> 12 1
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1195 b) s -> 12 2
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1196 c) s -> 12 3
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1197 */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1198 if (uids[uid_idx].seq1 +
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1199 child_shift_count > uids[uid_idx].seq2) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1200 /* removed completely */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1201 array_delete(uids_arr, uid_idx, 1);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1202 uids = array_get_modifiable(uids_arr,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1203 &uid_count);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1204 } else {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1205 /* the next loop iteration fixes the UIDs */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1206 uids[uid_idx].seq1 += child_shift_count;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1207 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1208 shift_sum += child_shift_count;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1209 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1210 if (!do_shifts) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1211 /* root node - UIDs are only removed, not shifted */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1212 shift_sum = 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1213 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1214 }
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1215
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1216 if (uid_count == 0) {
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1217 /* no UIDs left, delete the node's children and mark it
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1218 unused */
7223 71af49ba330e Assert crashfix Timo Sirainen <tss@iki.fi> parents: 7221 diff changeset	1219 if (!NODE_IS_DYNAMIC_LEAF(node))
71af49ba330e Assert crashfix Timo Sirainen <tss@iki.fi> parents: 7221 diff changeset	1220 node_free(trie, node);
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1221
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1222 node->child_count = 0;
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1223 node->have_sequential = FALSE;
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1224 node->next_uid = 0;
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1225 } else {
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1226 if (do_shifts)
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1227 node->next_uid = uids[uid_count-1].seq2 + 1;
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1228 else {
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1229 node->unused_uids += (node->next_uid - 1) -
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1230 uids[uid_count-1].seq2;
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1231 }
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1232 }
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1233 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1234
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1235 static int
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1236 squat_trie_expunge_uidlists(struct squat_trie_build_context *ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1237 struct squat_uidlist_rebuild_context *rebuild_ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1238 struct squat_trie_iterate_context *iter,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1239 const ARRAY_TYPE(seq_range) *expunged_uids)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1240 {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1241 struct squat_node *node;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1242 ARRAY_TYPE(seq_range) uid_range, shifts;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1243 bool shift = FALSE;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1244 int ret = 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1245
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1246 node = squat_trie_iterate_first(iter);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1247 if (node->uid_list_idx == 0)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1248 return 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1249
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1250 i_array_init(&uid_range, 1024);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1251 i_array_init(&shifts, array_count(expunged_uids));
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1252 array_append_array(&shifts, expunged_uids);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1253
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7219 diff changeset	1254 do {
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1255 i_assert(node->uid_list_idx != 0);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1256 array_clear(&uid_range);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1257 if (squat_uidlist_get_seqrange(ctx->trie->uidlist,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1258 node->uid_list_idx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1259 &uid_range) < 0) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1260 ret = -1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1261 break;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1262 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1263 squat_uidlist_update_expunged_uids(&shifts, &iter->cur.shifts,
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1264 &uid_range, ctx->trie, node,
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1265 shift);
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1266 node->uid_list_idx =
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1267 squat_uidlist_rebuild_nextu(rebuild_ctx, &uid_range);
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1268 i_assert(node->uid_list_idx != 0 \|\| node->next_uid == 0);
e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1269
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1270 node = squat_trie_iterate_next(iter, &shifts);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1271 shift = TRUE;
7220 21d1d0ced13e Expunge fixes Timo Sirainen <tss@iki.fi> parents: 7219 diff changeset	1272 } while (node != NULL);
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1273 array_free(&uid_range);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1274 return ret;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1275 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1276
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1277 static int
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1278 squat_trie_renumber_uidlists2(struct squat_trie_build_context *ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1279 struct squat_uidlist_rebuild_context *rebuild_ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1280 struct squat_trie_iterate_context *iter)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1281 {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1282 struct squat_node *node;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1283 ARRAY_TYPE(seq_range) shifts;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1284 ARRAY_TYPE(uint32_t) uids;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1285 int ret = 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1286
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1287 node = squat_trie_iterate_first(iter);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1288 if (node->uid_list_idx == 0)
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1289 return 0;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1290
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1291 i_array_init(&uids, 1024);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1292 while (node != NULL) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1293 i_assert(node->uid_list_idx != 0);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1294 if (!UIDLIST_IS_SINGLETON(node->uid_list_idx)) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1295 /* rebuild the uidlist */
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1296 array_clear(&uids);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1297 if (squat_uidlist_get(ctx->trie->uidlist,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1298 node->uid_list_idx, &uids) < 0) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1299 ret = -1;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1300 break;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1301 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1302 node->uid_list_idx =
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1303 squat_uidlist_rebuild_next(rebuild_ctx, &uids);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1304 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1305 node = squat_trie_iterate_next(iter, &shifts);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1306 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1307 array_free(&uids);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1308 return ret;
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1309 }
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1310
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1311 static int
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1312 squat_trie_renumber_uidlists(struct squat_trie_build_context *ctx,
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1313 const ARRAY_TYPE(seq_range) *expunged_uids,
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1314 bool compress)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1315 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1316 struct squat_trie_iterate_context *iter;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1317 struct squat_uidlist_rebuild_context *rebuild_ctx;
7159 2de2058a5cdc Don't use ioloop_time. Fixes squat-test which doesn't create ioloop. Timo Sirainen <tss@iki.fi> parents: 7158 diff changeset	1318 time_t now;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1319 int ret = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1320
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1321 if ((ret = squat_uidlist_rebuild_init(ctx->uidlist_build_ctx,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1322 compress, &rebuild_ctx)) <= 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1323 return ret;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1324
7159 2de2058a5cdc Don't use ioloop_time. Fixes squat-test which doesn't create ioloop. Timo Sirainen <tss@iki.fi> parents: 7158 diff changeset	1325 now = time(NULL);
7043 7708a8c166d6 Compiler warning fixes on 32bit systems. Timo Sirainen <tss@iki.fi> parents: 6945 diff changeset	1326 ctx->trie->hdr.indexid =
7159 2de2058a5cdc Don't use ioloop_time. Fixes squat-test which doesn't create ioloop. Timo Sirainen <tss@iki.fi> parents: 7158 diff changeset	1327 I_MAX((unsigned int)now, ctx->trie->hdr.indexid + 1);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1328
7197 bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1329 iter = squat_trie_iterate_init(ctx->trie);
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1330 if (expunged_uids != NULL) {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1331 ret = squat_trie_expunge_uidlists(ctx, rebuild_ctx, iter,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1332 expunged_uids);
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1333 } else {
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1334 ret = squat_trie_renumber_uidlists2(ctx, rebuild_ctx, iter);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1335 }
7197 bb9304dee6d4 uidlist renumbering: changed iteration code to iterate children first. This Timo Sirainen <tss@iki.fi> parents: 7187 diff changeset	1336 squat_trie_iterate_deinit(iter);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1337
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1338 /* lock the trie before we rename uidlist */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1339 if (squat_trie_lock(ctx->trie, F_WRLCK, &ctx->file_lock) <= 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1340 ret = -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1341 return squat_uidlist_rebuild_finish(rebuild_ctx, ret < 0);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1342 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1343
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1344 static bool squat_trie_check_header(struct squat_trie *trie)
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1345 {
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1346 if (trie->hdr.version != SQUAT_TRIE_VERSION \|\|
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1347 trie->hdr.uidvalidity != trie->uidvalidity)
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1348 return FALSE;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1349
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1350 if (trie->hdr.partial_len > trie->hdr.full_len) {
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1351 i_error("Corrupted %s: partial len > full len", trie->path);
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1352 return FALSE;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1353 }
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1354 if (trie->hdr.full_len == 0) {
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1355 i_error("Corrupted %s: full len=0", trie->path);
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1356 return FALSE;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1357 }
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1358 return TRUE;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1359 }
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1360
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1361 static int squat_trie_map_header(struct squat_trie *trie)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1362 {
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1363 int ret;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1364
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1365 if (trie->locked_file_size == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1366 /* newly created file */
7158 f41bd3065ec8 Init header for newly created files. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	1367 squat_trie_header_init(trie);
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1368 return 1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1369 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1370 i_assert(trie->fd != -1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1371
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1372 if (trie->mmap_disable) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1373 ret = pread_full(trie->fd, &trie->hdr, sizeof(trie->hdr), 0);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1374 if (ret <= 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1375 if (ret < 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1376 i_error("pread(%s) failed: %m", trie->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1377 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1378 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1379 i_error("Corrupted %s: File too small", trie->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1380 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1381 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1382 trie->data = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1383 trie->data_size = 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1384 } else {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1385 if (trie->locked_file_size < sizeof(trie->hdr)) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1386 i_error("Corrupted %s: File too small", trie->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1387 return 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1388 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1389 if (trie->mmap_size != 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1390 if (munmap(trie->mmap_base, trie->mmap_size) < 0)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1391 i_error("munmap(%s) failed: %m", trie->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1392 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1393
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1394 trie->mmap_size = trie->locked_file_size;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1395 trie->mmap_base = mmap(NULL, trie->mmap_size,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1396 PROT_READ \| PROT_WRITE,
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1397 MAP_SHARED, trie->fd, 0);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1398 if (trie->mmap_base == MAP_FAILED) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1399 trie->data = trie->mmap_base = NULL;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1400 trie->data_size = trie->mmap_size = 0;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1401 i_error("mmap(%s) failed: %m", trie->path);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1402 return -1;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1403 }
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1404 memcpy(&trie->hdr, trie->mmap_base, sizeof(trie->hdr));
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1405 trie->data = trie->mmap_base;
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1406 trie->data_size = trie->mmap_size;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1407 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1408
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1409 return squat_trie_check_header(trie) ? 1 : 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1410 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1411
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1412 static int squat_trie_map(struct squat_trie *trie, bool building)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1413 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1414 struct file_lock *file_lock = NULL;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1415 bool changed;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1416 int ret;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1417
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1418 if (trie->fd != -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1419 if (squat_trie_lock(trie, F_RDLCK, &file_lock) <= 0)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1420 return -1;
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1421 if (trie->mmap_disable && trie->file_cache == NULL)
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1422 trie->file_cache = file_cache_new(trie->fd);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1423 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1424
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1425 ret = squat_trie_map_header(trie);
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1426 if (ret == 0) {
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1427 file_lock_free(&file_lock);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1428 squat_trie_delete(trie);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1429 squat_trie_close(trie);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1430 squat_trie_header_init(trie);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1431 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1432 changed = trie->root.children.offset != trie->hdr.root_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1433
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1434 if (changed \|\| trie->hdr.root_offset == 0) {
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1435 node_free(trie, &trie->root);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1436 memset(&trie->root, 0, sizeof(trie->root));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1437 trie->root.want_sequential = TRUE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1438 trie->root.unused_uids = trie->hdr.root_unused_uids;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1439 trie->root.next_uid = trie->hdr.root_next_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1440 trie->root.uid_list_idx = trie->hdr.root_uidlist_idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1441 trie->root.children.offset = trie->hdr.root_offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1442
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1443 if (trie->hdr.root_offset == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1444 trie->unmapped_child_count = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1445 trie->root.children_not_mapped = FALSE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1446 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1447 trie->unmapped_child_count = 1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1448 trie->root.children_not_mapped = TRUE;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1449 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1450 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1451
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1452 if (ret >= 0 && !building) {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1453 /* do this while we're still locked */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1454 ret = squat_uidlist_refresh(trie->uidlist);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1455 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1456
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1457 if (file_lock != NULL)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1458 file_unlock(&file_lock);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1459 if (ret < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1460 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1461
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1462 return trie->hdr.root_offset == 0 \|\| !changed ? 0 :
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1463 node_read_children(trie, &trie->root, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1464 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1465
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1466 int squat_trie_build_init(struct squat_trie trie, uint32_t last_uid_r,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1467 struct squat_trie_build_context **ctx_r)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1468 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1469 struct squat_trie_build_context *ctx;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1470 struct squat_uidlist_build_context *uidlist_build_ctx;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1471
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1472 if (trie->fd == -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1473 trie->fd = open(trie->path, O_RDWR \| O_CREAT, 0600);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1474 if (trie->fd == -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1475 i_error("creat(%s) failed: %m", trie->path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1476 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1477 }
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1478 if (trie->file_cache != NULL)
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1479 file_cache_set_fd(trie->file_cache, trie->fd);
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1480 i_assert(trie->locked_file_size == 0);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1481 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1482
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1483 /* uidlist locks building */
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1484 if (squat_uidlist_build_init(trie->uidlist, &uidlist_build_ctx) < 0)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1485 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1486
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1487 if (squat_trie_map(trie, TRUE) < 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1488 squat_uidlist_build_deinit(&uidlist_build_ctx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1489 return -1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1490 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1491
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1492 ctx = i_new(struct squat_trie_build_context, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1493 ctx->trie = trie;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1494 ctx->uidlist_build_ctx = uidlist_build_ctx;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1495 ctx->first_uid = trie->root.next_uid;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1496
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1497 *last_uid_r = I_MAX((trie->root.next_uid+1)/2, 1) - 1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1498 *ctx_r = ctx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1499 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1500 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1501
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1502 static int squat_trie_write_lock(struct squat_trie_build_context *ctx)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1503 {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1504 if (ctx->file_lock != NULL)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1505 return 0;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1506
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1507 if (squat_trie_lock(ctx->trie, F_WRLCK, &ctx->file_lock) <= 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1508 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1509 return 0;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1510 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1511
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1512 static int squat_trie_write(struct squat_trie_build_context *ctx)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1513 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1514 struct squat_trie *trie = ctx->trie;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1515 struct ostream *output;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1516 const char *path;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1517 int fd = -1, ret = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1518
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1519 if ((trie->hdr.used_file_size > sizeof(trie->hdr) &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1520 trie->unmapped_child_count < trie->hdr.node_count/4) \|\| 1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1521 /* we might as well recreate the file */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1522 ctx->compress_nodes = TRUE;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1523
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1524 path = t_strconcat(trie->path, ".tmp", NULL);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1525 fd = open(path, O_RDWR \| O_CREAT, 0600);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1526 if (fd == -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1527 i_error("creat(%s) failed: %m", path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1528 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1529 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1530 ret = file_wait_lock(fd, path, F_WRLCK, trie->lock_method,
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1531 SQUAT_TRIE_LOCK_TIMEOUT, &ctx->file_lock);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1532 if (ret <= 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1533 if (ret == 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1534 i_error("file_wait_lock(%s) failed: %m", path);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1535 (void)close(fd);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1536 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1537 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1538
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1539 output = o_stream_create_fd(fd, 0, FALSE);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1540 o_stream_cork(output);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1541 o_stream_send(output, &trie->hdr, sizeof(trie->hdr));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1542 } else {
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1543 /* we need to lock only while header is being written */
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1544 path = trie->path;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1545 ctx->compress_nodes =
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1546 trie->hdr.used_file_size == sizeof(trie->hdr);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1547 output = o_stream_create_fd(trie->fd, 0, FALSE);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1548 o_stream_cork(output);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1549
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1550 if (trie->hdr.used_file_size != 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1551 o_stream_seek(output, trie->hdr.used_file_size);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1552 else {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1553 if (squat_trie_write_lock(ctx) < 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1554 o_stream_unref(&output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1555 return -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1556 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1557 o_stream_send(output, &trie->hdr, sizeof(trie->hdr));
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1558 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1559 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1560
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1561 ctx->output = output;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1562 ret = squat_write_nodes(ctx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1563 ctx->output = NULL;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1564
6944 66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1565 /* write 1 byte guard at the end of file, so that we can verify broken
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1566 squat_unpack_num() input by checking if data==end */
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1567 o_stream_send(output, "", 1);
66b3e894041b Support mmap_disable=yes and some error handling improvements. Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	1568
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1569 if (trie->corrupted)
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1570 ret = -1;
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1571 if (ret == 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1572 ret = squat_trie_write_lock(ctx);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1573 if (ret == 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1574 trie->hdr.used_file_size = output->offset;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1575 o_stream_seek(output, 0);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1576 o_stream_send(output, &trie->hdr, sizeof(trie->hdr));
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1577 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1578 if (output->last_failed_errno != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1579 errno = output->last_failed_errno;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1580 i_error("write() to %s failed: %m", path);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1581 ret = -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1582 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1583 o_stream_destroy(&output);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1584
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1585 if (fd == -1) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1586 /* appended to the existing file */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1587 return ret;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1588 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1589
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1590 /* recreating the trie file */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1591 if (ret < 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1592 if (close(fd) < 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1593 i_error("close(%s) failed: %m", path);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1594 fd = -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1595 } else if (rename(path, trie->path) < 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1596 i_error("rename(%s, %s) failed: %m", path, trie->path);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1597 ret = -1;
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1598 }
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1599
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1600 if (ret < 0) {
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1601 if (unlink(path) < 0 && errno != ENOENT)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1602 i_error("unlink(%s) failed: %m", path);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1603 } else {
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1604 squat_trie_close_fd(trie);
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1605 trie->fd = fd;
7221 e042b6385c7b More squat fixes. Timo Sirainen <tss@iki.fi> parents: 7220 diff changeset	1606 trie->locked_file_size = trie->hdr.used_file_size;
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1607 if (trie->file_cache != NULL)
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1608 file_cache_set_fd(trie->file_cache, trie->fd);
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1609 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1610 return ret;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1611 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1612
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1613 int squat_trie_build_deinit(struct squat_trie_build_context **_ctx,
f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1614 const ARRAY_TYPE(seq_range) *expunged_uids)
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1615 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1616 struct squat_trie_build_context ctx = _ctx;
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1617 bool compress, unlock = TRUE;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1618 int ret;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1619
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1620 *_ctx = NULL;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1621
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1622 compress = (ctx->trie->root.next_uid - ctx->first_uid) > 10;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1623
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1624 /* keep trie locked while header is being written and when files are
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1625 being renamed, so that while trie is read locked, uidlist can't
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1626 change under. */
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1627 squat_uidlist_build_flush(ctx->uidlist_build_ctx);
7210 f5f77a3ae203 Initial code to support expunging from squat indexes, probably still buggy. Timo Sirainen <tss@iki.fi> parents: 7197 diff changeset	1628 ret = squat_trie_renumber_uidlists(ctx, expunged_uids, compress);
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1629 if (ret == 0) {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1630 ret = squat_trie_write(ctx);
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1631 if (ret < 0)
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1632 unlock = FALSE;
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1633 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1634
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1635 if (ret == 0)
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1636 ret = squat_uidlist_build_finish(ctx->uidlist_build_ctx);
7218 43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1637 if (ctx->file_lock != NULL) {
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1638 if (unlock)
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1639 file_unlock(&ctx->file_lock);
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1640 else
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1641 file_lock_free(&ctx->file_lock);
43d2f70fb279 Squat fixes Timo Sirainen <tss@iki.fi> parents: 7210 diff changeset	1642 }
6931 97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1643 squat_uidlist_build_deinit(&ctx->uidlist_build_ctx);
97702c9c4111 Locking and error handling fixes. Timo Sirainen <tss@iki.fi> parents: 6914 diff changeset	1644
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1645 i_free(ctx);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1646 return ret;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1647 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1648
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1649 int squat_trie_get_last_uid(struct squat_trie trie, uint32_t last_uid_r)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1650 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1651 if (trie->fd == -1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1652 if (squat_trie_open(trie) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1653 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1654 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1655
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1656 *last_uid_r = I_MAX((trie->root.next_uid+1)/2, 1) - 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1657 return 0;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1658 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1659
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1660 static int
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1661 squat_trie_lookup_data(struct squat_trie trie, const unsigned char data,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1662 unsigned int size, ARRAY_TYPE(seq_range) *uids)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1663 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1664 struct squat_node *node = &trie->root;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1665 unsigned char *chars;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1666 unsigned int idx;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1667 int level = 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1668
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1669 array_clear(uids);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1670
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1671 for (;;) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1672 if (node->children_not_mapped) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1673 if (node_read_children(trie, node, level) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1674 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1675 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1676 if (node->leaf_string_length != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1677 unsigned int str_len = node->leaf_string_length;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1678 const unsigned char *str;
5414 3ec03a4c0f26 fixes Timo Sirainen <tss@iki.fi> parents: 5336 diff changeset	1679
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1680 if (str_len > sizeof(node->children.static_leaf_string))
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1681 str = node->children.leaf_string;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1682 else
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1683 str = node->children.static_leaf_string;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1684
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1685 if (size > str_len \|\| memcmp(data, str, size) != 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1686 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1687
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1688 /* match */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1689 break;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1690 }
4898 07a038b57946 Optimization. Timo Sirainen <tss@iki.fi> parents: 4897 diff changeset	1691
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1692 if (size == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1693 break;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1694 level++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1695
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1696 if (node->have_sequential) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1697 if (*data < SEQUENTIAL_COUNT) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1698 idx = *data;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1699 goto found;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1700 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1701 idx = SEQUENTIAL_COUNT;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1702 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1703 idx = 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1704 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1705 chars = NODE_CHILDREN_CHARS(node);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1706 for (; idx < node->child_count; idx++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1707 if (chars[idx] == *data)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1708 goto found;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1709 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1710 return 0;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1711 found:
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1712 /* follow to children */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1713 if (level == 1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1714 /* root level, add all UIDs */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1715 if (squat_uidlist_get_seqrange(trie->uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1716 node->uid_list_idx,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1717 uids) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1718 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1719 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1720 if (squat_uidlist_filter(trie->uidlist,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1721 node->uid_list_idx, uids) < 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1722 return -1;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1723 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1724 data++;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1725 size--;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1726 node = NODE_CHILDREN_NODES(node) + idx;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1727 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1728
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1729 if (squat_uidlist_filter(trie->uidlist, node->uid_list_idx, uids) < 0)
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	1730 return -1;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1731 return 1;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1732 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1733
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	1734 static void
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1735 squat_trie_filter_type(enum squat_index_type type,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1736 const ARRAY_TYPE(seq_range) *src,
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1737 ARRAY_TYPE(seq_range) *dest)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1738 {
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1739 const struct seq_range *src_range;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1740 struct seq_range new_range;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1741 unsigned int i, count, mask;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1742 uint32_t next_seq, uid;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1743
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1744 array_clear(dest);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1745 src_range = array_get(src, &count);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1746 if (count == 0)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1747 return;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1748
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1749 if ((type & SQUAT_INDEX_TYPE_HEADER) != 0 &&
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1750 (type & SQUAT_INDEX_TYPE_BODY) != 0) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1751 /* everything is fine, just fix the UIDs */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1752 new_range.seq1 = src_range[0].seq1 / 2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1753 new_range.seq2 = src_range[0].seq2 / 2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1754 for (i = 1; i < count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1755 next_seq = src_range[i].seq1 / 2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1756 if (next_seq == new_range.seq2 + 1) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1757 /* we can continue the previous range */
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1758 } else {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1759 array_append(dest, &new_range, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1760 new_range.seq1 = src_range[i].seq1 / 2;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1761 }
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1762 new_range.seq2 = src_range[i].seq2 / 2;
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	1763 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1764 array_append(dest, &new_range, 1);
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1765 return;
4886 ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	1766 }
ae2114f11a0e Memory leak fixes. Also when building a large mailbox flush once in a while Timo Sirainen <tss@iki.fi> parents: 4883 diff changeset	1767
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1768 /* we'll have to drop either header or body UIDs */
6914 9c3f0e180751 Index message's all headers (including MIME headers) after the body. Timo Sirainen <tss@iki.fi> parents: 6912 diff changeset	1769 mask = (type & SQUAT_INDEX_TYPE_HEADER) != 0 ? 1 : 0;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1770 for (i = 0; i < count; i++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1771 for (uid = src_range[i].seq1; uid <= src_range[i].seq2; uid++) {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1772 if ((uid & 1) == mask)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1773 seq_range_array_add(dest, 0, uid/2);
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1774 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1775 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1776 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1777
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1778 struct squat_trie_lookup_context {
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1779 struct squat_trie *trie;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1780 enum squat_index_type type;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1781
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1782 ARRAY_TYPE(seq_range) definite_uids, maybe_uids;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1783 ARRAY_TYPE(seq_range) tmp_uids, tmp_uids2;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1784 bool first;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1785 };
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1786
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1787 static int
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1788 squat_trie_lookup_partial(struct squat_trie_lookup_context *ctx,
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1789 const unsigned char data, uint8_t char_lengths,
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1790 unsigned int size)
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1791 {
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1792 const unsigned int partial_len = ctx->trie->hdr.partial_len;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1793 unsigned int char_idx, max_chars, i, j, bytelen;
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1794 int ret;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1795
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1796 max_chars = uni_utf8_strlen_n(data, size);
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1797 if (max_chars > ctx->trie->hdr.partial_len)
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1798 max_chars = partial_len;
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1799
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1800 for (i = 0, char_idx = 0; char_idx < max_chars; char_idx++) {
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1801 bytelen = 0;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1802 for (j = 0; j < partial_len && i+bytelen < size; j++)
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1803 bytelen += char_lengths[i + bytelen];
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1804
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1805 ret = squat_trie_lookup_data(ctx->trie, data + i, bytelen,
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1806 &ctx->tmp_uids);
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1807 if (ret <= 0) {
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1808 array_clear(ctx->maybe_uids);
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1809 return ret;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1810 }
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1811
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1812 if (ctx->first) {
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1813 squat_trie_filter_type(ctx->type, &ctx->tmp_uids,
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1814 ctx->maybe_uids);
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1815 ctx->first = FALSE;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1816 } else {
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1817 squat_trie_filter_type(ctx->type, &ctx->tmp_uids,
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1818 &ctx->tmp_uids2);
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1819 seq_range_array_remove_invert_range(ctx->maybe_uids,
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1820 &ctx->tmp_uids2);
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1821 }
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1822
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1823 i += char_lengths[i];
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1824 }
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1825 return 1;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1826 }
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1827
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1828 static void squat_trie_add_unknown(struct squat_trie *trie,
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1829 ARRAY_TYPE(seq_range) *maybe_uids)
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1830 {
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1831 struct seq_range *range, new_range;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1832 unsigned int count;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1833 uint32_t last_uid;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1834
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1835 last_uid = I_MAX((trie->root.next_uid+1)/2, 1) - 1;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1836
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1837 range = array_get_modifiable(maybe_uids, &count);
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1838 if (count > 0 && range[count-1].seq2 == last_uid) {
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1839 /* increase the range */
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1840 range[count-1].seq2 = (uint32_t)-1;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1841 } else {
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1842 new_range.seq1 = last_uid + 1;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1843 new_range.seq2 = (uint32_t)-1;
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1844 array_append(maybe_uids, &new_range, 1);
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1845 }
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1846 }
8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1847
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1848 static int
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1849 squat_trie_lookup_real(struct squat_trie trie, const char str,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1850 enum squat_index_type type,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1851 ARRAY_TYPE(seq_range) *definite_uids,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1852 ARRAY_TYPE(seq_range) *maybe_uids)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1853 {
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1854 struct squat_trie_lookup_context ctx;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1855 unsigned char *data;
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1856 uint8_t *char_lengths;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1857 unsigned int i, start, bytes, str_bytelen, str_charlen;
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1858 int ret = 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1859
7161 b2333f0f23af squat_trie_lookup(): Make sure we clear result arrays first. Timo Sirainen <tss@iki.fi> parents: 7159 diff changeset	1860 array_clear(definite_uids);
b2333f0f23af squat_trie_lookup(): Make sure we clear result arrays first. Timo Sirainen <tss@iki.fi> parents: 7159 diff changeset	1861 array_clear(maybe_uids);
b2333f0f23af squat_trie_lookup(): Make sure we clear result arrays first. Timo Sirainen <tss@iki.fi> parents: 7159 diff changeset	1862
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1863 memset(&ctx, 0, sizeof(ctx));
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1864 ctx.trie = trie;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1865 ctx.type = type;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1866 ctx.definite_uids = definite_uids;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1867 ctx.maybe_uids = maybe_uids;
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1868 i_array_init(&ctx.tmp_uids, 128);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1869 i_array_init(&ctx.tmp_uids2, 128);
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1870 ctx.first = TRUE;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1871
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1872 str_bytelen = strlen(str);
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1873 char_lengths = t_malloc0(str_bytelen);
6912 b326327b1f10 Fix for searching <4 char keys. Timo Sirainen <tss@iki.fi> parents: 6911 diff changeset	1874 for (i = 0, str_charlen = 0; i < str_bytelen; str_charlen++) {
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1875 bytes = uni_utf8_char_bytes(str[i]);
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1876 char_lengths[i] = bytes;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1877 i += bytes;
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1878 }
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1879 data = squat_data_normalize(trie, (const unsigned char *)str,
a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1880 str_bytelen);
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1881
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1882 for (i = start = 0; i < str_bytelen && ret >= 0; i += char_lengths[i]) {
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1883 if (data[i] != '\0')
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1884 continue;
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1885
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1886 /* string has nonindexed characters.
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1887 search it in parts. */
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1888 if (i != start) {
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1889 ret = squat_trie_lookup_partial(&ctx, data + start,
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1890 char_lengths,
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1891 i - start);
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1892 }
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1893 start = i + char_lengths[i];
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1894 }
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1895
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1896 if (start != 0) {
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1897 /* string has nonindexed characters. finish the search. */
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1898 array_clear(definite_uids);
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1899 if (i != start && ret >= 0) {
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1900 ret = squat_trie_lookup_partial(&ctx, data + start,
6911 a215deb3de8d Fixed non-ASCII searches. Timo Sirainen <tss@iki.fi> parents: 6905 diff changeset	1901 char_lengths,
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1902 i - start);
cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1903 }
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1904 } else {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1905 if (str_charlen <= trie->hdr.partial_len \|\|
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1906 trie->hdr.full_len > trie->hdr.partial_len) {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1907 ret = squat_trie_lookup_data(trie, data, str_bytelen,
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1908 &ctx.tmp_uids);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1909 if (ret > 0) {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1910 squat_trie_filter_type(type, &ctx.tmp_uids,
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1911 definite_uids);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1912 }
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1913 } else {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1914 array_clear(definite_uids);
6902 cff5428c3c4d Fixed handling non-indexed characters in search key. Timo Sirainen <tss@iki.fi> parents: 6898 diff changeset	1915 }
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	1916
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1917 if (str_charlen <= trie->hdr.partial_len \|\|
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1918 trie->hdr.partial_len == 0) {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1919 /* we have the result */
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1920 array_clear(maybe_uids);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1921 } else {
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1922 ret = squat_trie_lookup_partial(&ctx, data + start,
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1923 char_lengths,
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1924 i - start);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1925 }
4878 88a91d9a867b Fixes. Should be pretty much working now. Timo Sirainen <tss@iki.fi> parents: 4855 diff changeset	1926 }
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1927 squat_trie_add_unknown(trie, maybe_uids);
6945 5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1928 array_free(&ctx.tmp_uids);
5915aea5f070 UID ranges can be large, don't allocate them from data stack. Timo Sirainen <tss@iki.fi> parents: 6944 diff changeset	1929 array_free(&ctx.tmp_uids2);
6905 8c779f3e16be Fixes to FTS handling. Timo Sirainen <tss@iki.fi> parents: 6902 diff changeset	1930 return ret < 0 ? -1 : 0;
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1931 }
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1932
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1933 int squat_trie_lookup(struct squat_trie trie, const char str,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1934 enum squat_index_type type,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1935 ARRAY_TYPE(seq_range) *definite_uids,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1936 ARRAY_TYPE(seq_range) *maybe_uids)
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1937 {
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1938 int ret;
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1939
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	1940 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1941 ret = squat_trie_lookup_real(trie, str, type,
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1942 definite_uids, maybe_uids);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7224 diff changeset	1943 } T_END;
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1944 return ret;
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1945 }
414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6931 diff changeset	1946
6418 46d9ee79f292 Removed _ prefix from all public APIs. Timo Sirainen <tss@iki.fi> parents: 6414 diff changeset	1947 struct squat_uidlist squat_trie_get_uidlist(struct squat_trie trie)
4855 5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1948 {
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1949 return trie->uidlist;
5bc593f1a8f6 Added "squat" full text search indexer backend. Its name and basic ideas Timo Sirainen <tss@iki.fi> parents: diff changeset	1950 }
6898 e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1951
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1952 size_t squat_trie_mem_used(struct squat_trie trie, unsigned int count_r)
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1953 {
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1954 *count_r = trie->hdr.node_count;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1955 return trie->node_alloc_size;
e739cffd05ef FTS API changes and Squat rewrite. Squat is still missing expunge handling. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	1956 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/plugins/fts-squat/squat-trie.c @ 7226:e6693a0ec8e1 HEAD