Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-index/mail-cache-compress.c @ 2407:fed6d07bd8ee HEAD

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Transaction log file is now read-lockless.
author Timo Sirainen <tss@iki.fi>
date Sat, 31 Jul 2004 06:06:48 +0300
parents 406692edc12d
children 87af1f39a69e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
1 /* Copyright (C) 2003-2004 Timo Sirainen */
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
2
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
3 #include "lib.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
4 #include "buffer.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
5 #include "ostream.h"
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
6 #include "file-set-size.h"
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
7 #include "mail-cache-private.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
8
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
9 static unsigned char null4[4] = { 0, 0, 0, 0 };
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
10
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
11 struct mail_cache_copy_context {
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
12 int new_msg;
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
13 buffer_t *buffer, *field_seen;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
14 uint8_t field_seen_value;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
15 };
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
16
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
17 static int
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
18 mail_cache_compress_callback(struct mail_cache_view *view, uint32_t file_field,
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
19 const void *data, size_t data_size, void *context)
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
20 {
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
21 struct mail_cache_copy_context *ctx = context;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
22 enum mail_cache_decision_type dec;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
23 unsigned int field;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
24 uint8_t *field_seen;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
25 uint32_t size32;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
26
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
27 field_seen = buffer_get_space_unsafe(ctx->field_seen, file_field, 1);
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
28 if (*field_seen == ctx->field_seen_value) {
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
29 /* duplicate */
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
30 return 1;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
31 }
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
32 *field_seen = ctx->field_seen_value;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
33
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
34 field = view->cache->file_field_map[file_field];
2339
406692edc12d Cache fixes. Decisions are saved again.
Timo Sirainen <tss@iki.fi>
parents: 2327
diff changeset
35 dec = view->cache->fields[field].field.decision &
406692edc12d Cache fixes. Decisions are saved again.
Timo Sirainen <tss@iki.fi>
parents: 2327
diff changeset
36 ~MAIL_CACHE_DECISION_FORCED;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
37 if (ctx->new_msg) {
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
38 if (dec == MAIL_CACHE_DECISION_NO)
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
39 return 1;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
40 } else {
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
41 if (dec != MAIL_CACHE_DECISION_YES)
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
42 return 1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
43 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
44
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
45 buffer_append(ctx->buffer, &file_field, sizeof(file_field));
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
46
2339
406692edc12d Cache fixes. Decisions are saved again.
Timo Sirainen <tss@iki.fi>
parents: 2327
diff changeset
47 if (view->cache->fields[field].field.field_size == (unsigned int)-1) {
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
48 size32 = (uint32_t)data_size;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
49 buffer_append(ctx->buffer, &size32, sizeof(size32));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
50 }
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
51
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
52 buffer_append(ctx->buffer, data, data_size);
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
53 if ((data_size & 3) != 0)
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
54 buffer_append(ctx->buffer, null4, 4 - (data_size & 3));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
55 return 1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
56 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
57
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
58 static int
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
59 mail_cache_copy(struct mail_cache *cache, struct mail_index_view *view, int fd)
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
60 {
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
61 struct mail_cache_copy_context ctx;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
62 struct mail_cache_view *cache_view;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
63 struct mail_index_transaction *t;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
64 const struct mail_index_header *idx_hdr;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
65 struct mail_cache_header hdr;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
66 struct mail_cache_record cache_rec;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
67 struct ostream *output;
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
68 buffer_t *buffer;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
69 uint32_t message_count, seq, first_new_seq, old_offset;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
70 uoff_t offset;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
71
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
72 /* get sequence of first message which doesn't need it's temp fields
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
73 removed. */
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
74 if (mail_index_get_header(view, &idx_hdr) < 0)
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
75 return -1;
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
76 if (idx_hdr->day_first_uid[7] == 0) {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
77 first_new_seq = 1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
78 message_count = mail_index_view_get_message_count(view);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
79 } else {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
80 if (mail_index_lookup_uid_range(view, idx_hdr->day_first_uid[7],
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
81 (uint32_t)-1, &first_new_seq,
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
82 &message_count) < 0)
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
83 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
84 if (first_new_seq == 0)
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
85 first_new_seq = message_count+1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
86 }
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
87
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
88 cache_view = mail_cache_view_open(cache, view);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
89 t = mail_index_transaction_begin(view, FALSE);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
90 output = o_stream_create_file(fd, default_pool, 0, FALSE);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
91
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
92 memset(&hdr, 0, sizeof(hdr));
2276
5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code -
Timo Sirainen <tss@iki.fi>
parents: 2275
diff changeset
93 hdr.version = MAIL_CACHE_VERSION;
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
94 hdr.indexid = idx_hdr->indexid;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
95 hdr.file_seq = idx_hdr->cache_file_seq + 1;
2339
406692edc12d Cache fixes. Decisions are saved again.
Timo Sirainen <tss@iki.fi>
parents: 2327
diff changeset
96 o_stream_send(output, &hdr, sizeof(hdr));
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
97
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
98 if (cache->fields_count != 0) {
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
99 hdr.field_header_offset =
2407
fed6d07bd8ee Transaction log file is now read-lockless.
Timo Sirainen <tss@iki.fi>
parents: 2339
diff changeset
100 mail_index_uint32_to_offset(output->offset);
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
101
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
102 t_push();
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
103 buffer = buffer_create_dynamic(pool_datastack_create(),
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
104 256, (size_t)-1);
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
105 mail_cache_header_fields_get(cache, buffer);
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
106 o_stream_send(output, buffer_get_data(buffer, NULL),
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
107 buffer_get_used_size(buffer));
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
108 t_pop();
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
109 }
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
110
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
111 memset(&ctx, 0, sizeof(ctx));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
112 ctx.buffer = buffer_create_dynamic(default_pool, 4096, (size_t)-1);
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
113 ctx.field_seen = buffer_create_dynamic(default_pool, 64, (size_t)-1);
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
114 ctx.field_seen_value = 0;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
115
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
116 mail_index_reset_cache(t, hdr.file_seq);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
117
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
118 for (seq = 1; seq <= message_count; seq++) {
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
119 ctx.new_msg = seq >= first_new_seq;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
120 buffer_set_used_size(ctx.buffer, 0);
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
121
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
122 if (++ctx.field_seen_value == 0) {
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
123 memset(buffer_get_modifyable_data(ctx.field_seen, NULL),
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
124 0, buffer_get_size(ctx.field_seen));
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
125 ctx.field_seen_value++;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
126 }
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
127
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
128 memset(&cache_rec, 0, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
129 buffer_append(ctx.buffer, &cache_rec, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
130
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
131 (void)mail_cache_foreach(cache_view, seq,
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
132 mail_cache_compress_callback, &ctx);
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
133
2299
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
134 cache_rec.size = buffer_get_used_size(ctx.buffer);
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
135 if (cache_rec.size == sizeof(cache_rec))
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
136 continue;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
137
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
138 mail_index_update_cache(t, seq, hdr.file_seq,
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
139 output->offset, &old_offset);
2299
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
140
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
141 buffer_write(ctx.buffer, 0, &cache_rec, sizeof(cache_rec));
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
142 o_stream_send(output, buffer_get_data(ctx.buffer, NULL),
2299
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
143 cache_rec.size);
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
144 }
2276
5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code -
Timo Sirainen <tss@iki.fi>
parents: 2275
diff changeset
145 hdr.used_file_size = output->offset;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
146 buffer_free(ctx.buffer);
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
147
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
148 o_stream_seek(output, 0);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
149 o_stream_send(output, &hdr, sizeof(hdr));
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
150
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
151 mail_cache_view_close(cache_view);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
152
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
153 if (o_stream_flush(output) < 0) {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
154 errno = output->stream_errno;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
155 mail_cache_set_syscall_error(cache, "o_stream_flush()");
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
156 (void)mail_index_transaction_rollback(t);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
157 o_stream_unref(output);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
158 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
159 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
160
2299
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
161 if (hdr.used_file_size < MAIL_CACHE_INITIAL_SIZE) {
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
162 /* grow the file some more. doesn't matter if it fails */
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
163 (void)file_set_size(fd, MAIL_CACHE_INITIAL_SIZE);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
164 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
165
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
166 o_stream_unref(output);
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
167
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
168 if (fdatasync(fd) < 0) {
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
169 mail_cache_set_syscall_error(cache, "fdatasync()");
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
170 (void)mail_index_transaction_rollback(t);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
171 return -1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
172 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
173
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
174 return mail_index_transaction_commit(t, &seq, &offset);
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
175 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
176
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
177 int mail_cache_compress(struct mail_cache *cache, struct mail_index_view *view)
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
178 {
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
179 int fd, ret, locked;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
180
2277
41e56f28d085 Cache updating is done now by first reserving space where to write, and then
Timo Sirainen <tss@iki.fi>
parents: 2276
diff changeset
181 if ((ret = mail_cache_lock(cache)) < 0)
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
182 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
183 locked = ret > 0;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
184
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
185 /* get the latest info on fields */
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
186 if (mail_cache_header_fields_read(cache) < 0) {
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
187 if (locked) mail_cache_unlock(cache);
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
188 return -1;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
189 }
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
190
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
191 #ifdef DEBUG
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
192 i_warning("Compressing cache file %s", cache->filepath);
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
193 #endif
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
194
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
195 fd = file_dotlock_open(cache->filepath, NULL, NULL,
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
196 MAIL_CACHE_LOCK_TIMEOUT,
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
197 MAIL_CACHE_LOCK_CHANGE_TIMEOUT,
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
198 MAIL_CACHE_LOCK_IMMEDIATE_TIMEOUT, NULL, NULL);
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
199 if (fd == -1) {
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
200 mail_cache_set_syscall_error(cache, "file_dotlock_open()");
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
201 if (locked) mail_cache_unlock(cache);
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
202 return -1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
203 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
204
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
205 // FIXME: check that cache file wasn't just recreated
2276
5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code -
Timo Sirainen <tss@iki.fi>
parents: 2275
diff changeset
206
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
207 ret = 0;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
208 if (mail_cache_copy(cache, view, fd) < 0) {
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
209 (void)file_dotlock_delete(cache->filepath, NULL, fd);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
210 ret = -1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
211 } else {
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
212 if (file_dotlock_replace(cache->filepath, NULL,
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
213 -1, FALSE) < 0) {
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
214 mail_cache_set_syscall_error(cache,
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
215 "file_dotlock_replace()");
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
216 (void)close(fd);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
217 ret = -1;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
218 } else {
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
219 mail_cache_file_close(cache);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
220 cache->fd = fd;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
221
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
222 if (mail_cache_map(cache, 0, 0) < 0)
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
223 ret = -1;
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
224 else if (mail_cache_header_fields_read(cache) < 0)
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
225 ret = -1;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
226 }
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
227 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
228
2276
5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code -
Timo Sirainen <tss@iki.fi>
parents: 2275
diff changeset
229 if (locked)
5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code -
Timo Sirainen <tss@iki.fi>
parents: 2275
diff changeset
230 mail_cache_unlock(cache);
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
231
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
232 if (ret == 0)
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
233 cache->need_compress = FALSE;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
234 return ret;
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
235 }
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
236
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
237 int mail_cache_need_compress(struct mail_cache *cache)
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
238 {
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
239 return cache->need_compress;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
240 }