Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-index/mail-cache-compress.c @ 4451:1a35d53c18fc HEAD

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Array API redesigned to work using unions. It now provides type safety without having to enable DEBUG, as long as the compiler supports typeof(). Its API changed a bit. It now allows directly accessing the array contents, although that's not necessarily recommended. Changed existing array usage to be type safe in a bit more places. Removed array_t completely. Also did s/modifyable/modifiable/.
author Timo Sirainen <tss@iki.fi>
date Wed, 28 Jun 2006 16:10:25 +0300
parents 9c3f19110538
children a2a9b6a92831
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
1 /* Copyright (C) 2003-2004 Timo Sirainen */
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
2
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
3 #include "lib.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
4 #include "buffer.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
5 #include "ostream.h"
4247
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
6 #include "nfs-workarounds.h"
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
7 #include "read-full.h"
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
8 #include "close-keep-errno.h"
3106
9c4aa309dbac Changed dotlocking API.
Timo Sirainen <tss@iki.fi>
parents: 2950
diff changeset
9 #include "file-dotlock.h"
2866
bf1e718e7370 Cache file works now with mmap_disable=yes. Still needs a few optimizations.
Timo Sirainen <tss@iki.fi>
parents: 2853
diff changeset
10 #include "file-cache.h"
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
11 #include "file-set-size.h"
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
12 #include "mail-cache-private.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
13
2879
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
14 #include <sys/stat.h>
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
15
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
16 struct mail_cache_copy_context {
3863
55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool.
Timo Sirainen <tss@iki.fi>
parents: 3819
diff changeset
17 bool new_msg;
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
18 buffer_t *buffer, *field_seen;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
19 uint8_t field_seen_value;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
20 };
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
21
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
22 static void mail_cache_merge_bitmask(struct mail_cache *cache, buffer_t *buffer,
2619
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
23 uint32_t field, const void *data,
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
24 size_t data_size)
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
25 {
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
26 void *buf_data;
2620
Timo Sirainen <tss@iki.fi>
parents: 2619
diff changeset
27 uint32_t buf_field;
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
28 unsigned int i, buf_data_size;
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
29 size_t pos, buf_size;
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
30
4451
1a35d53c18fc Array API redesigned to work using unions. It now provides type safety
Timo Sirainen <tss@iki.fi>
parents: 4247
diff changeset
31 buf_data = buffer_get_modifiable_data(buffer, &buf_size);
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
32 for (pos = sizeof(struct mail_cache_record); pos < buf_size; ) {
2620
Timo Sirainen <tss@iki.fi>
parents: 2619
diff changeset
33 buf_field = *((uint32_t *)PTR_OFFSET(buf_data, pos));
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
34 pos += sizeof(uint32_t);
2700
5b07b6fd9bf8 Added asserts
Timo Sirainen <tss@iki.fi>
parents: 2671
diff changeset
35 i_assert(buf_field < cache->fields_count);
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
36
2623
0157839e3c78 bitmask merging code was corrupting the cache file
Timo Sirainen <tss@iki.fi>
parents: 2620
diff changeset
37 buf_data_size = cache->fields[buf_field].field.field_size;
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
38 if (buf_data_size == (unsigned int)-1) {
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
39 buf_data_size =
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
40 *((uint32_t *)PTR_OFFSET(buf_data, pos));
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
41 pos += sizeof(uint32_t);
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
42 }
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
43
2620
Timo Sirainen <tss@iki.fi>
parents: 2619
diff changeset
44 if (buf_field == field) {
2700
5b07b6fd9bf8 Added asserts
Timo Sirainen <tss@iki.fi>
parents: 2671
diff changeset
45 /* @UNSAFE: found it, do the merging */
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
46 unsigned char *dest = PTR_OFFSET(buf_data, pos);
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
47
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
48 i_assert(buf_data_size == data_size);
2700
5b07b6fd9bf8 Added asserts
Timo Sirainen <tss@iki.fi>
parents: 2671
diff changeset
49 i_assert(pos + buf_data_size <= buf_size);
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
50 for (i = 0; i < buf_data_size; i++)
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
51 dest[i] |= ((const unsigned char*)data)[i];
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
52 break;
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
53 }
2950
30110ac226a7 mail_cache_merge_bitmask() was broken, cache compression could have crashed
Timo Sirainen <tss@iki.fi>
parents: 2949
diff changeset
54 pos += (buf_data_size + 3) & ~3;
2700
5b07b6fd9bf8 Added asserts
Timo Sirainen <tss@iki.fi>
parents: 2671
diff changeset
55 i_assert(pos <= buf_size);
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
56 }
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
57 }
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
58
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
59 static int
2619
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
60 mail_cache_compress_callback(struct mail_cache_view *view, uint32_t field,
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
61 const void *data, size_t data_size, void *context)
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
62 {
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
63 struct mail_cache_copy_context *ctx = context;
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
64 struct mail_cache_field *cache_field;
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
65 enum mail_cache_decision_type dec;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
66 uint8_t *field_seen;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
67 uint32_t size32;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
68
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
69 cache_field = &view->cache->fields[field].field;
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
70
2619
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
71 field_seen = buffer_get_space_unsafe(ctx->field_seen, field, 1);
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
72 if (*field_seen == ctx->field_seen_value) {
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
73 /* duplicate */
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
74 if (cache_field->type == MAIL_CACHE_FIELD_BITMASK) {
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
75 mail_cache_merge_bitmask(view->cache, ctx->buffer,
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
76 field, data, data_size);
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
77 }
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
78 return 1;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
79 }
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
80 *field_seen = ctx->field_seen_value;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
81
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
82 dec = cache_field->decision & ~MAIL_CACHE_DECISION_FORCED;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
83 if (ctx->new_msg) {
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
84 if (dec == MAIL_CACHE_DECISION_NO)
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
85 return 1;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
86 } else {
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
87 if (dec != MAIL_CACHE_DECISION_YES)
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
88 return 1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
89 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
90
2619
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
91 buffer_append(ctx->buffer, &field, sizeof(field));
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
92
2555
87af1f39a69e Added support for bitmask type.
Timo Sirainen <tss@iki.fi>
parents: 2407
diff changeset
93 if (cache_field->field_size == (unsigned int)-1) {
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
94 size32 = (uint32_t)data_size;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
95 buffer_append(ctx->buffer, &size32, sizeof(size32));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
96 }
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
97
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
98 buffer_append(ctx->buffer, data, data_size);
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
99 if ((data_size & 3) != 0)
3617
904849549eac Removed null4 and replaced the few uses of it with buffer_append_zero().
Timo Sirainen <tss@iki.fi>
parents: 3551
diff changeset
100 buffer_append_zero(ctx->buffer, 4 - (data_size & 3));
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
101 return 1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
102 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
103
2853
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
104 static uint32_t
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
105 get_next_file_seq(struct mail_cache *cache, struct mail_index_view *view)
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
106 {
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
107 const struct mail_index_ext *ext;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
108 uint32_t file_seq;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
109
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
110 ext = mail_index_view_get_ext(view, cache->ext_id);
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
111 file_seq = ext != NULL ? ext->reset_id + 1 : (uint32_t)ioloop_time;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
112 return file_seq != 0 ? file_seq : 1;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
113 }
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
114
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
115 static int
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
116 mail_cache_copy(struct mail_cache *cache, struct mail_index_view *view, int fd)
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
117 {
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
118 struct mail_cache_copy_context ctx;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
119 struct mail_cache_view *cache_view;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
120 struct mail_index_transaction *t;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
121 const struct mail_index_header *idx_hdr;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
122 struct mail_cache_header hdr;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
123 struct mail_cache_record cache_rec;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
124 struct ostream *output;
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
125 buffer_t *buffer;
2619
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
126 uint32_t i, message_count, seq, first_new_seq, old_offset;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
127 uoff_t offset;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
128
3551
65dc45982df9 Removed fcntl() text from some locking error messages, because if
Timo Sirainen <tss@iki.fi>
parents: 3243
diff changeset
129 /* get sequence of first message which doesn't need its temp fields
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
130 removed. */
2892
62d53b49110d Changed mail_index_get_header() to return the header as return value because
Timo Sirainen <tss@iki.fi>
parents: 2879
diff changeset
131 idx_hdr = mail_index_get_header(view);
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
132 if (idx_hdr->day_first_uid[7] == 0) {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
133 first_new_seq = 1;
2904
642ba9cd3dac Removed view->messages_count, view->hdr.messages_count is enough. Also fixes
Timo Sirainen <tss@iki.fi>
parents: 2892
diff changeset
134 message_count = mail_index_view_get_messages_count(view);
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
135 } else {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
136 if (mail_index_lookup_uid_range(view, idx_hdr->day_first_uid[7],
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
137 (uint32_t)-1, &first_new_seq,
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
138 &message_count) < 0)
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
139 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
140 if (first_new_seq == 0)
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
141 first_new_seq = message_count+1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
142 }
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
143
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
144 cache_view = mail_cache_view_open(cache, view);
2877
3cb483d565a6 Allow specifying to transaction if it's external. Make mail saving
Timo Sirainen <tss@iki.fi>
parents: 2866
diff changeset
145 t = mail_index_transaction_begin(view, FALSE, TRUE);
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
146 output = o_stream_create_file(fd, default_pool, 0, FALSE);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
147
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
148 memset(&hdr, 0, sizeof(hdr));
2276
5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code -
Timo Sirainen <tss@iki.fi>
parents: 2275
diff changeset
149 hdr.version = MAIL_CACHE_VERSION;
3819
61f0e1fe9b55 Moved uoff_t and time_t compatibility checks to cache file, since they're
Timo Sirainen <tss@iki.fi>
parents: 3627
diff changeset
150 hdr.compat_sizeof_uoff_t = sizeof(uoff_t);
61f0e1fe9b55 Moved uoff_t and time_t compatibility checks to cache file, since they're
Timo Sirainen <tss@iki.fi>
parents: 3627
diff changeset
151 hdr.compat_sizeof_time_t = sizeof(time_t);
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
152 hdr.indexid = idx_hdr->indexid;
2853
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
153 hdr.file_seq = get_next_file_seq(cache, view);
2339
406692edc12d Cache fixes. Decisions are saved again.
Timo Sirainen <tss@iki.fi>
parents: 2327
diff changeset
154 o_stream_send(output, &hdr, sizeof(hdr));
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
155
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
156 memset(&ctx, 0, sizeof(ctx));
2708
f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where
Timo Sirainen <tss@iki.fi>
parents: 2700
diff changeset
157 ctx.buffer = buffer_create_dynamic(default_pool, 4096);
f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where
Timo Sirainen <tss@iki.fi>
parents: 2700
diff changeset
158 ctx.field_seen = buffer_create_dynamic(default_pool, 64);
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
159 ctx.field_seen_value = 0;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
160
2853
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
161 mail_index_ext_reset(t, cache->ext_id, hdr.file_seq);
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
162
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
163 for (seq = 1; seq <= message_count; seq++) {
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
164 ctx.new_msg = seq >= first_new_seq;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
165 buffer_set_used_size(ctx.buffer, 0);
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
166
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
167 if (++ctx.field_seen_value == 0) {
4451
1a35d53c18fc Array API redesigned to work using unions. It now provides type safety
Timo Sirainen <tss@iki.fi>
parents: 4247
diff changeset
168 memset(buffer_get_modifiable_data(ctx.field_seen, NULL),
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
169 0, buffer_get_size(ctx.field_seen));
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
170 ctx.field_seen_value++;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
171 }
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
172
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
173 memset(&cache_rec, 0, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
174 buffer_append(ctx.buffer, &cache_rec, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
175
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
176 (void)mail_cache_foreach(cache_view, seq,
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
177 mail_cache_compress_callback, &ctx);
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
178
2299
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
179 cache_rec.size = buffer_get_used_size(ctx.buffer);
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
180 if (cache_rec.size == sizeof(cache_rec))
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
181 continue;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
182
2853
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
183 mail_index_update_ext(t, seq, cache->ext_id, &output->offset,
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
184 &old_offset);
2299
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
185
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
186 buffer_write(ctx.buffer, 0, &cache_rec, sizeof(cache_rec));
2853
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension
Timo Sirainen <tss@iki.fi>
parents: 2708
diff changeset
187 o_stream_send(output, ctx.buffer->data, cache_rec.size);
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
188 }
2619
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
189
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
190 if (cache->fields_count != 0) {
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
191 hdr.field_header_offset =
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
192 mail_index_uint32_to_offset(output->offset);
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
193
2671
3b70ddb51771 Renamed "extra record info" and variations of it to "extension" or "ext" in
Timo Sirainen <tss@iki.fi>
parents: 2641
diff changeset
194 /* we wrote everything using our internal field ids. so we want
2619
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
195 mail_cache_header_fields_get() to use them and ignore any
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
196 existing id mappings in the old cache file. */
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
197 cache->file_fields_count = 0;
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
198 for (i = 0; i < cache->fields_count; i++)
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
199 cache->field_file_map[i] = (uint32_t)-1;
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
200
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
201 t_push();
2708
f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where
Timo Sirainen <tss@iki.fi>
parents: 2700
diff changeset
202 buffer = buffer_create_dynamic(pool_datastack_create(), 256);
2619
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
203 mail_cache_header_fields_get(cache, buffer);
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
204 o_stream_send(output, buffer_get_data(buffer, NULL),
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
205 buffer_get_used_size(buffer));
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
206 t_pop();
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
207 }
ce8647d94bfc ID mapping fixes.
Timo Sirainen <tss@iki.fi>
parents: 2604
diff changeset
208
2276
5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code -
Timo Sirainen <tss@iki.fi>
parents: 2275
diff changeset
209 hdr.used_file_size = output->offset;
2298
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but
Timo Sirainen <tss@iki.fi>
parents: 2282
diff changeset
210 buffer_free(ctx.buffer);
2641
e357ebb3137b Memory leak fixes
Timo Sirainen <tss@iki.fi>
parents: 2623
diff changeset
211 buffer_free(ctx.field_seen);
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
212
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
213 o_stream_seek(output, 0);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
214 o_stream_send(output, &hdr, sizeof(hdr));
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
215
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
216 mail_cache_view_close(cache_view);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
217
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
218 if (o_stream_flush(output) < 0) {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
219 errno = output->stream_errno;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
220 mail_cache_set_syscall_error(cache, "o_stream_flush()");
3879
928229f8b3e6 deinit, unref, destroy, close, free, etc. functions now take a pointer to
Timo Sirainen <tss@iki.fi>
parents: 3863
diff changeset
221 (void)mail_index_transaction_rollback(&t);
4070
71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of
Timo Sirainen <tss@iki.fi>
parents: 3879
diff changeset
222 o_stream_destroy(&output);
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
223 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
224 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
225
2299
ef0cd18d674c Fixed compression.
Timo Sirainen <tss@iki.fi>
parents: 2298
diff changeset
226 if (hdr.used_file_size < MAIL_CACHE_INITIAL_SIZE) {
2275
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
227 /* grow the file some more. doesn't matter if it fails */
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
228 (void)file_set_size(fd, MAIL_CACHE_INITIAL_SIZE);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
229 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened.
Timo Sirainen <tss@iki.fi>
parents: 2257
diff changeset
230
4070
71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of
Timo Sirainen <tss@iki.fi>
parents: 3879
diff changeset
231 o_stream_destroy(&output);
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
232
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
233 if (fdatasync(fd) < 0) {
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
234 mail_cache_set_syscall_error(cache, "fdatasync()");
3879
928229f8b3e6 deinit, unref, destroy, close, free, etc. functions now take a pointer to
Timo Sirainen <tss@iki.fi>
parents: 3863
diff changeset
235 (void)mail_index_transaction_rollback(&t);
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
236 return -1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
237 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
238
3879
928229f8b3e6 deinit, unref, destroy, close, free, etc. functions now take a pointer to
Timo Sirainen <tss@iki.fi>
parents: 3863
diff changeset
239 return mail_index_transaction_commit(&t, &seq, &offset);
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
240 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
241
4247
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
242 static int mail_cache_compress_has_file_changed(struct mail_cache *cache)
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
243 {
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
244 struct mail_cache_header hdr;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
245 unsigned int i;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
246 int fd, ret;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
247
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
248 for (i = 0;; i++) {
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
249 fd = nfs_safe_open(cache->filepath, O_RDONLY);
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
250 if (fd == -1) {
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
251 if (errno == ENOENT)
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
252 return 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
253
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
254 mail_cache_set_syscall_error(cache, "open()");
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
255 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
256 }
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
257
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
258 ret = read_full(fd, &hdr, sizeof(hdr));
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
259 close_keep_errno(fd);
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
260
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
261 if (ret >= 0) {
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
262 if (ret == 0)
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
263 return 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
264 if (cache->need_compress_file_seq == (uint32_t)-1) {
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
265 /* previously it didn't exist */
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
266 return 1;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
267 }
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
268 return hdr.file_seq != cache->need_compress_file_seq;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
269 } else if (errno != ESTALE || i >= NFS_ESTALE_RETRY_COUNT) {
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
270 mail_cache_set_syscall_error(cache, "read()");
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
271 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
272 }
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
273 }
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
274 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
275 }
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
276
2929
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
277 static int mail_cache_compress_locked(struct mail_cache *cache,
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
278 struct mail_index_view *view)
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
279 {
3106
9c4aa309dbac Changed dotlocking API.
Timo Sirainen <tss@iki.fi>
parents: 2950
diff changeset
280 struct dotlock *dotlock;
2879
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
281 mode_t old_mask;
4247
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
282 int fd, ret;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
283
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
284 /* get the latest info on fields */
2929
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
285 if (mail_cache_header_fields_read(cache) < 0)
2327
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
286 return -1;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so
Timo Sirainen <tss@iki.fi>
parents: 2299
diff changeset
287
2879
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
288 old_mask = umask(cache->index->mode ^ 0666);
3106
9c4aa309dbac Changed dotlocking API.
Timo Sirainen <tss@iki.fi>
parents: 2950
diff changeset
289 fd = file_dotlock_open(&cache->dotlock_settings, cache->filepath,
9c4aa309dbac Changed dotlocking API.
Timo Sirainen <tss@iki.fi>
parents: 2950
diff changeset
290 0, &dotlock);
2879
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
291 umask(old_mask);
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
292
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
293 if (fd == -1) {
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
294 mail_cache_set_syscall_error(cache, "file_dotlock_open()");
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
295 return -1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
296 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
297
4247
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
298 if ((ret = mail_cache_compress_has_file_changed(cache)) != 0) {
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
299 if (ret < 0)
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
300 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
301
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
302 /* was just compressed, forget this */
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
303 cache->need_compress_file_seq = 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
304 file_dotlock_delete(&dotlock);
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
305 return mail_cache_reopen(cache);
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
306 }
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
307
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
308 #ifdef DEBUG
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
309 i_warning("Compressing cache file %s (%u)",
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
310 cache->filepath, cache->need_compress_file_seq);
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
311 #endif
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
312
2879
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
313 if (cache->index->gid != (gid_t)-1 &&
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
314 fchown(fd, (uid_t)-1, cache->index->gid) < 0) {
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
315 mail_cache_set_syscall_error(cache, "fchown()");
3106
9c4aa309dbac Changed dotlocking API.
Timo Sirainen <tss@iki.fi>
parents: 2950
diff changeset
316 file_dotlock_delete(&dotlock);
2879
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
317 return -1;
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
318 }
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created
Timo Sirainen <tss@iki.fi>
parents: 2877
diff changeset
319
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
320 if (mail_cache_copy(cache, view, fd) < 0) {
3106
9c4aa309dbac Changed dotlocking API.
Timo Sirainen <tss@iki.fi>
parents: 2950
diff changeset
321 (void)file_dotlock_delete(&dotlock);
2929
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
322 return -1;
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
323 }
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
324
3106
9c4aa309dbac Changed dotlocking API.
Timo Sirainen <tss@iki.fi>
parents: 2950
diff changeset
325 if (file_dotlock_replace(&dotlock,
9c4aa309dbac Changed dotlocking API.
Timo Sirainen <tss@iki.fi>
parents: 2950
diff changeset
326 DOTLOCK_REPLACE_FLAG_DONT_CLOSE_FD) < 0) {
2929
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
327 mail_cache_set_syscall_error(cache,
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
328 "file_dotlock_replace()");
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
329 (void)close(fd);
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
330 return -1;
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
331 }
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
332
2929
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
333 mail_cache_file_close(cache);
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
334 cache->fd = fd;
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
335
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
336 if (cache->file_cache != NULL)
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
337 file_cache_set_fd(cache->file_cache, cache->fd);
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
338
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
339 if (mail_cache_map(cache, 0, 0) < 0)
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
340 return -1;
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
341 if (mail_cache_header_fields_read(cache) < 0)
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
342 return -1;
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
343
4247
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
344 cache->need_compress_file_seq = 0;
2929
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
345 return 0;
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
346 }
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
347
2929
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
348 int mail_cache_compress(struct mail_cache *cache, struct mail_index_view *view)
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
349 {
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
350 int ret;
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
351
3243
40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc().
Timo Sirainen <tss@iki.fi>
parents: 3106
diff changeset
352 if (MAIL_INDEX_IS_IN_MEMORY(cache->index))
40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc().
Timo Sirainen <tss@iki.fi>
parents: 3106
diff changeset
353 return 0;
40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc().
Timo Sirainen <tss@iki.fi>
parents: 3106
diff changeset
354
2949
05ae72649fd0 Cache compression didn't work with lock_method = dotlock.
Timo Sirainen <tss@iki.fi>
parents: 2929
diff changeset
355 if (cache->index->lock_method == MAIL_INDEX_LOCK_DOTLOCK) {
05ae72649fd0 Cache compression didn't work with lock_method = dotlock.
Timo Sirainen <tss@iki.fi>
parents: 2929
diff changeset
356 /* we're using dotlocking, cache file creation itself creates
05ae72649fd0 Cache compression didn't work with lock_method = dotlock.
Timo Sirainen <tss@iki.fi>
parents: 2929
diff changeset
357 the dotlock file we need. */
05ae72649fd0 Cache compression didn't work with lock_method = dotlock.
Timo Sirainen <tss@iki.fi>
parents: 2929
diff changeset
358 return mail_cache_compress_locked(cache, view);
05ae72649fd0 Cache compression didn't work with lock_method = dotlock.
Timo Sirainen <tss@iki.fi>
parents: 2929
diff changeset
359 }
05ae72649fd0 Cache compression didn't work with lock_method = dotlock.
Timo Sirainen <tss@iki.fi>
parents: 2929
diff changeset
360
2929
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
361 switch (mail_cache_lock(cache)) {
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
362 case -1:
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
363 return -1;
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
364 case 0:
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
365 /* couldn't lock, either it's broken or doesn't exist.
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
366 just start creating it. */
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
367 return mail_cache_compress_locked(cache, view);
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
368 default:
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
369 /* locking succeeded. */
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
370 ret = mail_cache_compress_locked(cache, view);
3627
c0e35566dd14 Unlocking cache file can also corrupt it since it modifies it. Added a
Timo Sirainen <tss@iki.fi>
parents: 3617
diff changeset
371 if (mail_cache_unlock(cache) < 0)
c0e35566dd14 Unlocking cache file can also corrupt it since it modifies it. Added a
Timo Sirainen <tss@iki.fi>
parents: 3617
diff changeset
372 ret = -1;
2929
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
373 return ret;
ba9062032877 Locking fixes and cleanups
Timo Sirainen <tss@iki.fi>
parents: 2904
diff changeset
374 }
1915
79790750c349 importing new index code. mbox still broken.
Timo Sirainen <tss@iki.fi>
parents:
diff changeset
375 }
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
376
3863
55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool.
Timo Sirainen <tss@iki.fi>
parents: 3819
diff changeset
377 bool mail_cache_need_compress(struct mail_cache *cache)
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
378 {
4247
9c3f19110538 Don't compress cache file if it was just compressed by another process.
Timo Sirainen <tss@iki.fi>
parents: 4070
diff changeset
379 return cache->need_compress_file_seq != 0;
2247
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested.
Timo Sirainen <tss@iki.fi>
parents: 2200
diff changeset
380 }