dovecot/core-2.2: src/lib-index/mail-cache-compress.c annotate

annotate src/lib-index/mail-cache-compress.c @ 22691:dca05b22217b

director: Fix crash when handling expired USER timestamps. The fix in 154f91726624265fce15097eb4bbbf6e55f8c477 wasn't complete.

author	Timo Sirainen <timo.sirainen@dovecot.fi>
date	Tue, 28 Nov 2017 13:10:35 +0200
parents	65ada9976a7f
children	cb108f786fb4

rev	line source
21390 2e2563132d5f Updated copyright notices to include the year 2017. Stephan Bosch <stephan.bosch@dovecot.fi> parents: 21389 diff changeset	1 /* Copyright (c) 2003-2017 Dovecot authors, see the included COPYING file */
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	2
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	3 #include "lib.h"
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	4 #include "array.h"
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	5 #include "ostream.h"
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	6 #include "nfs-workarounds.h"
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	7 #include "read-full.h"
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	8 #include "file-dotlock.h"
2866 bf1e718e7370 Cache file works now with mmap_disable=yes. Still needs a few optimizations. Timo Sirainen <tss@iki.fi> parents: 2853 diff changeset	9 #include "file-cache.h"
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	10 #include "file-set-size.h"
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	11 #include "mail-cache-private.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	12
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	13 #include <stdio.h>
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	14 #include <sys/stat.h>
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	15
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	16 struct mail_cache_copy_context {
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	17 struct mail_cache *cache;
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	18
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	19 buffer_t buffer, field_seen;
14920 a097ef0a9d6d Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Timo Sirainen <tss@iki.fi> parents: 14691 diff changeset	20 ARRAY(unsigned int) bitmask_pos;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	21 uint32_t *field_file_map;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	22
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	23 uint8_t field_seen_value;
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	24 bool new_msg;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	25 };
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	26
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	27 struct mail_cache_compress_lock {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	28 struct dotlock *dotlock;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	29 };
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	30
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	31 static void
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	32 mail_cache_merge_bitmask(struct mail_cache_copy_context *ctx,
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	33 const struct mail_cache_iterate_field *field)
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	34 {
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	35 unsigned char *dest;
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	36 unsigned int i, *pos;
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	37
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	38 pos = array_idx_modifiable(&ctx->bitmask_pos, field->field_idx);
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	39 if (*pos == 0) {
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	40 /* we decided to drop this field */
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	41 return;
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	42 }
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	43
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	44 dest = buffer_get_space_unsafe(ctx->buffer, *pos, field->size);
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	45 for (i = 0; i < field->size; i++)
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	46 dest[i] \|= ((const unsigned char*)field->data)[i];
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	47 }
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	48
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	49 static void
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	50 mail_cache_compress_field(struct mail_cache_copy_context *ctx,
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	51 const struct mail_cache_iterate_field *field)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	52 {
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	53 struct mail_cache_field *cache_field;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	54 enum mail_cache_decision_type dec;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	55 uint32_t file_field_idx, size32;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	56 uint8_t *field_seen;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	57
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	58 file_field_idx = ctx->field_file_map[field->field_idx];
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	59 if (file_field_idx == (uint32_t)-1)
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	60 return;
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	61
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	62 cache_field = &ctx->cache->fields[field->field_idx].field;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	63
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	64 field_seen = buffer_get_space_unsafe(ctx->field_seen,
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	65 field->field_idx, 1);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	66 if (*field_seen == ctx->field_seen_value) {
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	67 /* duplicate */
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	68 if (cache_field->type == MAIL_CACHE_FIELD_BITMASK)
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	69 mail_cache_merge_bitmask(ctx, field);
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	70 return;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	71 }
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	72 *field_seen = ctx->field_seen_value;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	73
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	74 dec = cache_field->decision & ~MAIL_CACHE_DECISION_FORCED;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	75 if (ctx->new_msg) {
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	76 if (dec == MAIL_CACHE_DECISION_NO)
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	77 return;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	78 } else {
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	79 if (dec != MAIL_CACHE_DECISION_YES)
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	80 return;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	81 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	82
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	83 buffer_append(ctx->buffer, &file_field_idx, sizeof(file_field_idx));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	84
15904 d3cf06639864 Replaced all -1U and (unsigned int)-1 with UINT_MAX. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	85 if (cache_field->field_size == UINT_MAX) {
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	86 size32 = (uint32_t)field->size;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	87 buffer_append(ctx->buffer, &size32, sizeof(size32));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	88 }
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	89
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	90 if (cache_field->type == MAIL_CACHE_FIELD_BITMASK) {
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	91 /* remember the position in case we need to update it */
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	92 unsigned int pos = ctx->buffer->used;
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	93
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	94 array_idx_set(&ctx->bitmask_pos, field->field_idx, &pos);
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	95 }
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	96 buffer_append(ctx->buffer, field->data, field->size);
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	97 if ((field->size & 3) != 0)
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	98 buffer_append_zero(ctx->buffer, 4 - (field->size & 3));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	99 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	100
17903 bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	101 static uint32_t get_next_file_seq(struct mail_cache *cache)
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	102 {
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	103 const struct mail_index_ext *ext;
17903 bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	104 struct mail_index_view *view;
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	105 uint32_t file_seq;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	106
17903 bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	107 /* make sure we look up the latest reset_id */
bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	108 if (mail_index_refresh(cache->index) < 0)
bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	109 return -1;
bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	110
bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	111 view = mail_index_view_open(cache->index);
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	112 ext = mail_index_view_get_ext(view, cache->ext_id);
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	113 file_seq = ext != NULL ? ext->reset_id + 1 : (uint32_t)ioloop_time;
5994 8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	114
8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	115 if (cache->hdr != NULL && file_seq <= cache->hdr->file_seq)
8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	116 file_seq = cache->hdr->file_seq + 1;
17903 bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	117 mail_index_view_close(&view);
5994 8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	118
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	119 return file_seq != 0 ? file_seq : 1;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	120 }
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	121
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	122 static void
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	123 mail_cache_compress_get_fields(struct mail_cache_copy_context *ctx,
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	124 unsigned int used_fields_count)
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	125 {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	126 struct mail_cache *cache = ctx->cache;
6297 3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	127 struct mail_cache_field *field;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	128 unsigned int i, j, idx;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	129
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	130 /* Make mail_cache_header_fields_get() return the fields in
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	131 the same order as we saved them. */
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	132 memcpy(cache->field_file_map, ctx->field_file_map,
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	133 sizeof(uint32_t) * cache->fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	134
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	135 /* reverse mapping */
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	136 cache->file_fields_count = used_fields_count;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	137 i_free(cache->file_field_map);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	138 cache->file_field_map = used_fields_count == 0 ? NULL :
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	139 i_new(unsigned int, used_fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	140 for (i = j = 0; i < cache->fields_count; i++) {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	141 idx = cache->field_file_map[i];
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	142 if (idx != (uint32_t)-1) {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	143 i_assert(idx < used_fields_count &&
11076 fa08ed2b7560 Small code changes to make static analyzer happier. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	144 cache->file_field_map != NULL &&
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	145 cache->file_field_map[idx] == 0);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	146 cache->file_field_map[idx] = i;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	147 j++;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	148 }
6297 3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	149
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	150 /* change permanent decisions to temporary decisions.
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	151 if they're still permanent they'll get updated later. */
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	152 field = &cache->fields[i].field;
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	153 if (field->decision == MAIL_CACHE_DECISION_YES)
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	154 field->decision = MAIL_CACHE_DECISION_TEMP;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	155 }
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	156 i_assert(j == used_fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	157
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	158 buffer_set_used_size(ctx->buffer, 0);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	159 mail_cache_header_fields_get(cache, ctx->buffer);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	160 }
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	161
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	162 static int
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	163 mail_cache_copy(struct mail_cache cache, struct mail_index_transaction trans,
20561 6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	164 int fd, uint32_t file_seq_r, uoff_t file_size_r, uint32_t *max_uid_r,
5733 4b5ce642957d Declare a global uint32_t array type. Timo Sirainen <tss@iki.fi> parents: 5652 diff changeset	165 ARRAY_TYPE(uint32_t) *ext_offsets)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	166 {
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	167 struct mail_cache_copy_context ctx;
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	168 struct mail_cache_lookup_iterate_ctx iter;
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	169 struct mail_cache_iterate_field field;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	170 struct mail_index_view *view;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	171 struct mail_cache_view *cache_view;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	172 const struct mail_index_header *idx_hdr;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	173 struct mail_cache_header hdr;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	174 struct mail_cache_record cache_rec;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	175 struct ostream *output;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	176 uint32_t message_count, seq, first_new_seq, ext_offset;
15374 debecd057f9c lib-index: Replaced cache's deleted_space with deleted_record_count. Timo Sirainen <tss@iki.fi> parents: 15373 diff changeset	177 unsigned int i, used_fields_count, orig_fields_count, record_count;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	178 time_t max_drop_time;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	179
20561 6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	180 *max_uid_r = 0;
6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	181
17903 bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	182 /* get the latest info on fields */
bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	183 if (mail_cache_header_fields_read(cache) < 0)
bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	184 return -1;
bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	185
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	186 view = mail_index_transaction_get_view(trans);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	187 cache_view = mail_cache_view_open(cache, view);
6161 c62f7ee79446 Split o_stream_create_file() to _create_fd() and _create_fd_file(). Timo Sirainen <tss@iki.fi> parents: 6142 diff changeset	188 output = o_stream_create_fd_file(fd, 0, FALSE);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	189
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20561 diff changeset	190 i_zero(&hdr);
15373 8323e81785e2 lib-index: Added minor version to dovecot.index.cache file header. Timo Sirainen <tss@iki.fi> parents: 15372 diff changeset	191 hdr.major_version = MAIL_CACHE_MAJOR_VERSION;
8323e81785e2 lib-index: Added minor version to dovecot.index.cache file header. Timo Sirainen <tss@iki.fi> parents: 15372 diff changeset	192 hdr.minor_version = MAIL_CACHE_MINOR_VERSION;
3819 61f0e1fe9b55 Moved uoff_t and time_t compatibility checks to cache file, since they're Timo Sirainen <tss@iki.fi> parents: 3627 diff changeset	193 hdr.compat_sizeof_uoff_t = sizeof(uoff_t);
5819 4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5737 diff changeset	194 hdr.indexid = cache->index->indexid;
17903 bd55bdba0e75 lib-index: Try to minimize race conditions while compressing cache. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	195 hdr.file_seq = get_next_file_seq(cache);
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14637 diff changeset	196 o_stream_nsend(output, &hdr, sizeof(hdr));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	197
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20561 diff changeset	198 i_zero(&ctx);
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	199 ctx.cache = cache;
2708 f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where Timo Sirainen <tss@iki.fi> parents: 2700 diff changeset	200 ctx.buffer = buffer_create_dynamic(default_pool, 4096);
f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where Timo Sirainen <tss@iki.fi> parents: 2700 diff changeset	201 ctx.field_seen = buffer_create_dynamic(default_pool, 64);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	202 ctx.field_seen_value = 0;
11076 fa08ed2b7560 Small code changes to make static analyzer happier. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	203 ctx.field_file_map = t_new(uint32_t, cache->fields_count + 1);
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	204 t_array_init(&ctx.bitmask_pos, 32);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	205
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	206 /* @UNSAFE: drop unused fields and create a field mapping for
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	207 used fields */
6841 0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	208 idx_hdr = mail_index_get_header(view);
6678 6601ee607b6f Don't drop all fields if day_stamp is still 0 in index header. Timo Sirainen <tss@iki.fi> parents: 6677 diff changeset	209 max_drop_time = idx_hdr->day_stamp == 0 ? 0 :
22611 65ada9976a7f lib-index: Add mail_index_cache_optimization_settings Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21390 diff changeset	210 idx_hdr->day_stamp -
65ada9976a7f lib-index: Add mail_index_cache_optimization_settings Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21390 diff changeset	211 cache->index->optimization_set.cache.unaccessed_field_drop_secs;
6717 02014f5b0068 Don't write fields to cache file with last_used=0. If fields were just Timo Sirainen <tss@iki.fi> parents: 6696 diff changeset	212
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	213 orig_fields_count = cache->fields_count;
6696 68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	214 if (cache->file_fields_count == 0) {
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	215 /* creating the initial cache file. add all fields. */
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	216 for (i = 0; i < orig_fields_count; i++)
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	217 ctx.field_file_map[i] = i;
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	218 used_fields_count = i;
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	219 } else {
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	220 for (i = used_fields_count = 0; i < orig_fields_count; i++) {
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	221 struct mail_cache_field_private *priv =
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	222 &cache->fields[i];
7068 180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	223 enum mail_cache_decision_type dec =
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	224 priv->field.decision;
7068 180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	225
180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	226 /* if the decision isn't forced and this field hasn't
180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	227 been accessed for a while, drop it */
180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	228 if ((dec & MAIL_CACHE_DECISION_FORCED) == 0 &&
13842 ce29836e369e lib-index: Allow updating cache's last_used field with mail_cache_register_fields() Timo Sirainen <tss@iki.fi> parents: 13339 diff changeset	229 priv->field.last_used < max_drop_time &&
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	230 !priv->adding) {
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	231 dec = MAIL_CACHE_DECISION_NO;
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	232 priv->field.decision = dec;
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	233 }
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	234
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	235 /* drop all fields we don't want */
f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	236 if ((dec & ~MAIL_CACHE_DECISION_FORCED) ==
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	237 MAIL_CACHE_DECISION_NO && !priv->adding) {
1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	238 priv->used = FALSE;
13842 ce29836e369e lib-index: Allow updating cache's last_used field with mail_cache_register_fields() Timo Sirainen <tss@iki.fi> parents: 13339 diff changeset	239 priv->field.last_used = 0;
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	240 }
f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	241
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	242 ctx.field_file_map[i] = !priv->used ?
6696 68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	243 (uint32_t)-1 : used_fields_count++;
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	244 }
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	245 }
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	246
6841 0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	247 /* get sequence of first message which doesn't need its temp fields
0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	248 removed. */
0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	249 first_new_seq = mail_cache_get_first_new_seq(view);
0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	250 message_count = mail_index_view_get_messages_count(view);
0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	251
15374 debecd057f9c lib-index: Replaced cache's deleted_space with deleted_record_count. Timo Sirainen <tss@iki.fi> parents: 15373 diff changeset	252 i_array_init(ext_offsets, message_count); record_count = 0;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	253 for (seq = 1; seq <= message_count; seq++) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	254 if (mail_index_transaction_is_expunged(trans, seq)) {
14686 9ff19c1d5f69 Added array_append_zero() to write a zero-filled record to an array. Timo Sirainen <tss@iki.fi> parents: 14682 diff changeset	255 array_append_zero(ext_offsets);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	256 continue;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	257 }
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	258
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	259 ctx.new_msg = seq >= first_new_seq;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	260 buffer_set_used_size(ctx.buffer, 0);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	261
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	262 if (++ctx.field_seen_value == 0) {
4451 1a35d53c18fc Array API redesigned to work using unions. It now provides type safety Timo Sirainen <tss@iki.fi> parents: 4247 diff changeset	263 memset(buffer_get_modifiable_data(ctx.field_seen, NULL),
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	264 0, buffer_get_size(ctx.field_seen));
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	265 ctx.field_seen_value++;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	266 }
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	267
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20561 diff changeset	268 i_zero(&cache_rec);
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	269 buffer_append(ctx.buffer, &cache_rec, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	270
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	271 mail_cache_lookup_iter_init(cache_view, seq, &iter);
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	272 while (mail_cache_lookup_iter_next(&iter, &field) > 0)
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	273 mail_cache_compress_field(&ctx, &field);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	274
15372 d2860e8fc774 lib-index: If a cache record is larger than 64 kB, don't add it to cache file. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	275 if (ctx.buffer->used == sizeof(cache_rec) \|\|
22611 65ada9976a7f lib-index: Add mail_index_cache_optimization_settings Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21390 diff changeset	276 ctx.buffer->used > cache->index->optimization_set.cache.record_max_size) {
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	277 /* nothing cached */
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	278 ext_offset = 0;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	279 } else {
20561 6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	280 mail_index_lookup_uid(view, seq, max_uid_r);
15372 d2860e8fc774 lib-index: If a cache record is larger than 64 kB, don't add it to cache file. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	281 cache_rec.size = ctx.buffer->used;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	282 ext_offset = output->offset;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	283 buffer_write(ctx.buffer, 0, &cache_rec,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	284 sizeof(cache_rec));
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14637 diff changeset	285 o_stream_nsend(output, ctx.buffer->data, cache_rec.size);
15374 debecd057f9c lib-index: Replaced cache's deleted_space with deleted_record_count. Timo Sirainen <tss@iki.fi> parents: 15373 diff changeset	286 record_count++;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	287 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	288
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	289 array_append(ext_offsets, &ext_offset, 1);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	290 }
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	291 i_assert(orig_fields_count == cache->fields_count);
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	292
15374 debecd057f9c lib-index: Replaced cache's deleted_space with deleted_record_count. Timo Sirainen <tss@iki.fi> parents: 15373 diff changeset	293 hdr.record_count = record_count;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	294 hdr.field_header_offset = mail_index_uint32_to_offset(output->offset);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	295 mail_cache_compress_get_fields(&ctx, used_fields_count);
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14637 diff changeset	296 o_stream_nsend(output, ctx.buffer->data, ctx.buffer->used);
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	297
15190 60c4815778fb lib-index: Simplified writing to dovecot.index.cache file. Timo Sirainen <tss@iki.fi> parents: 14920 diff changeset	298 hdr.backwards_compat_used_file_size = output->offset;
6414 a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6319 diff changeset	299 buffer_free(&ctx.buffer);
a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6319 diff changeset	300 buffer_free(&ctx.field_seen);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	301
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14681 diff changeset	302 (void)o_stream_seek(output, 0);
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14637 diff changeset	303 o_stream_nsend(output, &hdr, sizeof(hdr));
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	304
13339 b355309955f4 lib-index: Changed mail_cache_view_close() API to take pointer-to-pointer. Timo Sirainen <tss@iki.fi> parents: 12782 diff changeset	305 mail_cache_view_close(&cache_view);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	306
14681 ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14637 diff changeset	307 if (o_stream_nfinish(output) < 0) {
ca37d1577291 Added o_stream_nsend() and related functions to make delayed error handling safer. Timo Sirainen <tss@iki.fi>* parents: 14637 diff changeset	308 mail_cache_set_syscall_error(cache, "write()");
4070 71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of Timo Sirainen <tss@iki.fi> parents: 3879 diff changeset	309 o_stream_destroy(&output);
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	310 array_free(ext_offsets);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	311 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	312 }
20561 6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	313 *file_size_r = output->offset;
4070 71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of Timo Sirainen <tss@iki.fi> parents: 3879 diff changeset	314 o_stream_destroy(&output);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	315
11634 7f19062f58fd Renamed fsync_disable to mail_fsync=optimized\|always\|never. Timo Sirainen <tss@iki.fi> parents: 11076 diff changeset	316 if (cache->index->fsync_mode == FSYNC_MODE_ALWAYS) {
4918 2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	317 if (fdatasync(fd) < 0) {
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	318 mail_cache_set_syscall_error(cache, "fdatasync()");
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	319 array_free(ext_offsets);
4918 2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	320 return -1;
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	321 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	322 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	323
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	324 *file_seq_r = hdr.file_seq;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	325 return 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	326 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	327
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	328 static int
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	329 mail_cache_compress_write(struct mail_cache *cache,
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	330 struct mail_index_transaction *trans,
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	331 int fd, const char temp_path, bool unlock)
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	332 {
6823 ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	333 struct stat st;
20561 6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	334 uint32_t file_seq, old_offset, max_uid;
5733 4b5ce642957d Declare a global uint32_t array type. Timo Sirainen <tss@iki.fi> parents: 5652 diff changeset	335 ARRAY_TYPE(uint32_t) ext_offsets;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	336 const uint32_t *offsets;
20561 6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	337 uoff_t file_size;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	338 unsigned int i, count;
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	339
20561 6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	340 if (mail_cache_copy(cache, trans, fd, &file_seq, &file_size,
6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	341 &max_uid, &ext_offsets) < 0)
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	342 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	343
6823 ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	344 if (fstat(fd, &st) < 0) {
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	345 mail_cache_set_syscall_error(cache, "fstat()");
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	346 array_free(&ext_offsets);
6823 ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	347 return -1;
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	348 }
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	349 if (rename(temp_path, cache->filepath) < 0) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	350 mail_cache_set_syscall_error(cache, "rename()");
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	351 array_free(&ext_offsets);
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	352 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	353 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	354
20561 6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	355 if ((cache->index->flags & MAIL_INDEX_OPEN_FLAG_DEBUG) != 0) {
6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	356 i_debug("%s: Compressed, file_seq changed %u -> %u, "
6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	357 "size=%"PRIuUOFF_T", max_uid=%u", cache->filepath,
6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	358 cache->need_compress_file_seq, file_seq,
6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	359 file_size, max_uid);
6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	360 }
6f381c5ad896 lib-index: If mail_debug=yes, log cache compressions. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20195 diff changeset	361
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	362 /* once we're sure that the compression was successful,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	363 update the offsets */
7563 6de1aed24ce5 Added mail_index_ext_reset_inc() to atomically increase extension's Timo Sirainen <tss@iki.fi> parents: 7561 diff changeset	364 mail_index_ext_reset(trans, cache->ext_id, file_seq, TRUE);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	365 offsets = array_get(&ext_offsets, &count);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	366 for (i = 0; i < count; i++) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	367 if (offsets[i] != 0) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	368 mail_index_update_ext(trans, i + 1, cache->ext_id,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	369 &offsets[i], &old_offset);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	370 }
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	371 }
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	372 array_free(&ext_offsets);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	373
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	374 if (*unlock) {
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	375 (void)mail_cache_unlock(cache);
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	376 *unlock = FALSE;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	377 }
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	378
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	379 mail_cache_file_close(cache);
20041 4d83654e61d6 lib-index: If opening a cache file fails, try again later. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	380 cache->opened = TRUE;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	381 cache->fd = fd;
6823 ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	382 cache->st_ino = st.st_ino;
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	383 cache->st_dev = st.st_dev;
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	384 cache->field_header_write_pending = FALSE;
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	385 return 0;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	386 }
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	387
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	388 static int mail_cache_compress_has_file_changed(struct mail_cache *cache)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	389 {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	390 struct mail_cache_header hdr;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	391 unsigned int i;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	392 int fd, ret;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	393
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	394 for (i = 0;; i++) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	395 fd = nfs_safe_open(cache->filepath, O_RDONLY);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	396 if (fd == -1) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	397 if (errno == ENOENT)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	398 return 0;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	399
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	400 mail_cache_set_syscall_error(cache, "open()");
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	401 return -1;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	402 }
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	403
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	404 ret = read_full(fd, &hdr, sizeof(hdr));
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	405 i_close_fd(&fd);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	406
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	407 if (ret >= 0) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	408 if (ret == 0)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	409 return 0;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	410 if (cache->need_compress_file_seq == 0) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	411 /* previously it didn't exist or it
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	412 was unusable and was just unlinked */
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	413 return 1;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	414 }
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	415 return hdr.file_seq != cache->need_compress_file_seq;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	416 } else if (errno != ESTALE \|\| i >= NFS_ESTALE_RETRY_COUNT) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	417 mail_cache_set_syscall_error(cache, "read()");
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	418 return -1;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	419 }
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	420 }
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	421 }
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	422
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	423 static int mail_cache_compress_dotlock(struct mail_cache *cache,
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	424 struct dotlock **dotlock_r)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	425 {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	426 if (file_dotlock_create(&cache->dotlock_settings, cache->filepath,
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	427 DOTLOCK_CREATE_FLAG_NONBLOCK, dotlock_r) <= 0) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	428 if (errno != EAGAIN)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	429 mail_cache_set_syscall_error(cache, "file_dotlock_open()");
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	430 return -1;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	431 }
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	432 return 0;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	433 }
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	434
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	435 static int mail_cache_compress_locked(struct mail_cache *cache,
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	436 struct mail_index_transaction *trans,
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	437 bool unlock, struct dotlock *dotlock_r)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	438 {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	439 const char *temp_path;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	440 const void *data;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	441 int fd, ret;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	442
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	443 /* There are two possible locking situations here:
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	444 a) Cache is locked against any modifications.
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	445 b) Cache doesn't exist or is unusable. There's no lock.
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	446 Because the cache lock itself is unreliable, we'll be using a
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	447 separate dotlock to guard against two processes compressing the
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	448 cache at the same time. */
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	449
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	450 if (mail_cache_compress_dotlock(cache, dotlock_r) < 0)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	451 return -1;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	452 /* we've locked the cache compression now. if somebody else had just
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	453 recreated the cache, reopen the cache and return success. */
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	454 if ((ret = mail_cache_compress_has_file_changed(cache)) != 0) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	455 if (ret < 0)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	456 return -1;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	457
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	458 /* was just compressed, forget this */
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	459 cache->need_compress_file_seq = 0;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	460 file_dotlock_delete(dotlock_r);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	461
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	462 if (*unlock) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	463 (void)mail_cache_unlock(cache);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	464 *unlock = FALSE;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	465 }
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	466
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	467 return mail_cache_reopen(cache) < 0 ? -1 : 0;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	468 }
20195 bdd3b23552fc lib-index: Fixed assert-crash caused by previous cache compression change. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20194 diff changeset	469 if (cache->fd != -1) {
bdd3b23552fc lib-index: Fixed assert-crash caused by previous cache compression change. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20194 diff changeset	470 /* make sure we have mapped it before reading. */
bdd3b23552fc lib-index: Fixed assert-crash caused by previous cache compression change. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20194 diff changeset	471 if (mail_cache_map(cache, 0, 0, &data) < 0)
bdd3b23552fc lib-index: Fixed assert-crash caused by previous cache compression change. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20194 diff changeset	472 return -1;
bdd3b23552fc lib-index: Fixed assert-crash caused by previous cache compression change. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20194 diff changeset	473 }
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	474
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	475 /* we want to recreate the cache. write it first to a temporary file */
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	476 fd = mail_index_create_tmp_file(cache->index, cache->filepath, &temp_path);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	477 if (fd == -1)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	478 return -1;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	479 if (mail_cache_compress_write(cache, trans, fd, temp_path, unlock) < 0) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	480 i_close_fd(&fd);
19136 fefaa6d09a81 Replaced unlink() calls with i_unlink() wherever possible. Timo Sirainen <tss@iki.fi>* parents: 18137 diff changeset	481 i_unlink(temp_path);
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	482 return -1;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	483 }
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	484 if (cache->file_cache != NULL)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	485 file_cache_set_fd(cache->file_cache, cache->fd);
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	486
15311 022d0d21e56d lib-index: mail_cache_map() API cleanup Timo Sirainen <tss@iki.fi> parents: 14133 diff changeset	487 if (mail_cache_map(cache, 0, 0, &data) < 0)
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	488 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	489 if (mail_cache_header_fields_read(cache) < 0)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	490 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	491
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	492 cache->need_compress_file_seq = 0;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	493 return 0;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	494 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	495
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	496 int mail_cache_compress(struct mail_cache *cache,
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	497 struct mail_index_transaction *trans,
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	498 struct mail_cache_compress_lock **lock_r)
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	499 {
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	500 struct dotlock *dotlock = NULL;
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	501 bool unlock = FALSE;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	502 int ret;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	503
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	504 i_assert(!cache->compressing);
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	505
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	506 *lock_r = NULL;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	507
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	508 if (MAIL_INDEX_IS_IN_MEMORY(cache->index) \|\| cache->index->readonly) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	509 *lock_r = i_new(struct mail_cache_compress_lock, 1);
3243 40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	510 return 0;
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	511 }
3243 40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	512
15313 51f2be9aa8ad lib-index: Added MAIL_INDEX_OPEN_FLAG_SAVEONLY to do only minimal reads from cache file. Timo Sirainen <tss@iki.fi> parents: 15311 diff changeset	513 /* compression isn't very efficient with small read()s */
15567 32ce915e046a lib-index: If map_with_read gets disabled, reset the cache buffers properly. Timo Sirainen <tss@iki.fi> parents: 15313 diff changeset	514 if (cache->map_with_read) {
32ce915e046a lib-index: If map_with_read gets disabled, reset the cache buffers properly. Timo Sirainen <tss@iki.fi> parents: 15313 diff changeset	515 cache->map_with_read = FALSE;
32ce915e046a lib-index: If map_with_read gets disabled, reset the cache buffers properly. Timo Sirainen <tss@iki.fi> parents: 15313 diff changeset	516 if (cache->read_buf != NULL)
32ce915e046a lib-index: If map_with_read gets disabled, reset the cache buffers properly. Timo Sirainen <tss@iki.fi> parents: 15313 diff changeset	517 buffer_set_used_size(cache->read_buf, 0);
32ce915e046a lib-index: If map_with_read gets disabled, reset the cache buffers properly. Timo Sirainen <tss@iki.fi> parents: 15313 diff changeset	518 cache->hdr = NULL;
32ce915e046a lib-index: If map_with_read gets disabled, reset the cache buffers properly. Timo Sirainen <tss@iki.fi> parents: 15313 diff changeset	519 cache->mmap_length = 0;
32ce915e046a lib-index: If map_with_read gets disabled, reset the cache buffers properly. Timo Sirainen <tss@iki.fi> parents: 15313 diff changeset	520 }
15313 51f2be9aa8ad lib-index: Added MAIL_INDEX_OPEN_FLAG_SAVEONLY to do only minimal reads from cache file. Timo Sirainen <tss@iki.fi> parents: 15311 diff changeset	521
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	522 if (cache->index->lock_method == FILE_LOCK_METHOD_DOTLOCK) {
2949 05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	523 /* we're using dotlocking, cache file creation itself creates
05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	524 the dotlock file we need. */
6829 dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	525 if (!MAIL_CACHE_IS_UNUSABLE(cache)) {
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	526 mail_index_flush_read_cache(cache->index,
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	527 cache->filepath, cache->fd,
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	528 FALSE);
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	529 }
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	530 } else {
8188 2db48458d73b Mail cache compression: If we can't get lock immediately, do it later. Timo Sirainen <tss@iki.fi> parents: 8187 diff changeset	531 switch (mail_cache_try_lock(cache)) {
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	532 case -1:
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	533 /* already locked or some other error */
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	534 return -1;
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	535 case 0:
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	536 /* cache is broken or doesn't exist.
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	537 just start creating it. */
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	538 break;
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	539 default:
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	540 /* locking succeeded. */
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	541 unlock = TRUE;
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	542 }
2949 05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	543 }
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	544 cache->compressing = TRUE;
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	545 ret = mail_cache_compress_locked(cache, trans, &unlock, &dotlock);
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	546 cache->compressing = FALSE;
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	547 if (unlock) {
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	548 if (mail_cache_unlock(cache) < 0)
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	549 ret = -1;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	550 }
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	551 if (ret < 0) {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	552 if (dotlock != NULL)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	553 file_dotlock_delete(&dotlock);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	554 /* the fields may have been updated in memory already.
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	555 reverse those changes by re-reading them from file. */
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	556 (void)mail_cache_header_fields_read(cache);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	557 } else {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	558 *lock_r = i_new(struct mail_cache_compress_lock, 1);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	559 (*lock_r)->dotlock = dotlock;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	560 }
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	561 return ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	562 }
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	563
17905 51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	564 void mail_cache_compress_unlock(struct mail_cache_compress_lock **_lock)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	565 {
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	566 struct mail_cache_compress_lock lock = _lock;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	567
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	568 *_lock = NULL;
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	569
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	570 if (lock->dotlock != NULL)
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	571 file_dotlock_delete(&lock->dotlock);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	572 i_free(lock);
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	573 }
51f5680d4108 lib-index: Delay unlocking cache compression until changes to transaction log are committed. Timo Sirainen <tss@iki.fi> parents: 17903 diff changeset	574
3863 55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool. Timo Sirainen <tss@iki.fi> parents: 3819 diff changeset	575 bool mail_cache_need_compress(struct mail_cache *cache)
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	576 {
6686 610f3d9813b5 Added MAIL_INDEX_OPEN_FLAG_READONLY. Timo Sirainen <tss@iki.fi> parents: 6678 diff changeset	577 return cache->need_compress_file_seq != 0 &&
15313 51f2be9aa8ad lib-index: Added MAIL_INDEX_OPEN_FLAG_SAVEONLY to do only minimal reads from cache file. Timo Sirainen <tss@iki.fi> parents: 15311 diff changeset	578 (cache->index->flags & MAIL_INDEX_OPEN_FLAG_SAVEONLY) == 0 &&
6686 610f3d9813b5 Added MAIL_INDEX_OPEN_FLAG_READONLY. Timo Sirainen <tss@iki.fi> parents: 6678 diff changeset	579 !cache->index->readonly;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	580 }

Mercurial > dovecot > core-2.2

annotate src/lib-index/mail-cache-compress.c @ 22691:dca05b22217b