dovecot/original-hg/dovecot-1.2: src/lib-index/mail-cache-compress.c annotate

annotate src/lib-index/mail-cache-compress.c @ 6677:90bcf480312d HEAD

Renamed mail_index_lookup_uid_range() to mail_index_lookup_seq_range(), made it return a bool and added mail_index_lookup_seq(). Cleaned up the code to use these functions.

author	Timo Sirainen <tss@iki.fi>
date	Sat, 03 Nov 2007 21:56:05 +0200
parents	d79975141df2
children	6601ee607b6f

rev	line source
6429 65c69a53a7be Replaced my Copyright notices. The year range always ends with 2007 now. Timo Sirainen <tss@iki.fi> parents: 6414 diff changeset	1 /* Copyright (c) 2003-2007 Dovecot authors, see the included COPYING file */
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	2
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	3 #include "lib.h"
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	4 #include "array.h"
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	5 #include "ostream.h"
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	6 #include "nfs-workarounds.h"
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	7 #include "read-full.h"
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	8 #include "close-keep-errno.h"
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	9 #include "file-dotlock.h"
2866 bf1e718e7370 Cache file works now with mmap_disable=yes. Still needs a few optimizations. Timo Sirainen <tss@iki.fi> parents: 2853 diff changeset	10 #include "file-cache.h"
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	11 #include "file-set-size.h"
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	12 #include "mail-cache-private.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	13
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	14 #include <sys/stat.h>
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	15
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	16 struct mail_cache_copy_context {
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	17 struct mail_cache *cache;
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	18
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	19 buffer_t buffer, field_seen;
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	20 ARRAY_DEFINE(bitmask_pos, unsigned int);
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	21 uint32_t *field_file_map;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	22
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	23 uint8_t field_seen_value;
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	24 bool new_msg;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	25 };
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	26
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	27 static void
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	28 mail_cache_merge_bitmask(struct mail_cache_copy_context *ctx,
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	29 const struct mail_cache_iterate_field *field)
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	30 {
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	31 unsigned char *dest;
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	32 unsigned int i, *pos;
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	33
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	34 pos = array_idx_modifiable(&ctx->bitmask_pos, field->field_idx);
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	35 if (*pos == 0) {
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	36 /* we decided to drop this field */
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	37 return;
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	38 }
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	39
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	40 dest = buffer_get_space_unsafe(ctx->buffer, *pos, field->size);
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	41 for (i = 0; i < field->size; i++)
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	42 dest[i] \|= ((const unsigned char*)field->data)[i];
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	43 }
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	44
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	45 static void
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	46 mail_cache_compress_field(struct mail_cache_copy_context *ctx,
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	47 const struct mail_cache_iterate_field *field)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	48 {
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	49 struct mail_cache_field *cache_field;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	50 enum mail_cache_decision_type dec;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	51 uint32_t file_field_idx, size32;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	52 uint8_t *field_seen;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	53
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	54 file_field_idx = ctx->field_file_map[field->field_idx];
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	55 if (file_field_idx == (uint32_t)-1)
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	56 return;
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	57
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	58 cache_field = &ctx->cache->fields[field->field_idx].field;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	59
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	60 field_seen = buffer_get_space_unsafe(ctx->field_seen,
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	61 field->field_idx, 1);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	62 if (*field_seen == ctx->field_seen_value) {
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	63 /* duplicate */
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	64 if (cache_field->type == MAIL_CACHE_FIELD_BITMASK)
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	65 mail_cache_merge_bitmask(ctx, field);
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	66 return;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	67 }
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	68 *field_seen = ctx->field_seen_value;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	69
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	70 dec = cache_field->decision & ~MAIL_CACHE_DECISION_FORCED;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	71 if (ctx->new_msg) {
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	72 if (dec == MAIL_CACHE_DECISION_NO)
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	73 return;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	74 } else {
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	75 if (dec != MAIL_CACHE_DECISION_YES)
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	76 return;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	77 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	78
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	79 buffer_append(ctx->buffer, &file_field_idx, sizeof(file_field_idx));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	80
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	81 if (cache_field->field_size == (unsigned int)-1) {
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	82 size32 = (uint32_t)field->size;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	83 buffer_append(ctx->buffer, &size32, sizeof(size32));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	84 }
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	85
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	86 if (cache_field->type == MAIL_CACHE_FIELD_BITMASK) {
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	87 /* remember the position in case we need to update it */
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	88 unsigned int pos = ctx->buffer->used;
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	89
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	90 array_idx_set(&ctx->bitmask_pos, field->field_idx, &pos);
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	91 }
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	92 buffer_append(ctx->buffer, field->data, field->size);
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	93 if ((field->size & 3) != 0)
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	94 buffer_append_zero(ctx->buffer, 4 - (field->size & 3));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	95 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	96
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	97 static uint32_t
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	98 get_next_file_seq(struct mail_cache cache, struct mail_index_view view)
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	99 {
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	100 const struct mail_index_ext *ext;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	101 uint32_t file_seq;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	102
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	103 ext = mail_index_view_get_ext(view, cache->ext_id);
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	104 file_seq = ext != NULL ? ext->reset_id + 1 : (uint32_t)ioloop_time;
5994 8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	105
8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	106 if (cache->hdr != NULL && file_seq <= cache->hdr->file_seq)
8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	107 file_seq = cache->hdr->file_seq + 1;
8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	108
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	109 return file_seq != 0 ? file_seq : 1;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	110 }
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	111
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	112 static void
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	113 mail_cache_compress_get_fields(struct mail_cache_copy_context *ctx,
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	114 unsigned int used_fields_count)
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	115 {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	116 struct mail_cache *cache = ctx->cache;
6297 3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	117 struct mail_cache_field *field;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	118 unsigned int i, j, idx;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	119
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	120 /* Make mail_cache_header_fields_get() return the fields in
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	121 the same order as we saved them. */
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	122 memcpy(cache->field_file_map, ctx->field_file_map,
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	123 sizeof(uint32_t) * cache->fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	124
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	125 /* reverse mapping */
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	126 cache->file_fields_count = used_fields_count;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	127 i_free(cache->file_field_map);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	128 cache->file_field_map = used_fields_count == 0 ? NULL :
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	129 i_new(unsigned int, used_fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	130 for (i = j = 0; i < cache->fields_count; i++) {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	131 idx = cache->field_file_map[i];
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	132 if (idx != (uint32_t)-1) {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	133 i_assert(idx < used_fields_count &&
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	134 cache->file_field_map[idx] == 0);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	135 cache->file_field_map[idx] = i;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	136 j++;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	137 }
6297 3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	138
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	139 /* change permanent decisions to temporary decisions.
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	140 if they're still permanent they'll get updated later. */
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	141 field = &cache->fields[i].field;
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	142 if (field->decision == MAIL_CACHE_DECISION_YES)
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	143 field->decision = MAIL_CACHE_DECISION_TEMP;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	144 }
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	145 i_assert(j == used_fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	146
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	147 buffer_set_used_size(ctx->buffer, 0);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	148 mail_cache_header_fields_get(cache, ctx->buffer);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	149 }
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	150
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	151 static int
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	152 mail_cache_copy(struct mail_cache cache, struct mail_index_transaction trans,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	153 int fd, uint32_t *file_seq_r,
5733 4b5ce642957d Declare a global uint32_t array type. Timo Sirainen <tss@iki.fi> parents: 5652 diff changeset	154 ARRAY_TYPE(uint32_t) *ext_offsets)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	155 {
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	156 struct mail_cache_copy_context ctx;
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	157 struct mail_cache_lookup_iterate_ctx iter;
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	158 struct mail_cache_iterate_field field;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	159 struct mail_index_view *view;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	160 struct mail_cache_view *cache_view;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	161 const struct mail_index_header *idx_hdr;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	162 struct mail_cache_header hdr;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	163 struct mail_cache_record cache_rec;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	164 struct ostream *output;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	165 uint32_t message_count, seq, first_new_seq, ext_offset;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	166 unsigned int i, used_fields_count, orig_fields_count;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	167 time_t max_drop_time;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	168
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	169 view = mail_index_transaction_get_view(trans);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	170
3551 65dc45982df9 Removed fcntl() text from some locking error messages, because if Timo Sirainen <tss@iki.fi> parents: 3243 diff changeset	171 /* get sequence of first message which doesn't need its temp fields
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	172 removed. */
2892 62d53b49110d Changed mail_index_get_header() to return the header as return value because Timo Sirainen <tss@iki.fi> parents: 2879 diff changeset	173 idx_hdr = mail_index_get_header(view);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	174 if (idx_hdr->day_first_uid[7] == 0) {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	175 first_new_seq = 1;
2904 642ba9cd3dac Removed view->messages_count, view->hdr.messages_count is enough. Also fixes Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	176 message_count = mail_index_view_get_messages_count(view);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	177 } else {
6677 90bcf480312d Renamed mail_index_lookup_uid_range() to mail_index_lookup_seq_range(), made Timo Sirainen <tss@iki.fi> parents: 6510 diff changeset	178 if (!mail_index_lookup_seq_range(view,
90bcf480312d Renamed mail_index_lookup_uid_range() to mail_index_lookup_seq_range(), made Timo Sirainen <tss@iki.fi> parents: 6510 diff changeset	179 idx_hdr->day_first_uid[7],
90bcf480312d Renamed mail_index_lookup_uid_range() to mail_index_lookup_seq_range(), made Timo Sirainen <tss@iki.fi> parents: 6510 diff changeset	180 (uint32_t)-1, &first_new_seq,
90bcf480312d Renamed mail_index_lookup_uid_range() to mail_index_lookup_seq_range(), made Timo Sirainen <tss@iki.fi> parents: 6510 diff changeset	181 &message_count))
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	182 first_new_seq = message_count+1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	183 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	184
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	185 cache_view = mail_cache_view_open(cache, view);
6161 c62f7ee79446 Split o_stream_create_file() to _create_fd() and _create_fd_file(). Timo Sirainen <tss@iki.fi> parents: 6142 diff changeset	186 output = o_stream_create_fd_file(fd, 0, FALSE);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	187
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	188 memset(&hdr, 0, sizeof(hdr));
2276 5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code - Timo Sirainen <tss@iki.fi> parents: 2275 diff changeset	189 hdr.version = MAIL_CACHE_VERSION;
3819 61f0e1fe9b55 Moved uoff_t and time_t compatibility checks to cache file, since they're Timo Sirainen <tss@iki.fi> parents: 3627 diff changeset	190 hdr.compat_sizeof_uoff_t = sizeof(uoff_t);
5819 4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5737 diff changeset	191 hdr.indexid = cache->index->indexid;
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	192 hdr.file_seq = get_next_file_seq(cache, view);
2339 406692edc12d Cache fixes. Decisions are saved again. Timo Sirainen <tss@iki.fi> parents: 2327 diff changeset	193 o_stream_send(output, &hdr, sizeof(hdr));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	194
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	195 memset(&ctx, 0, sizeof(ctx));
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	196 ctx.cache = cache;
2708 f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where Timo Sirainen <tss@iki.fi> parents: 2700 diff changeset	197 ctx.buffer = buffer_create_dynamic(default_pool, 4096);
f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where Timo Sirainen <tss@iki.fi> parents: 2700 diff changeset	198 ctx.field_seen = buffer_create_dynamic(default_pool, 64);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	199 ctx.field_seen_value = 0;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	200 ctx.field_file_map = t_new(uint32_t, cache->fields_count);
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	201 t_array_init(&ctx.bitmask_pos, 32);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	202
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	203 /* @UNSAFE: drop unused fields and create a field mapping for
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	204 used fields */
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	205 max_drop_time = idx_hdr->day_stamp - MAIL_CACHE_FIELD_DROP_SECS;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	206 orig_fields_count = cache->fields_count;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	207 for (i = used_fields_count = 0; i < orig_fields_count; i++) {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	208 if (cache->fields[i].last_used < max_drop_time)
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	209 cache->fields[i].used = FALSE;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	210
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	211 ctx.field_file_map[i] = !cache->fields[i].used ? (uint32_t)-1 :
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	212 used_fields_count++;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	213 }
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	214
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	215 i_array_init(ext_offsets, message_count);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	216 for (seq = 1; seq <= message_count; seq++) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	217 if (mail_index_transaction_is_expunged(trans, seq)) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	218 (void)array_append_space(ext_offsets);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	219 continue;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	220 }
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	221
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	222 ctx.new_msg = seq >= first_new_seq;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	223 buffer_set_used_size(ctx.buffer, 0);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	224
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	225 if (++ctx.field_seen_value == 0) {
4451 1a35d53c18fc Array API redesigned to work using unions. It now provides type safety Timo Sirainen <tss@iki.fi> parents: 4247 diff changeset	226 memset(buffer_get_modifiable_data(ctx.field_seen, NULL),
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	227 0, buffer_get_size(ctx.field_seen));
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	228 ctx.field_seen_value++;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	229 }
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	230
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	231 memset(&cache_rec, 0, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	232 buffer_append(ctx.buffer, &cache_rec, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	233
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	234 mail_cache_lookup_iter_init(cache_view, seq, &iter);
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	235 while (mail_cache_lookup_iter_next(&iter, &field) > 0)
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	236 mail_cache_compress_field(&ctx, &field);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	237
2299 ef0cd18d674c Fixed compression. Timo Sirainen <tss@iki.fi> parents: 2298 diff changeset	238 cache_rec.size = buffer_get_used_size(ctx.buffer);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	239 if (cache_rec.size == sizeof(cache_rec)) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	240 /* nothing cached */
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	241 ext_offset = 0;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	242 } else {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	243 ext_offset = output->offset;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	244 buffer_write(ctx.buffer, 0, &cache_rec,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	245 sizeof(cache_rec));
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	246 o_stream_send(output, ctx.buffer->data, cache_rec.size);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	247 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	248
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	249 array_append(ext_offsets, &ext_offset, 1);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	250 }
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	251 i_assert(array_count(ext_offsets) == message_count);
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	252 i_assert(orig_fields_count == cache->fields_count);
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	253
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	254 hdr.field_header_offset = mail_index_uint32_to_offset(output->offset);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	255 mail_cache_compress_get_fields(&ctx, used_fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	256 o_stream_send(output, ctx.buffer->data, ctx.buffer->used);
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	257
2276 5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code - Timo Sirainen <tss@iki.fi> parents: 2275 diff changeset	258 hdr.used_file_size = output->offset;
6414 a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6319 diff changeset	259 buffer_free(&ctx.buffer);
a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6319 diff changeset	260 buffer_free(&ctx.field_seen);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	261
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	262 o_stream_seek(output, 0);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	263 o_stream_send(output, &hdr, sizeof(hdr));
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	264
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	265 mail_cache_view_close(cache_view);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	266
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	267 if (o_stream_flush(output) < 0) {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	268 errno = output->stream_errno;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	269 mail_cache_set_syscall_error(cache, "o_stream_flush()");
4070 71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of Timo Sirainen <tss@iki.fi> parents: 3879 diff changeset	270 o_stream_destroy(&output);
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	271 array_free(ext_offsets);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	272 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	273 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	274
2299 ef0cd18d674c Fixed compression. Timo Sirainen <tss@iki.fi> parents: 2298 diff changeset	275 if (hdr.used_file_size < MAIL_CACHE_INITIAL_SIZE) {
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	276 /* grow the file some more. doesn't matter if it fails */
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	277 (void)file_set_size(fd, MAIL_CACHE_INITIAL_SIZE);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	278 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	279
4070 71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of Timo Sirainen <tss@iki.fi> parents: 3879 diff changeset	280 o_stream_destroy(&output);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	281
4918 2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	282 if (!cache->index->fsync_disable) {
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	283 if (fdatasync(fd) < 0) {
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	284 mail_cache_set_syscall_error(cache, "fdatasync()");
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	285 array_free(ext_offsets);
4918 2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	286 return -1;
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	287 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	288 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	289
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	290 *file_seq_r = hdr.file_seq;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	291 return 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	292 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	293
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	294 static int mail_cache_compress_has_file_changed(struct mail_cache *cache)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	295 {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	296 struct mail_cache_header hdr;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	297 unsigned int i;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	298 int fd, ret;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	299
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	300 for (i = 0;; i++) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	301 fd = nfs_safe_open(cache->filepath, O_RDONLY);
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	302 if (fd == -1) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	303 if (errno == ENOENT)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	304 return 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	305
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	306 mail_cache_set_syscall_error(cache, "open()");
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	307 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	308 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	309
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	310 ret = read_full(fd, &hdr, sizeof(hdr));
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	311 close_keep_errno(fd);
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	312
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	313 if (ret >= 0) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	314 if (ret == 0)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	315 return 0;
5993 c66c80b23f95 need_compress_file_seq=0 nowadays when file doesn't exist, not -1. Timo Sirainen <tss@iki.fi> parents: 5972 diff changeset	316 if (cache->need_compress_file_seq == 0) {
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	317 /* previously it didn't exist */
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	318 return 1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	319 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	320 return hdr.file_seq != cache->need_compress_file_seq;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	321 } else if (errno != ESTALE \|\| i >= NFS_ESTALE_RETRY_COUNT) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	322 mail_cache_set_syscall_error(cache, "read()");
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	323 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	324 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	325 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	326 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	327
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	328 static int mail_cache_compress_locked(struct mail_cache *cache,
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	329 struct mail_index_transaction *trans,
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	330 bool *unlock)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	331 {
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	332 struct dotlock *dotlock;
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	333 mode_t old_mask;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	334 uint32_t file_seq, old_offset;
5733 4b5ce642957d Declare a global uint32_t array type. Timo Sirainen <tss@iki.fi> parents: 5652 diff changeset	335 ARRAY_TYPE(uint32_t) ext_offsets;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	336 const uint32_t *offsets;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	337 unsigned int i, count;
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	338 int fd, ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	339
6510 d79975141df2 Fixes to NFS read cache flushing. Timo Sirainen <tss@iki.fi> parents: 6429 diff changeset	340 if (!MAIL_CACHE_IS_UNUSABLE(cache))
5972 9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5947 diff changeset	341 mail_cache_flush_read_cache(cache, TRUE);
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5947 diff changeset	342
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	343 /* get the latest info on fields */
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	344 if (mail_cache_header_fields_read(cache) < 0)
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	345 return -1;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	346
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	347 old_mask = umask(cache->index->mode ^ 0666);
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	348 fd = file_dotlock_open(&cache->dotlock_settings, cache->filepath,
9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	349 0, &dotlock);
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	350 umask(old_mask);
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	351
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	352 if (fd == -1) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	353 mail_cache_set_syscall_error(cache, "file_dotlock_open()");
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	354 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	355 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	356
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	357 if ((ret = mail_cache_compress_has_file_changed(cache)) != 0) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	358 if (ret < 0)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	359 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	360
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	361 /* was just compressed, forget this */
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	362 cache->need_compress_file_seq = 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	363 file_dotlock_delete(&dotlock);
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	364
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	365 if (*unlock) {
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	366 (void)mail_cache_unlock(cache);
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	367 *unlock = FALSE;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	368 }
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	369
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	370 return mail_cache_reopen(cache);
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	371 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	372
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	373 if (cache->index->gid != (gid_t)-1 &&
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	374 fchown(fd, (uid_t)-1, cache->index->gid) < 0) {
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	375 mail_cache_set_syscall_error(cache, "fchown()");
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	376 file_dotlock_delete(&dotlock);
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	377 return -1;
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	378 }
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	379
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	380 if (mail_cache_copy(cache, trans, fd, &file_seq, &ext_offsets) < 0) {
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	381 (void)file_dotlock_delete(&dotlock);
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	382 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	383 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	384
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	385 if (file_dotlock_replace(&dotlock,
9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	386 DOTLOCK_REPLACE_FLAG_DONT_CLOSE_FD) < 0) {
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	387 mail_cache_set_syscall_error(cache,
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	388 "file_dotlock_replace()");
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	389 (void)close(fd);
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	390 array_free(&ext_offsets);
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	391 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	392 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	393
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	394 /* once we're sure that the compression was successful,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	395 update the offsets */
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	396 mail_index_ext_reset(trans, cache->ext_id, file_seq);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	397 offsets = array_get(&ext_offsets, &count);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	398 for (i = 0; i < count; i++) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	399 if (offsets[i] != 0) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	400 mail_index_update_ext(trans, i + 1, cache->ext_id,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	401 &offsets[i], &old_offset);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	402 }
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	403 }
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	404 array_free(&ext_offsets);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	405
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	406 if (*unlock) {
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	407 (void)mail_cache_unlock(cache);
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	408 *unlock = FALSE;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	409 }
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	410
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	411 mail_cache_file_close(cache);
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	412 cache->fd = fd;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	413
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	414 if (cache->file_cache != NULL)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	415 file_cache_set_fd(cache->file_cache, cache->fd);
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	416
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	417 if (mail_cache_map(cache, 0, 0) < 0)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	418 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	419 if (mail_cache_header_fields_read(cache) < 0)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	420 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	421
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	422 cache->need_compress_file_seq = 0;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	423 return 0;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	424 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	425
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	426 int mail_cache_compress(struct mail_cache *cache,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	427 struct mail_index_transaction *trans)
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	428 {
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	429 bool unlock = FALSE;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	430 int ret;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	431
3243 40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	432 if (MAIL_INDEX_IS_IN_MEMORY(cache->index))
40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	433 return 0;
40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	434
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	435 if (cache->index->lock_method == FILE_LOCK_METHOD_DOTLOCK) {
2949 05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	436 /* we're using dotlocking, cache file creation itself creates
05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	437 the dotlock file we need. */
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	438 return mail_cache_compress_locked(cache, trans, &unlock);
2949 05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	439 }
05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	440
5947 7ebe0593f488 Don't create cache file until something is actually being added to it. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	441 switch (mail_cache_lock(cache, FALSE)) {
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	442 case -1:
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	443 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	444 case 0:
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	445 /* couldn't lock, either it's broken or doesn't exist.
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	446 just start creating it. */
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	447 return mail_cache_compress_locked(cache, trans, &unlock);
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	448 default:
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	449 /* locking succeeded. */
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	450 unlock = TRUE;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	451 ret = mail_cache_compress_locked(cache, trans, &unlock);
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	452 if (unlock) {
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	453 if (mail_cache_unlock(cache) < 0)
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	454 ret = -1;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	455 }
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	456 return ret;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	457 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	458 }
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	459
3863 55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool. Timo Sirainen <tss@iki.fi> parents: 3819 diff changeset	460 bool mail_cache_need_compress(struct mail_cache *cache)
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	461 {
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	462 return cache->need_compress_file_seq != 0;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	463 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-index/mail-cache-compress.c @ 6677:90bcf480312d HEAD