dovecot/original-hg/dovecot-1.2: src/lib-index/mail-cache-compress.c annotate

annotate src/lib-index/mail-cache-compress.c @ 8999:afc1b0ef120d HEAD

When :MAILBOXDIR= was empty, we might have appended extra '/' to it, which caused problems.

author	Timo Sirainen <tss@iki.fi>
date	Thu, 30 Apr 2009 20:00:09 -0400
parents	b9faf4db2a9f
children	0f2f9e207644

rev	line source
8590 b9faf4db2a9f Updated copyright notices to include year 2009. Timo Sirainen <tss@iki.fi> parents: 8292 diff changeset	1 /* Copyright (c) 2003-2009 Dovecot authors, see the included COPYING file */
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	2
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	3 #include "lib.h"
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	4 #include "array.h"
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	5 #include "ostream.h"
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	6 #include "nfs-workarounds.h"
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	7 #include "read-full.h"
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	8 #include "close-keep-errno.h"
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	9 #include "file-dotlock.h"
2866 bf1e718e7370 Cache file works now with mmap_disable=yes. Still needs a few optimizations. Timo Sirainen <tss@iki.fi> parents: 2853 diff changeset	10 #include "file-cache.h"
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	11 #include "file-set-size.h"
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	12 #include "mail-cache-private.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	13
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	14 #include <sys/stat.h>
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	15
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	16 struct mail_cache_copy_context {
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	17 struct mail_cache *cache;
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	18
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	19 buffer_t buffer, field_seen;
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	20 ARRAY_DEFINE(bitmask_pos, unsigned int);
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	21 uint32_t *field_file_map;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	22
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	23 uint8_t field_seen_value;
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	24 bool new_msg;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	25 };
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	26
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	27 static void
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	28 mail_cache_merge_bitmask(struct mail_cache_copy_context *ctx,
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	29 const struct mail_cache_iterate_field *field)
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	30 {
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	31 unsigned char *dest;
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	32 unsigned int i, *pos;
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	33
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	34 pos = array_idx_modifiable(&ctx->bitmask_pos, field->field_idx);
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	35 if (*pos == 0) {
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	36 /* we decided to drop this field */
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	37 return;
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	38 }
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	39
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	40 dest = buffer_get_space_unsafe(ctx->buffer, *pos, field->size);
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	41 for (i = 0; i < field->size; i++)
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	42 dest[i] \|= ((const unsigned char*)field->data)[i];
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	43 }
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	44
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	45 static void
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	46 mail_cache_compress_field(struct mail_cache_copy_context *ctx,
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	47 const struct mail_cache_iterate_field *field)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	48 {
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	49 struct mail_cache_field *cache_field;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	50 enum mail_cache_decision_type dec;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	51 uint32_t file_field_idx, size32;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	52 uint8_t *field_seen;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	53
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	54 file_field_idx = ctx->field_file_map[field->field_idx];
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	55 if (file_field_idx == (uint32_t)-1)
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	56 return;
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	57
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	58 cache_field = &ctx->cache->fields[field->field_idx].field;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	59
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	60 field_seen = buffer_get_space_unsafe(ctx->field_seen,
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	61 field->field_idx, 1);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	62 if (*field_seen == ctx->field_seen_value) {
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	63 /* duplicate */
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	64 if (cache_field->type == MAIL_CACHE_FIELD_BITMASK)
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	65 mail_cache_merge_bitmask(ctx, field);
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	66 return;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	67 }
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	68 *field_seen = ctx->field_seen_value;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	69
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	70 dec = cache_field->decision & ~MAIL_CACHE_DECISION_FORCED;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	71 if (ctx->new_msg) {
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	72 if (dec == MAIL_CACHE_DECISION_NO)
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	73 return;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	74 } else {
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	75 if (dec != MAIL_CACHE_DECISION_YES)
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	76 return;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	77 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	78
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	79 buffer_append(ctx->buffer, &file_field_idx, sizeof(file_field_idx));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	80
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	81 if (cache_field->field_size == (unsigned int)-1) {
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	82 size32 = (uint32_t)field->size;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	83 buffer_append(ctx->buffer, &size32, sizeof(size32));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	84 }
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	85
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	86 if (cache_field->type == MAIL_CACHE_FIELD_BITMASK) {
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	87 /* remember the position in case we need to update it */
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	88 unsigned int pos = ctx->buffer->used;
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	89
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	90 array_idx_set(&ctx->bitmask_pos, field->field_idx, &pos);
615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	91 }
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	92 buffer_append(ctx->buffer, field->data, field->size);
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	93 if ((field->size & 3) != 0)
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	94 buffer_append_zero(ctx->buffer, 4 - (field->size & 3));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	95 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	96
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	97 static uint32_t
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	98 get_next_file_seq(struct mail_cache cache, struct mail_index_view view)
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	99 {
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	100 const struct mail_index_ext *ext;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	101 uint32_t file_seq;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	102
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	103 ext = mail_index_view_get_ext(view, cache->ext_id);
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	104 file_seq = ext != NULL ? ext->reset_id + 1 : (uint32_t)ioloop_time;
5994 8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	105
8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	106 if (cache->hdr != NULL && file_seq <= cache->hdr->file_seq)
8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	107 file_seq = cache->hdr->file_seq + 1;
8964d1b2b78b Make sure the file_seq always grows. Timo Sirainen <tss@iki.fi> parents: 5993 diff changeset	108
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	109 return file_seq != 0 ? file_seq : 1;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	110 }
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	111
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	112 static void
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	113 mail_cache_compress_get_fields(struct mail_cache_copy_context *ctx,
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	114 unsigned int used_fields_count)
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	115 {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	116 struct mail_cache *cache = ctx->cache;
6297 3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	117 struct mail_cache_field *field;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	118 unsigned int i, j, idx;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	119
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	120 /* Make mail_cache_header_fields_get() return the fields in
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	121 the same order as we saved them. */
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	122 memcpy(cache->field_file_map, ctx->field_file_map,
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	123 sizeof(uint32_t) * cache->fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	124
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	125 /* reverse mapping */
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	126 cache->file_fields_count = used_fields_count;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	127 i_free(cache->file_field_map);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	128 cache->file_field_map = used_fields_count == 0 ? NULL :
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	129 i_new(unsigned int, used_fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	130 for (i = j = 0; i < cache->fields_count; i++) {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	131 idx = cache->field_file_map[i];
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	132 if (idx != (uint32_t)-1) {
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	133 i_assert(idx < used_fields_count &&
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	134 cache->file_field_map[idx] == 0);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	135 cache->file_field_map[idx] = i;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	136 j++;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	137 }
6297 3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	138
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	139 /* change permanent decisions to temporary decisions.
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	140 if they're still permanent they'll get updated later. */
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	141 field = &cache->fields[i].field;
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	142 if (field->decision == MAIL_CACHE_DECISION_YES)
3f6fadbe6888 Change "permanent" cache decisions to "temporary" when compressing. They'll Timo Sirainen <tss@iki.fi> parents: 6296 diff changeset	143 field->decision = MAIL_CACHE_DECISION_TEMP;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	144 }
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	145 i_assert(j == used_fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	146
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	147 buffer_set_used_size(ctx->buffer, 0);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	148 mail_cache_header_fields_get(cache, ctx->buffer);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	149 }
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	150
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	151 static int
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	152 mail_cache_copy(struct mail_cache cache, struct mail_index_transaction trans,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	153 int fd, uint32_t *file_seq_r,
5733 4b5ce642957d Declare a global uint32_t array type. Timo Sirainen <tss@iki.fi> parents: 5652 diff changeset	154 ARRAY_TYPE(uint32_t) *ext_offsets)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	155 {
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	156 struct mail_cache_copy_context ctx;
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	157 struct mail_cache_lookup_iterate_ctx iter;
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	158 struct mail_cache_iterate_field field;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	159 struct mail_index_view *view;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	160 struct mail_cache_view *cache_view;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	161 const struct mail_index_header *idx_hdr;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	162 struct mail_cache_header hdr;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	163 struct mail_cache_record cache_rec;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	164 struct ostream *output;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	165 uint32_t message_count, seq, first_new_seq, ext_offset;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	166 unsigned int i, used_fields_count, orig_fields_count;
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	167 time_t max_drop_time;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	168
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	169 view = mail_index_transaction_get_view(trans);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	170 cache_view = mail_cache_view_open(cache, view);
6161 c62f7ee79446 Split o_stream_create_file() to _create_fd() and _create_fd_file(). Timo Sirainen <tss@iki.fi> parents: 6142 diff changeset	171 output = o_stream_create_fd_file(fd, 0, FALSE);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	172
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	173 memset(&hdr, 0, sizeof(hdr));
2276 5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code - Timo Sirainen <tss@iki.fi> parents: 2275 diff changeset	174 hdr.version = MAIL_CACHE_VERSION;
3819 61f0e1fe9b55 Moved uoff_t and time_t compatibility checks to cache file, since they're Timo Sirainen <tss@iki.fi> parents: 3627 diff changeset	175 hdr.compat_sizeof_uoff_t = sizeof(uoff_t);
5819 4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5737 diff changeset	176 hdr.indexid = cache->index->indexid;
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	177 hdr.file_seq = get_next_file_seq(cache, view);
2339 406692edc12d Cache fixes. Decisions are saved again. Timo Sirainen <tss@iki.fi> parents: 2327 diff changeset	178 o_stream_send(output, &hdr, sizeof(hdr));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	179
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	180 memset(&ctx, 0, sizeof(ctx));
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	181 ctx.cache = cache;
2708 f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where Timo Sirainen <tss@iki.fi> parents: 2700 diff changeset	182 ctx.buffer = buffer_create_dynamic(default_pool, 4096);
f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where Timo Sirainen <tss@iki.fi> parents: 2700 diff changeset	183 ctx.field_seen = buffer_create_dynamic(default_pool, 64);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	184 ctx.field_seen_value = 0;
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	185 ctx.field_file_map = t_new(uint32_t, cache->fields_count);
5737 615a8c31ff47 Code cleanup / optimization Timo Sirainen <tss@iki.fi> parents: 5736 diff changeset	186 t_array_init(&ctx.bitmask_pos, 32);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	187
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	188 /* @UNSAFE: drop unused fields and create a field mapping for
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	189 used fields */
6841 0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	190 idx_hdr = mail_index_get_header(view);
6678 6601ee607b6f Don't drop all fields if day_stamp is still 0 in index header. Timo Sirainen <tss@iki.fi> parents: 6677 diff changeset	191 max_drop_time = idx_hdr->day_stamp == 0 ? 0 :
6601ee607b6f Don't drop all fields if day_stamp is still 0 in index header. Timo Sirainen <tss@iki.fi> parents: 6677 diff changeset	192 idx_hdr->day_stamp - MAIL_CACHE_FIELD_DROP_SECS;
6717 02014f5b0068 Don't write fields to cache file with last_used=0. If fields were just Timo Sirainen <tss@iki.fi> parents: 6696 diff changeset	193
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	194 orig_fields_count = cache->fields_count;
6696 68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	195 if (cache->file_fields_count == 0) {
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	196 /* creating the initial cache file. add all fields. */
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	197 for (i = 0; i < orig_fields_count; i++)
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	198 ctx.field_file_map[i] = i;
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	199 used_fields_count = i;
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	200 } else {
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	201 for (i = used_fields_count = 0; i < orig_fields_count; i++) {
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	202 struct mail_cache_field_private *priv =
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	203 &cache->fields[i];
7068 180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	204 enum mail_cache_decision_type dec =
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	205 priv->field.decision;
7068 180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	206
180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	207 /* if the decision isn't forced and this field hasn't
180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	208 been accessed for a while, drop it */
180e219aab9a Don't drop fields whose decision has been forced. Timo Sirainen <tss@iki.fi> parents: 7043 diff changeset	209 if ((dec & MAIL_CACHE_DECISION_FORCED) == 0 &&
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	210 (time_t)priv->last_used < max_drop_time &&
1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	211 !priv->adding) {
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	212 dec = MAIL_CACHE_DECISION_NO;
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	213 priv->field.decision = dec;
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	214 }
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	215
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	216 /* drop all fields we don't want */
f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	217 if ((dec & ~MAIL_CACHE_DECISION_FORCED) ==
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	218 MAIL_CACHE_DECISION_NO && !priv->adding) {
1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	219 priv->used = FALSE;
1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	220 priv->last_used = 0;
7105 f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	221 }
f0ad529ac9ea Fixes to handling when fields are dropped from cache file. Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	222
7561 1a58b18652a6 Avoid using shadow variables. Unfortunately -Wshadow also complains about Timo Sirainen <tss@iki.fi> parents: 7105 diff changeset	223 ctx.field_file_map[i] = !priv->used ?
6696 68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	224 (uint32_t)-1 : used_fields_count++;
68adf46a6b3f When adding new fields to the cache file, add all the fields we currently Timo Sirainen <tss@iki.fi> parents: 6686 diff changeset	225 }
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	226 }
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	227
6841 0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	228 /* get sequence of first message which doesn't need its temp fields
0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	229 removed. */
0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	230 first_new_seq = mail_cache_get_first_new_seq(view);
0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	231 message_count = mail_index_view_get_messages_count(view);
0c970b3493ac mail_cache_field_want_add(): Return TRUE for temp fields only if we're Timo Sirainen <tss@iki.fi> parents: 6839 diff changeset	232
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	233 i_array_init(ext_offsets, message_count);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	234 for (seq = 1; seq <= message_count; seq++) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	235 if (mail_index_transaction_is_expunged(trans, seq)) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	236 (void)array_append_space(ext_offsets);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	237 continue;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	238 }
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	239
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	240 ctx.new_msg = seq >= first_new_seq;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	241 buffer_set_used_size(ctx.buffer, 0);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	242
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	243 if (++ctx.field_seen_value == 0) {
4451 1a35d53c18fc Array API redesigned to work using unions. It now provides type safety Timo Sirainen <tss@iki.fi> parents: 4247 diff changeset	244 memset(buffer_get_modifiable_data(ctx.field_seen, NULL),
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	245 0, buffer_get_size(ctx.field_seen));
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	246 ctx.field_seen_value++;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	247 }
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	248
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	249 memset(&cache_rec, 0, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	250 buffer_append(ctx.buffer, &cache_rec, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	251
5736 b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	252 mail_cache_lookup_iter_init(cache_view, seq, &iter);
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	253 while (mail_cache_lookup_iter_next(&iter, &field) > 0)
b3ed52bbeca7 Changed mail_cache_foreach() to be an iterator: mail_cache_lookup_iter_() Timo Sirainen <tss@iki.fi>* parents: 5733 diff changeset	254 mail_cache_compress_field(&ctx, &field);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	255
2299 ef0cd18d674c Fixed compression. Timo Sirainen <tss@iki.fi> parents: 2298 diff changeset	256 cache_rec.size = buffer_get_used_size(ctx.buffer);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	257 if (cache_rec.size == sizeof(cache_rec)) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	258 /* nothing cached */
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	259 ext_offset = 0;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	260 } else {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	261 ext_offset = output->offset;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	262 buffer_write(ctx.buffer, 0, &cache_rec,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	263 sizeof(cache_rec));
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	264 o_stream_send(output, ctx.buffer->data, cache_rec.size);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	265 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	266
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	267 array_append(ext_offsets, &ext_offset, 1);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	268 }
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	269 i_assert(orig_fields_count == cache->fields_count);
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	270
6296 205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	271 hdr.field_header_offset = mail_index_uint32_to_offset(output->offset);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	272 mail_cache_compress_get_fields(&ctx, used_fields_count);
205ee38f10d1 Drop fields that haven't been used for 30 days when compressing. Timo Sirainen <tss@iki.fi> parents: 6275 diff changeset	273 o_stream_send(output, ctx.buffer->data, ctx.buffer->used);
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	274
2276 5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code - Timo Sirainen <tss@iki.fi> parents: 2275 diff changeset	275 hdr.used_file_size = output->offset;
6414 a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6319 diff changeset	276 buffer_free(&ctx.buffer);
a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6319 diff changeset	277 buffer_free(&ctx.field_seen);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	278
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	279 o_stream_seek(output, 0);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	280 o_stream_send(output, &hdr, sizeof(hdr));
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	281
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	282 mail_cache_view_close(cache_view);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	283
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	284 if (o_stream_flush(output) < 0) {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	285 errno = output->stream_errno;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	286 mail_cache_set_syscall_error(cache, "o_stream_flush()");
4070 71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of Timo Sirainen <tss@iki.fi> parents: 3879 diff changeset	287 o_stream_destroy(&output);
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	288 array_free(ext_offsets);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	289 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	290 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	291
2299 ef0cd18d674c Fixed compression. Timo Sirainen <tss@iki.fi> parents: 2298 diff changeset	292 if (hdr.used_file_size < MAIL_CACHE_INITIAL_SIZE) {
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	293 /* grow the file some more. doesn't matter if it fails */
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	294 (void)file_set_size(fd, MAIL_CACHE_INITIAL_SIZE);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	295 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	296
4070 71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of Timo Sirainen <tss@iki.fi> parents: 3879 diff changeset	297 o_stream_destroy(&output);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	298
6735 a881755b3db8 We don't really care if we lost a cache file or if it gets broken, so don't Timo Sirainen <tss@iki.fi> parents: 6717 diff changeset	299 if (cache->index->nfs_flush) {
4918 2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	300 if (fdatasync(fd) < 0) {
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	301 mail_cache_set_syscall_error(cache, "fdatasync()");
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	302 array_free(ext_offsets);
4918 2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	303 return -1;
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	304 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	305 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	306
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	307 *file_seq_r = hdr.file_seq;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	308 return 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	309 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	310
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	311 static int mail_cache_compress_has_file_changed(struct mail_cache *cache)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	312 {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	313 struct mail_cache_header hdr;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	314 unsigned int i;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	315 int fd, ret;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	316
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	317 for (i = 0;; i++) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	318 fd = nfs_safe_open(cache->filepath, O_RDONLY);
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	319 if (fd == -1) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	320 if (errno == ENOENT)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	321 return 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	322
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	323 mail_cache_set_syscall_error(cache, "open()");
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	324 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	325 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	326
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	327 ret = read_full(fd, &hdr, sizeof(hdr));
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	328 close_keep_errno(fd);
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	329
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	330 if (ret >= 0) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	331 if (ret == 0)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	332 return 0;
5993 c66c80b23f95 need_compress_file_seq=0 nowadays when file doesn't exist, not -1. Timo Sirainen <tss@iki.fi> parents: 5972 diff changeset	333 if (cache->need_compress_file_seq == 0) {
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	334 /* previously it didn't exist */
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	335 return 1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	336 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	337 return hdr.file_seq != cache->need_compress_file_seq;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	338 } else if (errno != ESTALE \|\| i >= NFS_ESTALE_RETRY_COUNT) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	339 mail_cache_set_syscall_error(cache, "read()");
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	340 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	341 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	342 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	343 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	344
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	345 static int mail_cache_compress_locked(struct mail_cache *cache,
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	346 struct mail_index_transaction *trans,
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	347 bool *unlock)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	348 {
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	349 struct dotlock *dotlock;
6823 ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	350 struct stat st;
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	351 mode_t old_mask;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	352 uint32_t file_seq, old_offset;
5733 4b5ce642957d Declare a global uint32_t array type. Timo Sirainen <tss@iki.fi> parents: 5652 diff changeset	353 ARRAY_TYPE(uint32_t) ext_offsets;
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	354 const uint32_t *offsets;
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	355 unsigned int i, count;
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	356 int fd, ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	357
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	358 /* get the latest info on fields */
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	359 if (mail_cache_header_fields_read(cache) < 0)
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	360 return -1;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	361
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	362 old_mask = umask(cache->index->mode ^ 0666);
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	363 fd = file_dotlock_open(&cache->dotlock_settings, cache->filepath,
8188 2db48458d73b Mail cache compression: If we can't get lock immediately, do it later. Timo Sirainen <tss@iki.fi> parents: 8187 diff changeset	364 DOTLOCK_CREATE_FLAG_NONBLOCK, &dotlock);
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	365 umask(old_mask);
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	366
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	367 if (fd == -1) {
8188 2db48458d73b Mail cache compression: If we can't get lock immediately, do it later. Timo Sirainen <tss@iki.fi> parents: 8187 diff changeset	368 if (errno != EAGAIN)
2db48458d73b Mail cache compression: If we can't get lock immediately, do it later. Timo Sirainen <tss@iki.fi> parents: 8187 diff changeset	369 mail_cache_set_syscall_error(cache, "file_dotlock_open()");
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	370 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	371 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	372
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	373 if ((ret = mail_cache_compress_has_file_changed(cache)) != 0) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	374 if (ret < 0)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	375 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	376
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	377 /* was just compressed, forget this */
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	378 cache->need_compress_file_seq = 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	379 file_dotlock_delete(&dotlock);
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	380
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	381 if (*unlock) {
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	382 (void)mail_cache_unlock(cache);
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	383 *unlock = FALSE;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	384 }
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	385
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	386 return mail_cache_reopen(cache);
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	387 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	388
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	389 if (cache->index->gid != (gid_t)-1 &&
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	390 fchown(fd, (uid_t)-1, cache->index->gid) < 0) {
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	391 mail_cache_set_syscall_error(cache, "fchown()");
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	392 file_dotlock_delete(&dotlock);
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	393 return -1;
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	394 }
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	395
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	396 if (mail_cache_copy(cache, trans, fd, &file_seq, &ext_offsets) < 0) {
8292 9ed4ecd4a866 Fixes to handling "out of disk space/quota" write failures. Timo Sirainen <tss@iki.fi> parents: 8188 diff changeset	397 /* the fields may have been updated in memory already.
9ed4ecd4a866 Fixes to handling "out of disk space/quota" write failures. Timo Sirainen <tss@iki.fi> parents: 8188 diff changeset	398 reverse those changes by re-reading them from file. */
9ed4ecd4a866 Fixes to handling "out of disk space/quota" write failures. Timo Sirainen <tss@iki.fi> parents: 8188 diff changeset	399 if (mail_cache_header_fields_read(cache) < 0)
9ed4ecd4a866 Fixes to handling "out of disk space/quota" write failures. Timo Sirainen <tss@iki.fi> parents: 8188 diff changeset	400 return -1;
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	401 (void)file_dotlock_delete(&dotlock);
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	402 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	403 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	404
6823 ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	405 if (fstat(fd, &st) < 0) {
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	406 mail_cache_set_syscall_error(cache, "fstat()");
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	407 (void)file_dotlock_delete(&dotlock);
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	408 return -1;
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	409 }
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	410
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	411 if (file_dotlock_replace(&dotlock,
9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	412 DOTLOCK_REPLACE_FLAG_DONT_CLOSE_FD) < 0) {
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	413 mail_cache_set_syscall_error(cache,
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	414 "file_dotlock_replace()");
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	415 (void)close(fd);
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	416 array_free(&ext_offsets);
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	417 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	418 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	419
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	420 /* once we're sure that the compression was successful,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	421 update the offsets */
7563 6de1aed24ce5 Added mail_index_ext_reset_inc() to atomically increase extension's Timo Sirainen <tss@iki.fi> parents: 7561 diff changeset	422 mail_index_ext_reset(trans, cache->ext_id, file_seq, TRUE);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	423 offsets = array_get(&ext_offsets, &count);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	424 for (i = 0; i < count; i++) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	425 if (offsets[i] != 0) {
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	426 mail_index_update_ext(trans, i + 1, cache->ext_id,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	427 &offsets[i], &old_offset);
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	428 }
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	429 }
6319 c2c8ee636157 Minor memory usage optimizations. Timo Sirainen <tss@iki.fi> parents: 6297 diff changeset	430 array_free(&ext_offsets);
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	431
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	432 if (*unlock) {
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	433 (void)mail_cache_unlock(cache);
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	434 *unlock = FALSE;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	435 }
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	436
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	437 mail_cache_file_close(cache);
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	438 cache->fd = fd;
6823 ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	439 cache->st_ino = st.st_ino;
ee1c4c3d5e5a Update cache->st_ino and st_dev after compressing. Timo Sirainen <tss@iki.fi> parents: 6735 diff changeset	440 cache->st_dev = st.st_dev;
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	441 cache->field_header_write_pending = FALSE;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	442
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	443 if (cache->file_cache != NULL)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	444 file_cache_set_fd(cache->file_cache, cache->fd);
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	445
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	446 if (mail_cache_map(cache, 0, 0) < 0)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	447 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	448 if (mail_cache_header_fields_read(cache) < 0)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	449 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	450
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	451 cache->need_compress_file_seq = 0;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	452 return 0;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	453 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	454
5652 9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	455 int mail_cache_compress(struct mail_cache *cache,
9404d2c41cb2 When compressing cache file, update the cache file offsets using the Timo Sirainen <tss@iki.fi> parents: 5027 diff changeset	456 struct mail_index_transaction *trans)
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	457 {
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	458 bool unlock = FALSE;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	459 int ret;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	460
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	461 i_assert(!cache->compressing);
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	462
6686 610f3d9813b5 Added MAIL_INDEX_OPEN_FLAG_READONLY. Timo Sirainen <tss@iki.fi> parents: 6678 diff changeset	463 if (MAIL_INDEX_IS_IN_MEMORY(cache->index) \|\| cache->index->readonly)
3243 40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	464 return 0;
40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	465
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	466 if (cache->index->lock_method == FILE_LOCK_METHOD_DOTLOCK) {
2949 05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	467 /* we're using dotlocking, cache file creation itself creates
05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	468 the dotlock file we need. */
6829 dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	469 if (!MAIL_CACHE_IS_UNUSABLE(cache)) {
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	470 mail_index_flush_read_cache(cache->index,
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	471 cache->filepath, cache->fd,
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	472 FALSE);
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6823 diff changeset	473 }
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	474 } else {
8188 2db48458d73b Mail cache compression: If we can't get lock immediately, do it later. Timo Sirainen <tss@iki.fi> parents: 8187 diff changeset	475 switch (mail_cache_try_lock(cache)) {
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	476 case -1:
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	477 return -1;
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	478 case 0:
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	479 /* couldn't lock, either it's broken or doesn't exist.
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	480 just start creating it. */
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	481 break;
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	482 default:
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	483 /* locking succeeded. */
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	484 unlock = TRUE;
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	485 }
2949 05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	486 }
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	487 cache->compressing = TRUE;
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	488 ret = mail_cache_compress_locked(cache, trans, &unlock);
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	489 cache->compressing = FALSE;
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	490 if (unlock) {
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	491 if (mail_cache_unlock(cache) < 0)
f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	492 ret = -1;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	493 }
8187 f1203dce2d8c Mail cache: lock_method=dotlock could have caused a process to deadlock with itself. Timo Sirainen <tss@iki.fi> parents: 7563 diff changeset	494 return ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	495 }
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	496
3863 55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool. Timo Sirainen <tss@iki.fi> parents: 3819 diff changeset	497 bool mail_cache_need_compress(struct mail_cache *cache)
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	498 {
6686 610f3d9813b5 Added MAIL_INDEX_OPEN_FLAG_READONLY. Timo Sirainen <tss@iki.fi> parents: 6678 diff changeset	499 return cache->need_compress_file_seq != 0 &&
610f3d9813b5 Added MAIL_INDEX_OPEN_FLAG_READONLY. Timo Sirainen <tss@iki.fi> parents: 6678 diff changeset	500 !cache->index->readonly;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	501 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-index/mail-cache-compress.c @ 8999:afc1b0ef120d HEAD