dovecot/original-hg/dovecot-1.2: src/lib-index/mail-cache-compress.c annotate

annotate src/lib-index/mail-cache-compress.c @ 5027:787ead060fe3 HEAD

debug: Don't bother warning about cache compressions. It has lived its usefulness.

author	Timo Sirainen <tss@iki.fi>
date	Thu, 18 Jan 2007 17:36:11 +0200
parents	2f9173e103fd
children	9404d2c41cb2

rev	line source
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	1 /* Copyright (C) 2003-2004 Timo Sirainen */
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	2
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	3 #include "lib.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	4 #include "buffer.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	5 #include "ostream.h"
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	6 #include "nfs-workarounds.h"
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	7 #include "read-full.h"
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	8 #include "close-keep-errno.h"
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	9 #include "file-dotlock.h"
2866 bf1e718e7370 Cache file works now with mmap_disable=yes. Still needs a few optimizations. Timo Sirainen <tss@iki.fi> parents: 2853 diff changeset	10 #include "file-cache.h"
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	11 #include "file-set-size.h"
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	12 #include "mail-cache-private.h"
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	13
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	14 #include <sys/stat.h>
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	15
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	16 struct mail_cache_copy_context {
3863 55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool. Timo Sirainen <tss@iki.fi> parents: 3819 diff changeset	17 bool new_msg;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	18 buffer_t buffer, field_seen;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	19 uint8_t field_seen_value;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	20 };
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	21
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	22 static void mail_cache_merge_bitmask(struct mail_cache cache, buffer_t buffer,
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	23 uint32_t field, const void *data,
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	24 size_t data_size)
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	25 {
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	26 void *buf_data;
2620 f4f793129402 fix Timo Sirainen <tss@iki.fi> parents: 2619 diff changeset	27 uint32_t buf_field;
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	28 unsigned int i, buf_data_size;
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	29 size_t pos, buf_size;
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	30
4451 1a35d53c18fc Array API redesigned to work using unions. It now provides type safety Timo Sirainen <tss@iki.fi> parents: 4247 diff changeset	31 buf_data = buffer_get_modifiable_data(buffer, &buf_size);
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	32 for (pos = sizeof(struct mail_cache_record); pos < buf_size; ) {
2620 f4f793129402 fix Timo Sirainen <tss@iki.fi> parents: 2619 diff changeset	33 buf_field = ((uint32_t )PTR_OFFSET(buf_data, pos));
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	34 pos += sizeof(uint32_t);
2700 5b07b6fd9bf8 Added asserts Timo Sirainen <tss@iki.fi> parents: 2671 diff changeset	35 i_assert(buf_field < cache->fields_count);
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	36
2623 0157839e3c78 bitmask merging code was corrupting the cache file Timo Sirainen <tss@iki.fi> parents: 2620 diff changeset	37 buf_data_size = cache->fields[buf_field].field.field_size;
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	38 if (buf_data_size == (unsigned int)-1) {
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	39 buf_data_size =
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	40 ((uint32_t )PTR_OFFSET(buf_data, pos));
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	41 pos += sizeof(uint32_t);
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	42 }
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	43
2620 f4f793129402 fix Timo Sirainen <tss@iki.fi> parents: 2619 diff changeset	44 if (buf_field == field) {
2700 5b07b6fd9bf8 Added asserts Timo Sirainen <tss@iki.fi> parents: 2671 diff changeset	45 /* @UNSAFE: found it, do the merging */
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	46 unsigned char *dest = PTR_OFFSET(buf_data, pos);
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	47
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	48 i_assert(buf_data_size == data_size);
2700 5b07b6fd9bf8 Added asserts Timo Sirainen <tss@iki.fi> parents: 2671 diff changeset	49 i_assert(pos + buf_data_size <= buf_size);
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	50 for (i = 0; i < buf_data_size; i++)
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	51 dest[i] \|= ((const unsigned char*)data)[i];
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	52 break;
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	53 }
2950 30110ac226a7 mail_cache_merge_bitmask() was broken, cache compression could have crashed Timo Sirainen <tss@iki.fi> parents: 2949 diff changeset	54 pos += (buf_data_size + 3) & ~3;
2700 5b07b6fd9bf8 Added asserts Timo Sirainen <tss@iki.fi> parents: 2671 diff changeset	55 i_assert(pos <= buf_size);
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	56 }
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	57 }
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	58
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	59 static int
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	60 mail_cache_compress_callback(struct mail_cache_view *view, uint32_t field,
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	61 const void data, size_t data_size, void context)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	62 {
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	63 struct mail_cache_copy_context *ctx = context;
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	64 struct mail_cache_field *cache_field;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	65 enum mail_cache_decision_type dec;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	66 uint8_t *field_seen;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	67 uint32_t size32;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	68
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	69 cache_field = &view->cache->fields[field].field;
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	70
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	71 field_seen = buffer_get_space_unsafe(ctx->field_seen, field, 1);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	72 if (*field_seen == ctx->field_seen_value) {
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	73 /* duplicate */
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	74 if (cache_field->type == MAIL_CACHE_FIELD_BITMASK) {
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	75 mail_cache_merge_bitmask(view->cache, ctx->buffer,
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	76 field, data, data_size);
87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	77 }
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	78 return 1;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	79 }
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	80 *field_seen = ctx->field_seen_value;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	81
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	82 dec = cache_field->decision & ~MAIL_CACHE_DECISION_FORCED;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	83 if (ctx->new_msg) {
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	84 if (dec == MAIL_CACHE_DECISION_NO)
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	85 return 1;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	86 } else {
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	87 if (dec != MAIL_CACHE_DECISION_YES)
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	88 return 1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	89 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	90
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	91 buffer_append(ctx->buffer, &field, sizeof(field));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	92
2555 87af1f39a69e Added support for bitmask type. Timo Sirainen <tss@iki.fi> parents: 2407 diff changeset	93 if (cache_field->field_size == (unsigned int)-1) {
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	94 size32 = (uint32_t)data_size;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	95 buffer_append(ctx->buffer, &size32, sizeof(size32));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	96 }
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	97
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	98 buffer_append(ctx->buffer, data, data_size);
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	99 if ((data_size & 3) != 0)
3617 904849549eac Removed null4 and replaced the few uses of it with buffer_append_zero(). Timo Sirainen <tss@iki.fi> parents: 3551 diff changeset	100 buffer_append_zero(ctx->buffer, 4 - (data_size & 3));
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	101 return 1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	102 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	103
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	104 static uint32_t
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	105 get_next_file_seq(struct mail_cache cache, struct mail_index_view view)
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	106 {
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	107 const struct mail_index_ext *ext;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	108 uint32_t file_seq;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	109
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	110 ext = mail_index_view_get_ext(view, cache->ext_id);
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	111 file_seq = ext != NULL ? ext->reset_id + 1 : (uint32_t)ioloop_time;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	112 return file_seq != 0 ? file_seq : 1;
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	113 }
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	114
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	115 static int
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	116 mail_cache_copy(struct mail_cache cache, struct mail_index_view view, int fd)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	117 {
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	118 struct mail_cache_copy_context ctx;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	119 struct mail_cache_view *cache_view;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	120 struct mail_index_transaction *t;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	121 const struct mail_index_header *idx_hdr;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	122 struct mail_cache_header hdr;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	123 struct mail_cache_record cache_rec;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	124 struct ostream *output;
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	125 buffer_t *buffer;
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	126 uint32_t i, message_count, seq, first_new_seq, old_offset;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	127 uoff_t offset;
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	128
3551 65dc45982df9 Removed fcntl() text from some locking error messages, because if Timo Sirainen <tss@iki.fi> parents: 3243 diff changeset	129 /* get sequence of first message which doesn't need its temp fields
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	130 removed. */
2892 62d53b49110d Changed mail_index_get_header() to return the header as return value because Timo Sirainen <tss@iki.fi> parents: 2879 diff changeset	131 idx_hdr = mail_index_get_header(view);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	132 if (idx_hdr->day_first_uid[7] == 0) {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	133 first_new_seq = 1;
2904 642ba9cd3dac Removed view->messages_count, view->hdr.messages_count is enough. Also fixes Timo Sirainen <tss@iki.fi> parents: 2892 diff changeset	134 message_count = mail_index_view_get_messages_count(view);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	135 } else {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	136 if (mail_index_lookup_uid_range(view, idx_hdr->day_first_uid[7],
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	137 (uint32_t)-1, &first_new_seq,
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	138 &message_count) < 0)
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	139 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	140 if (first_new_seq == 0)
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	141 first_new_seq = message_count+1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	142 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	143
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	144 cache_view = mail_cache_view_open(cache, view);
2877 3cb483d565a6 Allow specifying to transaction if it's external. Make mail saving Timo Sirainen <tss@iki.fi> parents: 2866 diff changeset	145 t = mail_index_transaction_begin(view, FALSE, TRUE);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	146 output = o_stream_create_file(fd, default_pool, 0, FALSE);
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	147
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	148 memset(&hdr, 0, sizeof(hdr));
2276 5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code - Timo Sirainen <tss@iki.fi> parents: 2275 diff changeset	149 hdr.version = MAIL_CACHE_VERSION;
3819 61f0e1fe9b55 Moved uoff_t and time_t compatibility checks to cache file, since they're Timo Sirainen <tss@iki.fi> parents: 3627 diff changeset	150 hdr.compat_sizeof_uoff_t = sizeof(uoff_t);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	151 hdr.indexid = idx_hdr->indexid;
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	152 hdr.file_seq = get_next_file_seq(cache, view);
2339 406692edc12d Cache fixes. Decisions are saved again. Timo Sirainen <tss@iki.fi> parents: 2327 diff changeset	153 o_stream_send(output, &hdr, sizeof(hdr));
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	154
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	155 memset(&ctx, 0, sizeof(ctx));
2708 f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where Timo Sirainen <tss@iki.fi> parents: 2700 diff changeset	156 ctx.buffer = buffer_create_dynamic(default_pool, 4096);
f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where Timo Sirainen <tss@iki.fi> parents: 2700 diff changeset	157 ctx.field_seen = buffer_create_dynamic(default_pool, 64);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	158 ctx.field_seen_value = 0;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	159
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	160 mail_index_ext_reset(t, cache->ext_id, hdr.file_seq);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	161
2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	162 for (seq = 1; seq <= message_count; seq++) {
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	163 ctx.new_msg = seq >= first_new_seq;
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	164 buffer_set_used_size(ctx.buffer, 0);
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	165
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	166 if (++ctx.field_seen_value == 0) {
4451 1a35d53c18fc Array API redesigned to work using unions. It now provides type safety Timo Sirainen <tss@iki.fi> parents: 4247 diff changeset	167 memset(buffer_get_modifiable_data(ctx.field_seen, NULL),
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	168 0, buffer_get_size(ctx.field_seen));
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	169 ctx.field_seen_value++;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	170 }
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	171
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	172 memset(&cache_rec, 0, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	173 buffer_append(ctx.buffer, &cache_rec, sizeof(cache_rec));
5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	174
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	175 (void)mail_cache_foreach(cache_view, seq,
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	176 mail_cache_compress_callback, &ctx);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	177
2299 ef0cd18d674c Fixed compression. Timo Sirainen <tss@iki.fi> parents: 2298 diff changeset	178 cache_rec.size = buffer_get_used_size(ctx.buffer);
ef0cd18d674c Fixed compression. Timo Sirainen <tss@iki.fi> parents: 2298 diff changeset	179 if (cache_rec.size == sizeof(cache_rec))
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	180 continue;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	181
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	182 mail_index_update_ext(t, seq, cache->ext_id, &output->offset,
512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	183 &old_offset);
2299 ef0cd18d674c Fixed compression. Timo Sirainen <tss@iki.fi> parents: 2298 diff changeset	184
ef0cd18d674c Fixed compression. Timo Sirainen <tss@iki.fi> parents: 2298 diff changeset	185 buffer_write(ctx.buffer, 0, &cache_rec, sizeof(cache_rec));
2853 512dd7d76cdc Removed cache_offset from mail_index_record and changed it to use extension Timo Sirainen <tss@iki.fi> parents: 2708 diff changeset	186 o_stream_send(output, ctx.buffer->data, cache_rec.size);
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	187 }
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	188
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	189 if (cache->fields_count != 0) {
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	190 hdr.field_header_offset =
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	191 mail_index_uint32_to_offset(output->offset);
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	192
2671 3b70ddb51771 Renamed "extra record info" and variations of it to "extension" or "ext" in Timo Sirainen <tss@iki.fi> parents: 2641 diff changeset	193 /* we wrote everything using our internal field ids. so we want
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	194 mail_cache_header_fields_get() to use them and ignore any
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	195 existing id mappings in the old cache file. */
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	196 cache->file_fields_count = 0;
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	197 for (i = 0; i < cache->fields_count; i++)
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	198 cache->field_file_map[i] = (uint32_t)-1;
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	199
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	200 t_push();
2708 f1e9f3ec8135 Buffer API change: we no longer support limited sized buffers where Timo Sirainen <tss@iki.fi> parents: 2700 diff changeset	201 buffer = buffer_create_dynamic(pool_datastack_create(), 256);
2619 ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	202 mail_cache_header_fields_get(cache, buffer);
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	203 o_stream_send(output, buffer_get_data(buffer, NULL),
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	204 buffer_get_used_size(buffer));
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	205 t_pop();
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	206 }
ce8647d94bfc ID mapping fixes. Timo Sirainen <tss@iki.fi> parents: 2604 diff changeset	207
2276 5f374049abdb Caching fixes and optimizations. Removed all network byte ordering code - Timo Sirainen <tss@iki.fi> parents: 2275 diff changeset	208 hdr.used_file_size = output->offset;
2298 5beb0c20b6e8 Cache file fixes, API changes, etc. It's still in somewhat ugly state, but Timo Sirainen <tss@iki.fi> parents: 2282 diff changeset	209 buffer_free(ctx.buffer);
2641 e357ebb3137b Memory leak fixes Timo Sirainen <tss@iki.fi> parents: 2623 diff changeset	210 buffer_free(ctx.field_seen);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	211
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	212 o_stream_seek(output, 0);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	213 o_stream_send(output, &hdr, sizeof(hdr));
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	214
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	215 mail_cache_view_close(cache_view);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	216
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	217 if (o_stream_flush(output) < 0) {
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	218 errno = output->stream_errno;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	219 mail_cache_set_syscall_error(cache, "o_stream_flush()");
3879 928229f8b3e6 deinit, unref, destroy, close, free, etc. functions now take a pointer to Timo Sirainen <tss@iki.fi> parents: 3863 diff changeset	220 (void)mail_index_transaction_rollback(&t);
4070 71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of Timo Sirainen <tss@iki.fi> parents: 3879 diff changeset	221 o_stream_destroy(&output);
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	222 return -1;
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	223 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	224
2299 ef0cd18d674c Fixed compression. Timo Sirainen <tss@iki.fi> parents: 2298 diff changeset	225 if (hdr.used_file_size < MAIL_CACHE_INITIAL_SIZE) {
2275 c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	226 /* grow the file some more. doesn't matter if it fails */
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	227 (void)file_set_size(fd, MAIL_CACHE_INITIAL_SIZE);
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	228 }
c68a3c9f6d73 Cache file compression works now and compressed cache file is reopened. Timo Sirainen <tss@iki.fi> parents: 2257 diff changeset	229
4070 71b8faa84ec6 Added i_stream_destroy() and o_stream_destroy() and used them instead of Timo Sirainen <tss@iki.fi> parents: 3879 diff changeset	230 o_stream_destroy(&output);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	231
4918 2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	232 if (!cache->index->fsync_disable) {
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	233 if (fdatasync(fd) < 0) {
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	234 mail_cache_set_syscall_error(cache, "fdatasync()");
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	235 (void)mail_index_transaction_rollback(&t);
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	236 return -1;
2f9173e103fd Added fsync_disable setting. Also added missing fsync()ing to dbox when Timo Sirainen <tss@iki.fi> parents: 4876 diff changeset	237 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	238 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	239
3879 928229f8b3e6 deinit, unref, destroy, close, free, etc. functions now take a pointer to Timo Sirainen <tss@iki.fi> parents: 3863 diff changeset	240 return mail_index_transaction_commit(&t, &seq, &offset);
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	241 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	242
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	243 static int mail_cache_compress_has_file_changed(struct mail_cache *cache)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	244 {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	245 struct mail_cache_header hdr;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	246 unsigned int i;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	247 int fd, ret;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	248
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	249 for (i = 0;; i++) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	250 fd = nfs_safe_open(cache->filepath, O_RDONLY);
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	251 if (fd == -1) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	252 if (errno == ENOENT)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	253 return 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	254
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	255 mail_cache_set_syscall_error(cache, "open()");
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	256 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	257 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	258
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	259 ret = read_full(fd, &hdr, sizeof(hdr));
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	260 close_keep_errno(fd);
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	261
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	262 if (ret >= 0) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	263 if (ret == 0)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	264 return 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	265 if (cache->need_compress_file_seq == (uint32_t)-1) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	266 /* previously it didn't exist */
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	267 return 1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	268 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	269 return hdr.file_seq != cache->need_compress_file_seq;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	270 } else if (errno != ESTALE \|\| i >= NFS_ESTALE_RETRY_COUNT) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	271 mail_cache_set_syscall_error(cache, "read()");
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	272 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	273 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	274 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	275 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	276
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	277 static int mail_cache_compress_locked(struct mail_cache *cache,
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	278 struct mail_index_view *view,
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	279 bool *unlock)
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	280 {
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	281 struct dotlock *dotlock;
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	282 mode_t old_mask;
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	283 int fd, ret;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	284
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	285 /* get the latest info on fields */
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	286 if (mail_cache_header_fields_read(cache) < 0)
2327 7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	287 return -1;
7d02e2a7672d Header caching redesigned. New design allows caching decisions per field, so Timo Sirainen <tss@iki.fi> parents: 2299 diff changeset	288
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	289 old_mask = umask(cache->index->mode ^ 0666);
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	290 fd = file_dotlock_open(&cache->dotlock_settings, cache->filepath,
9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	291 0, &dotlock);
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	292 umask(old_mask);
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	293
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	294 if (fd == -1) {
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	295 mail_cache_set_syscall_error(cache, "file_dotlock_open()");
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	296 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	297 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	298
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	299 if ((ret = mail_cache_compress_has_file_changed(cache)) != 0) {
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	300 if (ret < 0)
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	301 return -1;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	302
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	303 /* was just compressed, forget this */
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	304 cache->need_compress_file_seq = 0;
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	305 file_dotlock_delete(&dotlock);
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	306
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	307 if (*unlock) {
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	308 (void)mail_cache_unlock(cache);
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	309 *unlock = FALSE;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	310 }
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	311
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	312 return mail_cache_reopen(cache);
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	313 }
9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	314
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	315 if (cache->index->gid != (gid_t)-1 &&
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	316 fchown(fd, (uid_t)-1, cache->index->gid) < 0) {
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	317 mail_cache_set_syscall_error(cache, "fchown()");
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	318 file_dotlock_delete(&dotlock);
2879 aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	319 return -1;
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	320 }
aa93c7216722 dovecot-shared file was supposed to show permissions/gid for files created Timo Sirainen <tss@iki.fi> parents: 2877 diff changeset	321
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	322 if (mail_cache_copy(cache, view, fd) < 0) {
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	323 (void)file_dotlock_delete(&dotlock);
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	324 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	325 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	326
3106 9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	327 if (file_dotlock_replace(&dotlock,
9c4aa309dbac Changed dotlocking API. Timo Sirainen <tss@iki.fi> parents: 2950 diff changeset	328 DOTLOCK_REPLACE_FLAG_DONT_CLOSE_FD) < 0) {
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	329 mail_cache_set_syscall_error(cache,
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	330 "file_dotlock_replace()");
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	331 (void)close(fd);
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	332 return -1;
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	333 }
79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	334
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	335 if (*unlock) {
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	336 (void)mail_cache_unlock(cache);
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	337 *unlock = FALSE;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	338 }
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	339
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	340 mail_cache_file_close(cache);
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	341 cache->fd = fd;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	342
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	343 if (cache->file_cache != NULL)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	344 file_cache_set_fd(cache->file_cache, cache->fd);
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	345
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	346 if (mail_cache_map(cache, 0, 0) < 0)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	347 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	348 if (mail_cache_header_fields_read(cache) < 0)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	349 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	350
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	351 cache->need_compress_file_seq = 0;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	352 return 0;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	353 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	354
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	355 int mail_cache_compress(struct mail_cache cache, struct mail_index_view view)
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	356 {
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	357 bool unlock = FALSE;
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	358 int ret;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	359
3243 40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	360 if (MAIL_INDEX_IS_IN_MEMORY(cache->index))
40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	361 return 0;
40b4ba3c55b8 In-memory indexes work again. Just pass dir as NULL to mail_index_alloc(). Timo Sirainen <tss@iki.fi> parents: 3106 diff changeset	362
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	363 if (cache->index->lock_method == FILE_LOCK_METHOD_DOTLOCK) {
2949 05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	364 /* we're using dotlocking, cache file creation itself creates
05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	365 the dotlock file we need. */
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	366 return mail_cache_compress_locked(cache, view, &unlock);
2949 05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	367 }
05ae72649fd0 Cache compression didn't work with lock_method = dotlock. Timo Sirainen <tss@iki.fi> parents: 2929 diff changeset	368
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	369 switch (mail_cache_lock(cache)) {
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	370 case -1:
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	371 return -1;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	372 case 0:
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	373 /* couldn't lock, either it's broken or doesn't exist.
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	374 just start creating it. */
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	375 return mail_cache_compress_locked(cache, view, &unlock);
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	376 default:
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	377 /* locking succeeded. */
4876 f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	378 unlock = TRUE;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	379 ret = mail_cache_compress_locked(cache, view, &unlock);
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	380 if (unlock) {
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	381 if (mail_cache_unlock(cache) < 0)
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	382 ret = -1;
f1d77064884c Lock handling changes. Everything goes through file-lock API now and there's Timo Sirainen <tss@iki.fi> parents: 4795 diff changeset	383 }
2929 ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	384 return ret;
ba9062032877 Locking fixes and cleanups Timo Sirainen <tss@iki.fi> parents: 2904 diff changeset	385 }
1915 79790750c349 importing new index code. mbox still broken. Timo Sirainen <tss@iki.fi> parents: diff changeset	386 }
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	387
3863 55df57c028d4 Added "bool" type and changed all ints that were used as booleans to bool. Timo Sirainen <tss@iki.fi> parents: 3819 diff changeset	388 bool mail_cache_need_compress(struct mail_cache *cache)
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	389 {
4247 9c3f19110538 Don't compress cache file if it was just compressed by another process. Timo Sirainen <tss@iki.fi> parents: 4070 diff changeset	390 return cache->need_compress_file_seq != 0;
2247 2d51bb58a070 Added some smartness for deciding what to cache. Cache compression code compiles, but untested. Timo Sirainen <tss@iki.fi> parents: 2200 diff changeset	391 }

Mercurial > dovecot > original-hg > dovecot-1.2

annotate src/lib-index/mail-cache-compress.c @ 5027:787ead060fe3 HEAD