dovecot/core-2.2: src/lib-index/mail-transaction-log-file.c annotate

annotate src/lib-index/mail-transaction-log-file.c @ 22691:dca05b22217b

director: Fix crash when handling expired USER timestamps. The fix in 154f91726624265fce15097eb4bbbf6e55f8c477 wasn't complete.

author	Timo Sirainen <timo.sirainen@dovecot.fi>
date	Tue, 28 Nov 2017 13:10:35 +0200
parents	ff8873d5675a
children	cb108f786fb4

rev	line source
21390 2e2563132d5f Updated copyright notices to include the year 2017. Stephan Bosch <stephan.bosch@dovecot.fi> parents: 21389 diff changeset	1 /* Copyright (c) 2003-2017 Dovecot authors, see the included COPYING file */
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	2
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	3 #include "lib.h"
14079 2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	4 #include "array.h"
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	5 #include "ioloop.h"
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	6 #include "file-dotlock.h"
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	7 #include "nfs-workarounds.h"
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	8 #include "read-full.h"
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	9 #include "write-full.h"
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	10 #include "mmap-util.h"
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	11 #include "mail-index-private.h"
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	12 #include "mail-index-modseq.h"
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	13 #include "mail-transaction-log-private.h"
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	14
11310 2937ca64faa9 lib-index: Use IO_BLOCK_SIZE Timo Sirainen <tss@iki.fi> parents: 11039 diff changeset	15 #define LOG_PREFETCH IO_BLOCK_SIZE
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	16 #define MEMORY_LOG_NAME "(in-memory transaction log file)"
12071 2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	17 #define LOG_NEW_DOTLOCK_SUFFIX ".newlock"
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	18
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	19 static int
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	20 mail_transaction_log_file_sync(struct mail_transaction_log_file *file,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	21 bool retry_r, const char *reason_r);
12395 878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	22
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	23 static void
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	24 log_file_set_syscall_error(struct mail_transaction_log_file *file,
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	25 const char *function)
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	26 {
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	27 mail_index_file_set_syscall_error(file->log->index,
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	28 file->filepath, function);
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	29 }
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	30
10289 38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	31 static void
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	32 mail_transaction_log_mark_corrupted(struct mail_transaction_log_file *file)
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	33 {
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	34 unsigned int offset =
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	35 offsetof(struct mail_transaction_log_header, indexid);
15644 6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	36 int flags;
10289 38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	37
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	38 if (MAIL_TRANSACTION_LOG_FILE_IN_MEMORY(file) \|\|
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	39 file->log->index->readonly)
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	40 return;
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	41
15644 6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	42 /* indexid=0 marks the log file as corrupted. we opened the file with
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	43 O_APPEND, and now we need to drop it for pwrite() to work (at least
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	44 in Linux) */
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	45 flags = fcntl(file->fd, F_GETFL, 0);
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	46 if (flags < 0) {
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	47 mail_index_file_set_syscall_error(file->log->index,
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	48 file->filepath, "fcntl(F_GETFL)");
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	49 return;
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	50 }
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	51 if (fcntl(file->fd, F_SETFL, flags & ~O_APPEND) < 0) {
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	52 mail_index_file_set_syscall_error(file->log->index,
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	53 file->filepath, "fcntl(F_SETFL)");
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	54 return;
6f62a31bf0d4 lib-index: Fixed marking transaction log file corrupted due to added O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15426 diff changeset	55 }
10289 38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	56 if (pwrite_full(file->fd, &file->hdr.indexid,
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	57 sizeof(file->hdr.indexid), offset) < 0) {
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	58 mail_index_file_set_syscall_error(file->log->index,
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	59 file->filepath, "pwrite()");
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	60 }
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	61 }
38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	62
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	63 void
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	64 mail_transaction_log_file_set_corrupted(struct mail_transaction_log_file *file,
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	65 const char *fmt, ...)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	66 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	67 va_list va;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	68
5819 4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	69 file->corrupted = TRUE;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	70 file->hdr.indexid = 0;
10289 38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	71 mail_transaction_log_mark_corrupted(file);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	72
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	73 va_start(va, fmt);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	74 T_BEGIN {
6940 414c9d631a81 Replaced t_push/t_pop calls with T_FRAME() macros. Timo Sirainen <tss@iki.fi>* parents: 6929 diff changeset	75 mail_index_set_error(file->log->index,
8809 8081eb6b1c61 Improved "corrupted transaction log file" error message. Timo Sirainen <tss@iki.fi> parents: 8805 diff changeset	76 "Corrupted transaction log file %s seq %u: %s "
8081eb6b1c61 Improved "corrupted transaction log file" error message. Timo Sirainen <tss@iki.fi> parents: 8805 diff changeset	77 "(sync_offset=%"PRIuUOFF_T")",
8081eb6b1c61 Improved "corrupted transaction log file" error message. Timo Sirainen <tss@iki.fi> parents: 8805 diff changeset	78 file->filepath, file->hdr.file_seq,
8081eb6b1c61 Improved "corrupted transaction log file" error message. Timo Sirainen <tss@iki.fi> parents: 8805 diff changeset	79 t_strdup_vprintf(fmt, va), file->sync_offset);
7226 e6693a0ec8e1 Renamed T_FRAME_BEGIN/END to T_BEGIN/END. Removed T_FRAME() macro and Timo Sirainen <tss@iki.fi> parents: 7086 diff changeset	80 } T_END;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	81 va_end(va);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	82 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	83
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	84 struct mail_transaction_log_file *
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	85 mail_transaction_log_file_alloc(struct mail_transaction_log *log,
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	86 const char *path)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	87 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	88 struct mail_transaction_log_file *file;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	89
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	90 file = i_new(struct mail_transaction_log_file, 1);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	91 file->log = log;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	92 file->filepath = i_strdup(path);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	93 file->fd = -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	94 return file;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	95 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	96
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	97 void mail_transaction_log_file_free(struct mail_transaction_log_file **_file)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	98 {
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	99 struct mail_transaction_log_file file = _file;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	100 struct mail_transaction_log_file **p;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	101 int old_errno = errno;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	102
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	103 *_file = NULL;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	104
17926 623a9f46c747 lib-index: Replaced some automatic transaction log unlocks with asserts. Timo Sirainen <tss@iki.fi> parents: 17925 diff changeset	105 i_assert(!file->locked);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	106
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	107 for (p = &file->log->files; p != NULL; p = &(p)->next) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	108 if (*p == file) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	109 *p = file->next;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	110 break;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	111 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	112 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	113
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	114 if (file == file->log->head)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	115 file->log->head = NULL;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	116
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	117 if (file->buffer != NULL)
6414 a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6360 diff changeset	118 buffer_free(&file->buffer);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	119
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	120 if (file->mmap_base != NULL) {
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	121 if (munmap(file->mmap_base, file->mmap_size) < 0)
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	122 log_file_set_syscall_error(file, "munmap()");
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	123 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	124
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	125 if (file->fd != -1) {
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	126 if (close(file->fd) < 0)
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	127 log_file_set_syscall_error(file, "close()");
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	128 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	129
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	130 i_free(file->filepath);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	131 i_free(file);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	132
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	133 errno = old_errno;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	134 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	135
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	136 static void
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	137 mail_transaction_log_file_skip_to_head(struct mail_transaction_log_file *file)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	138 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	139 struct mail_transaction_log *log = file->log;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	140 struct mail_index_map *map = log->index->map;
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	141 const struct mail_index_modseq_header *modseq_hdr;
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	142 uoff_t head_offset;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	143
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	144 if (map == NULL \|\| file->hdr.file_seq != map->hdr.log_file_seq \|\|
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	145 map->hdr.log_file_head_offset == 0)
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	146 return;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	147
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	148 /* we can get a valid log offset from index file. initialize
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	149 sync_offset from it so we don't have to read the whole log
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	150 file from beginning. */
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	151 head_offset = map->hdr.log_file_head_offset;
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	152
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	153 modseq_hdr = mail_index_map_get_modseq_header(map);
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	154 if (head_offset < file->hdr.hdr_size) {
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	155 mail_index_set_error(log->index,
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	156 "%s: log_file_head_offset too small",
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	157 log->index->filepath);
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	158 file->sync_offset = file->hdr.hdr_size;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	159 file->sync_highest_modseq = file->hdr.initial_modseq;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	160 } else if (modseq_hdr == NULL && file->hdr.initial_modseq == 0) {
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	161 /* modseqs not used yet */
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	162 file->sync_offset = head_offset;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	163 file->sync_highest_modseq = 0;
7962 465931d0a642 Index modseq handling crashfix. Timo Sirainen <tss@iki.fi> parents: 7932 diff changeset	164 } else if (modseq_hdr == NULL \|\|
465931d0a642 Index modseq handling crashfix. Timo Sirainen <tss@iki.fi> parents: 7932 diff changeset	165 modseq_hdr->log_seq != file->hdr.file_seq) {
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	166 /* highest_modseq not synced, start from beginning */
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	167 file->sync_offset = file->hdr.hdr_size;
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	168 file->sync_highest_modseq = file->hdr.initial_modseq;
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	169 } else if (modseq_hdr->log_offset > head_offset) {
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	170 mail_index_set_error(log->index,
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	171 "%s: modseq_hdr.log_offset too large",
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	172 log->index->filepath);
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	173 file->sync_offset = file->hdr.hdr_size;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	174 file->sync_highest_modseq = file->hdr.initial_modseq;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	175 } else {
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	176 /* start from where we last stopped tracking modseqs */
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	177 file->sync_offset = modseq_hdr->log_offset;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	178 file->sync_highest_modseq = modseq_hdr->highest_modseq;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	179 }
8811 69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	180 if (file->hdr.file_seq == log->index->map->hdr.log_file_seq) {
69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	181 file->saved_tail_offset =
69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	182 log->index->map->hdr.log_file_tail_offset;
69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	183 file->saved_tail_sync_offset = file->saved_tail_offset;
69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	184 }
8805 3feb2afdd81f indexes: Make sure we don't shrink log_file_tail_offset. Timo Sirainen <tss@iki.fi> parents: 8613 diff changeset	185 if (file->saved_tail_offset > file->max_tail_offset)
3feb2afdd81f indexes: Make sure we don't shrink log_file_tail_offset. Timo Sirainen <tss@iki.fi> parents: 8613 diff changeset	186 file->max_tail_offset = file->saved_tail_offset;
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	187 }
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	188
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	189 static void
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	190 mail_transaction_log_file_add_to_list(struct mail_transaction_log_file *file)
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	191 {
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	192 struct mail_transaction_log_file **p;
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	193 const char *reason;
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	194 bool retry;
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	195
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	196 file->sync_offset = file->hdr.hdr_size;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	197 file->sync_highest_modseq = file->hdr.initial_modseq;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	198 mail_transaction_log_file_skip_to_head(file);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	199
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	200 /* insert it to correct position */
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	201 for (p = &file->log->files; p != NULL; p = &(p)->next) {
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	202 if ((*p)->hdr.file_seq > file->hdr.file_seq)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	203 break;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	204 i_assert((*p)->hdr.file_seq < file->hdr.file_seq);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	205 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	206
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	207 file->next = *p;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	208 *p = file;
12395 878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	209
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	210 if (file->buffer != NULL) {
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	211 /* if we read any unfinished data, make sure the buffer gets
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	212 truncated. */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	213 (void)mail_transaction_log_file_sync(file, &retry, &reason);
12395 878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	214 buffer_set_used_size(file->buffer,
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	215 file->sync_offset - file->buffer_offset);
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	216 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	217 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	218
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	219 static int
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	220 mail_transaction_log_init_hdr(struct mail_transaction_log *log,
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	221 struct mail_transaction_log_header *hdr)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	222 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	223 struct mail_index *index = log->index;
11364 6d68b7b25ae7 Changed some index error handling conditions to log an error instead of assert-crash. Timo Sirainen <tss@iki.fi> parents: 11313 diff changeset	224 struct mail_transaction_log_file *file;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	225
21142 572ab8899bbc lib-index: Make sure an already-corrupted index/log isn't written Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20721 diff changeset	226 i_assert(index->indexid != 0);
572ab8899bbc lib-index: Make sure an already-corrupted index/log isn't written Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20721 diff changeset	227
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 21354 diff changeset	228 i_zero(hdr);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	229 hdr->major_version = MAIL_TRANSACTION_LOG_MAJOR_VERSION;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	230 hdr->minor_version = MAIL_TRANSACTION_LOG_MINOR_VERSION;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	231 hdr->hdr_size = sizeof(struct mail_transaction_log_header);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	232 hdr->indexid = log->index->indexid;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	233 hdr->create_stamp = ioloop_time;
10204 b55afe3e957c Test WORDS_BIGENDIAN with #if, not #ifdef. Timo Sirainen <tss@iki.fi> parents: 10081 diff changeset	234 #if !WORDS_BIGENDIAN
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	235 hdr->compat_flags \|= MAIL_INDEX_COMPAT_LITTLE_ENDIAN;
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	236 #endif
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	237
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	238 if (index->fd != -1) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	239 /* not creating index - make sure we have latest header */
5819 4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	240 if (!index->mapping) {
5854 eca7e29dce0d Rewrite index lock handling. Timo Sirainen <tss@iki.fi> parents: 5849 diff changeset	241 if (mail_index_map(index,
eca7e29dce0d Rewrite index lock handling. Timo Sirainen <tss@iki.fi> parents: 5849 diff changeset	242 MAIL_INDEX_SYNC_HANDLER_HEAD) <= 0)
5819 4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	243 return -1;
4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	244 } else {
4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	245 /* if we got here from mapping, the .log file is
4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	246 corrupted. use whatever values we got from index
4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	247 file */
4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	248 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	249 }
5772 002aa9bbfcb2 Removed index->hdr. Use index->map->hdr instead. Timo Sirainen <tss@iki.fi> parents: 5749 diff changeset	250 if (index->map != NULL) {
002aa9bbfcb2 Removed index->hdr. Use index->map->hdr instead. Timo Sirainen <tss@iki.fi> parents: 5749 diff changeset	251 hdr->prev_file_seq = index->map->hdr.log_file_seq;
002aa9bbfcb2 Removed index->hdr. Use index->map->hdr instead. Timo Sirainen <tss@iki.fi> parents: 5749 diff changeset	252 hdr->prev_file_offset = index->map->hdr.log_file_head_offset;
002aa9bbfcb2 Removed index->hdr. Use index->map->hdr instead. Timo Sirainen <tss@iki.fi> parents: 5749 diff changeset	253 hdr->file_seq = index->map->hdr.log_file_seq + 1;
7932 91758686277a View syncing: Keep track of highest modseq in views. If we lose transaction Timo Sirainen <tss@iki.fi> parents: 7931 diff changeset	254 hdr->initial_modseq =
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	255 mail_index_map_modseq_get_highest(index->map);
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	256 } else {
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	257 hdr->file_seq = 1;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	258 }
20720 866d29e08e70 lib-index: Make sure new dovecot.index.log files start tracking modseqs. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19655 diff changeset	259 if (hdr->initial_modseq == 0) {
866d29e08e70 lib-index: Make sure new dovecot.index.log files start tracking modseqs. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19655 diff changeset	260 /* modseq tracking in log files is required for many reasons
866d29e08e70 lib-index: Make sure new dovecot.index.log files start tracking modseqs. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19655 diff changeset	261 nowadays, even if per-message modseqs aren't enabled in
866d29e08e70 lib-index: Make sure new dovecot.index.log files start tracking modseqs. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19655 diff changeset	262 dovecot.index. */
866d29e08e70 lib-index: Make sure new dovecot.index.log files start tracking modseqs. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19655 diff changeset	263 hdr->initial_modseq = 1;
866d29e08e70 lib-index: Make sure new dovecot.index.log files start tracking modseqs. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19655 diff changeset	264 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	265
7842 6480642aba24 Make sure initial_modseq is correct when rotating transaction log. Timo Sirainen <tss@iki.fi> parents: 7812 diff changeset	266 if (log->head != NULL) {
11364 6d68b7b25ae7 Changed some index error handling conditions to log an error instead of assert-crash. Timo Sirainen <tss@iki.fi> parents: 11313 diff changeset	267 /* make sure the sequence always increases to avoid crashes
6d68b7b25ae7 Changed some index error handling conditions to log an error instead of assert-crash. Timo Sirainen <tss@iki.fi> parents: 11313 diff changeset	268 later. this catches the buggy case where two processes
6d68b7b25ae7 Changed some index error handling conditions to log an error instead of assert-crash. Timo Sirainen <tss@iki.fi> parents: 11313 diff changeset	269 happen to replace the same log file. */
6d68b7b25ae7 Changed some index error handling conditions to log an error instead of assert-crash. Timo Sirainen <tss@iki.fi> parents: 11313 diff changeset	270 for (file = log->head->next; file != NULL; file = file->next) {
6d68b7b25ae7 Changed some index error handling conditions to log an error instead of assert-crash. Timo Sirainen <tss@iki.fi> parents: 11313 diff changeset	271 if (hdr->file_seq <= file->hdr.file_seq)
6d68b7b25ae7 Changed some index error handling conditions to log an error instead of assert-crash. Timo Sirainen <tss@iki.fi> parents: 11313 diff changeset	272 hdr->file_seq = file->hdr.file_seq + 1;
6d68b7b25ae7 Changed some index error handling conditions to log an error instead of assert-crash. Timo Sirainen <tss@iki.fi> parents: 11313 diff changeset	273 }
6d68b7b25ae7 Changed some index error handling conditions to log an error instead of assert-crash. Timo Sirainen <tss@iki.fi> parents: 11313 diff changeset	274
7842 6480642aba24 Make sure initial_modseq is correct when rotating transaction log. Timo Sirainen <tss@iki.fi> parents: 7812 diff changeset	275 if (hdr->file_seq <= log->head->hdr.file_seq) {
6480642aba24 Make sure initial_modseq is correct when rotating transaction log. Timo Sirainen <tss@iki.fi> parents: 7812 diff changeset	276 /* make sure the sequence grows */
6480642aba24 Make sure initial_modseq is correct when rotating transaction log. Timo Sirainen <tss@iki.fi> parents: 7812 diff changeset	277 hdr->file_seq = log->head->hdr.file_seq+1;
6480642aba24 Make sure initial_modseq is correct when rotating transaction log. Timo Sirainen <tss@iki.fi> parents: 7812 diff changeset	278 }
6480642aba24 Make sure initial_modseq is correct when rotating transaction log. Timo Sirainen <tss@iki.fi> parents: 7812 diff changeset	279 if (hdr->initial_modseq < log->head->sync_highest_modseq) {
6480642aba24 Make sure initial_modseq is correct when rotating transaction log. Timo Sirainen <tss@iki.fi> parents: 7812 diff changeset	280 /* this should be always up-to-date */
6480642aba24 Make sure initial_modseq is correct when rotating transaction log. Timo Sirainen <tss@iki.fi> parents: 7812 diff changeset	281 hdr->initial_modseq = log->head->sync_highest_modseq;
6480642aba24 Make sure initial_modseq is correct when rotating transaction log. Timo Sirainen <tss@iki.fi> parents: 7812 diff changeset	282 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	283 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	284 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	285 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	286
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	287 struct mail_transaction_log_file *
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	288 mail_transaction_log_file_alloc_in_memory(struct mail_transaction_log *log)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	289 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	290 struct mail_transaction_log_file *file;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	291
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	292 file = mail_transaction_log_file_alloc(log, MEMORY_LOG_NAME);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	293 if (mail_transaction_log_init_hdr(log, &file->hdr) < 0) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	294 i_free(file);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	295 return NULL;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	296 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	297
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	298 file->buffer = buffer_create_dynamic(default_pool, 4096);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	299 file->buffer_offset = sizeof(file->hdr);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	300
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	301 mail_transaction_log_file_add_to_list(file);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	302 return file;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	303 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	304
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	305 static int
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	306 mail_transaction_log_file_dotlock(struct mail_transaction_log_file *file)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	307 {
12071 2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	308 struct dotlock_settings dotlock_set;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	309 int ret;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	310
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	311 if (file->log->dotlock_count > 0)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	312 ret = 1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	313 else {
12071 2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	314 mail_transaction_log_get_dotlock_set(file->log, &dotlock_set);
2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	315 ret = file_dotlock_create(&dotlock_set, file->filepath, 0,
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	316 &file->log->dotlock);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	317 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	318 if (ret > 0) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	319 file->log->dotlock_count++;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	320 file->locked = TRUE;
12154 6d6d9f49f61b lib-index: Log a warning if transaction log lock is kept over 3 minutes. Timo Sirainen <tss@iki.fi> parents: 12071 diff changeset	321 file->lock_created = time(NULL);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	322 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	323 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	324 if (ret < 0) {
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	325 log_file_set_syscall_error(file, "file_dotlock_create()");
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	326 return -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	327 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	328
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	329 mail_index_set_error(file->log->index,
12071 2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	330 "Timeout (%us) while waiting for "
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	331 "dotlock for transaction log file %s",
12071 2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	332 dotlock_set.timeout, file->filepath);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	333 file->log->index->index_lock_timeout = TRUE;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	334 return -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	335 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	336
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	337 static int
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	338 mail_transaction_log_file_undotlock(struct mail_transaction_log_file *file)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	339 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	340 int ret;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	341
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	342 if (--file->log->dotlock_count > 0)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	343 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	344
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	345 ret = file_dotlock_delete(&file->log->dotlock);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	346 if (ret < 0) {
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	347 log_file_set_syscall_error(file, "file_dotlock_delete()");
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	348 return -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	349 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	350
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	351 if (ret == 0) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	352 mail_index_set_error(file->log->index,
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	353 "Dotlock was lost for transaction log file %s",
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	354 file->filepath);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	355 return -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	356 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	357 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	358 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	359
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	360 int mail_transaction_log_file_lock(struct mail_transaction_log_file *file)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	361 {
11623 c3ee4421e86a Added mail_max_lock_timeout setting. Timo Sirainen <tss@iki.fi> parents: 11364 diff changeset	362 unsigned int lock_timeout_secs;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	363 int ret;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	364
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	365 if (file->locked)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	366 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	367
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	368 if (MAIL_TRANSACTION_LOG_FILE_IN_MEMORY(file)) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	369 file->locked = TRUE;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	370 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	371 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	372
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	373 if (file->log->index->lock_method == FILE_LOCK_METHOD_DOTLOCK)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	374 return mail_transaction_log_file_dotlock(file);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	375
15850 d40e3bf2de89 lib-index: Give a better error message if index is readonly and log is tried to be locked. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	376 if (file->log->index->readonly) {
d40e3bf2de89 lib-index: Give a better error message if index is readonly and log is tried to be locked. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	377 mail_index_set_error(file->log->index,
d40e3bf2de89 lib-index: Give a better error message if index is readonly and log is tried to be locked. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	378 "Index is read-only, can't write-lock %s",
d40e3bf2de89 lib-index: Give a better error message if index is readonly and log is tried to be locked. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	379 file->filepath);
d40e3bf2de89 lib-index: Give a better error message if index is readonly and log is tried to be locked. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	380 return -1;
d40e3bf2de89 lib-index: Give a better error message if index is readonly and log is tried to be locked. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	381 }
d40e3bf2de89 lib-index: Give a better error message if index is readonly and log is tried to be locked. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	382
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	383 i_assert(file->file_lock == NULL);
15182 ee0cf9c67571 Typofix: s/TRANSCATION/TRANSACTION/ Timo Sirainen <tss@iki.fi> parents: 15097 diff changeset	384 lock_timeout_secs = I_MIN(MAIL_TRANSACTION_LOG_LOCK_TIMEOUT,
11623 c3ee4421e86a Added mail_max_lock_timeout setting. Timo Sirainen <tss@iki.fi> parents: 11364 diff changeset	385 file->log->index->max_lock_timeout_secs);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	386 ret = mail_index_lock_fd(file->log->index, file->filepath, file->fd,
11623 c3ee4421e86a Added mail_max_lock_timeout setting. Timo Sirainen <tss@iki.fi> parents: 11364 diff changeset	387 F_WRLCK, lock_timeout_secs,
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	388 &file->file_lock);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	389 if (ret > 0) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	390 file->locked = TRUE;
12154 6d6d9f49f61b lib-index: Log a warning if transaction log lock is kept over 3 minutes. Timo Sirainen <tss@iki.fi> parents: 12071 diff changeset	391 file->lock_created = time(NULL);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	392 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	393 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	394 if (ret < 0) {
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	395 log_file_set_syscall_error(file, "mail_index_wait_lock_fd()");
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	396 return -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	397 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	398
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	399 mail_index_set_error(file->log->index,
12071 2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	400 "Timeout (%us) while waiting for lock for "
17925 ec1fd3dc0a74 lib-index: If locking transaction log times out, show the PID which has it locked. Timo Sirainen <tss@iki.fi> parents: 17758 diff changeset	401 "transaction log file %s%s",
ec1fd3dc0a74 lib-index: If locking transaction log times out, show the PID which has it locked. Timo Sirainen <tss@iki.fi> parents: 17758 diff changeset	402 lock_timeout_secs, file->filepath,
ec1fd3dc0a74 lib-index: If locking transaction log times out, show the PID which has it locked. Timo Sirainen <tss@iki.fi> parents: 17758 diff changeset	403 file_lock_find(file->fd, file->log->index->lock_method, F_WRLCK));
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	404 file->log->index->index_lock_timeout = TRUE;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	405 return -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	406 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	407
17927 fce66ec2ac1c lib-index: Include reason string in warnings about keeping transaction log locked for too long. Timo Sirainen <tss@iki.fi> parents: 17926 diff changeset	408 void mail_transaction_log_file_unlock(struct mail_transaction_log_file *file,
fce66ec2ac1c lib-index: Include reason string in warnings about keeping transaction log locked for too long. Timo Sirainen <tss@iki.fi> parents: 17926 diff changeset	409 const char *lock_reason)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	410 {
12154 6d6d9f49f61b lib-index: Log a warning if transaction log lock is kept over 3 minutes. Timo Sirainen <tss@iki.fi> parents: 12071 diff changeset	411 unsigned int lock_time;
6d6d9f49f61b lib-index: Log a warning if transaction log lock is kept over 3 minutes. Timo Sirainen <tss@iki.fi> parents: 12071 diff changeset	412
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	413 if (!file->locked)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	414 return;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	415
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	416 file->locked = FALSE;
12249 41e99ee5c1dd lib-index: Avoid unnecessary pread()s at end of transaction log file. Timo Sirainen <tss@iki.fi> parents: 12204 diff changeset	417 file->locked_sync_offset_updated = FALSE;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	418
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	419 if (MAIL_TRANSACTION_LOG_FILE_IN_MEMORY(file))
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	420 return;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	421
12154 6d6d9f49f61b lib-index: Log a warning if transaction log lock is kept over 3 minutes. Timo Sirainen <tss@iki.fi> parents: 12071 diff changeset	422 lock_time = time(NULL) - file->lock_created;
19655 5bdb2be321ba lib-index: If transaction log was locked for at least 30 secs, log a warning (not at 3min) Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	423 if (lock_time >= MAIL_TRANSACTION_LOG_LOCK_WARN_SECS && lock_reason != NULL) {
17927 fce66ec2ac1c lib-index: Include reason string in warnings about keeping transaction log locked for too long. Timo Sirainen <tss@iki.fi> parents: 17926 diff changeset	424 i_warning("Transaction log file %s was locked for %u seconds (%s)",
fce66ec2ac1c lib-index: Include reason string in warnings about keeping transaction log locked for too long. Timo Sirainen <tss@iki.fi> parents: 17926 diff changeset	425 file->filepath, lock_time, lock_reason);
12154 6d6d9f49f61b lib-index: Log a warning if transaction log lock is kept over 3 minutes. Timo Sirainen <tss@iki.fi> parents: 12071 diff changeset	426 }
6d6d9f49f61b lib-index: Log a warning if transaction log lock is kept over 3 minutes. Timo Sirainen <tss@iki.fi> parents: 12071 diff changeset	427
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	428 if (file->log->index->lock_method == FILE_LOCK_METHOD_DOTLOCK) {
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	429 (void)mail_transaction_log_file_undotlock(file);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	430 return;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	431 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	432
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	433 file_unlock(&file->file_lock);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	434 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	435
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	436 static ssize_t
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	437 mail_transaction_log_file_read_header(struct mail_transaction_log_file *file)
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	438 {
12204 062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	439 void *dest;
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	440 size_t pos, dest_size;
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	441 ssize_t ret;
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	442
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	443 i_assert(file->buffer == NULL && file->mmap_base == NULL);
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	444
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 21354 diff changeset	445 i_zero(&file->hdr);
12204 062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	446 if (file->last_size < mmap_get_page_size() && file->last_size > 0) {
12395 878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	447 /* just read the entire transaction log to memory.
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	448 note that if some of the data hasn't been fully committed
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	449 yet (hdr.size=0), the buffer must be truncated later */
12204 062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	450 file->buffer = buffer_create_dynamic(default_pool, 4096);
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	451 file->buffer_offset = 0;
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	452 dest_size = file->last_size;
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	453 dest = buffer_append_space_unsafe(file->buffer, dest_size);
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	454 } else {
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	455 /* read only the header */
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	456 dest = &file->hdr;
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	457 dest_size = sizeof(file->hdr);
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	458 }
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	459
12204 062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	460 /* it's not necessarily an error to read less than wanted header size,
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	461 since older versions of the log format used smaller headers. */
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	462 pos = 0;
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	463 do {
12204 062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	464 ret = pread(file->fd, PTR_OFFSET(dest, pos),
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	465 dest_size - pos, pos);
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	466 if (ret > 0)
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	467 pos += ret;
12204 062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	468 } while (ret > 0 && pos < dest_size);
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	469
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	470 if (file->buffer != NULL) {
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	471 buffer_set_used_size(file->buffer, pos);
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	472 memcpy(&file->hdr, file->buffer->data,
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	473 I_MIN(pos, sizeof(file->hdr)));
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	474 }
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	475
062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	476 return ret < 0 ? -1 : (ssize_t)pos;
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	477 }
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	478
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	479 static int
15316 be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	480 mail_transaction_log_file_fail_dupe(struct mail_transaction_log_file *file)
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	481 {
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	482 int ret;
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	483
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	484 /* mark the old file corrupted. we can't safely remove
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	485 it from the list however, so return failure. */
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	486 file->hdr.indexid = 0;
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	487 if (strcmp(file->filepath, file->log->head->filepath) != 0) {
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	488 /* only mark .2 corrupted, just to make sure we don't lose any
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	489 changes from .log in case we're somehow wrong */
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	490 mail_transaction_log_mark_corrupted(file);
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	491 ret = 0;
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	492 } else {
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	493 ret = -1;
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	494 }
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	495 if (!file->corrupted) {
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	496 file->corrupted = TRUE;
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	497 mail_index_set_error(file->log->index,
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	498 "Transaction log %s: "
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	499 "duplicate transaction log sequence (%u)",
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	500 file->filepath, file->hdr.file_seq);
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	501 }
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	502 return ret;
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	503 }
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	504
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	505 static int
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	506 mail_transaction_log_file_read_hdr(struct mail_transaction_log_file *file,
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	507 bool ignore_estale)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	508 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	509 struct mail_transaction_log_file *f;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	510 int ret;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	511
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	512 i_assert(!MAIL_TRANSACTION_LOG_FILE_IN_MEMORY(file));
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	513
5819 4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	514 if (file->corrupted)
4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	515 return 0;
4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	516
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	517 ret = mail_transaction_log_file_read_header(file);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	518 if (ret < 0) {
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	519 if (errno != ESTALE \|\| !ignore_estale)
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	520 log_file_set_syscall_error(file, "pread()");
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	521 return -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	522 }
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	523 if (file->hdr.major_version != MAIL_TRANSACTION_LOG_MAJOR_VERSION) {
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	524 /* incompatible version - fix silently */
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	525 return 0;
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	526 }
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	527 if (ret < MAIL_TRANSACTION_LOG_HEADER_MIN_SIZE) {
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	528 mail_transaction_log_file_set_corrupted(file,
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	529 "unexpected end of file while reading header");
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	530 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	531 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	532
21875 7c51ff56653a lib-index: Add macros to check if transaction log file version is new enough. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21525 diff changeset	533 const unsigned int hdr_version =
7c51ff56653a lib-index: Add macros to check if transaction log file version is new enough. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21525 diff changeset	534 MAIL_TRANSACTION_LOG_HDR_VERSION(&file->hdr);
7c51ff56653a lib-index: Add macros to check if transaction log file version is new enough. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21525 diff changeset	535 if (MAIL_TRANSACTION_LOG_VERSION_HAVE(hdr_version, COMPAT_FLAGS)) {
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	536 /* we have compatibility flags */
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	537 enum mail_index_header_compat_flags compat_flags = 0;
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	538
10204 b55afe3e957c Test WORDS_BIGENDIAN with #if, not #ifdef. Timo Sirainen <tss@iki.fi> parents: 10081 diff changeset	539 #if !WORDS_BIGENDIAN
8129 612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	540 compat_flags \|= MAIL_INDEX_COMPAT_LITTLE_ENDIAN;
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	541 #endif
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	542 if (file->hdr.compat_flags != compat_flags) {
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	543 /* architecture change */
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	544 mail_index_set_error(file->log->index,
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	545 "Rebuilding index file %s: "
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	546 "CPU architecture changed",
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	547 file->log->index->filepath);
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	548 return 0;
612eb505775f Write CPU endianess to transaction log header and check it's correct when reading. Timo Sirainen <tss@iki.fi> parents: 7962 diff changeset	549 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	550 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	551 if (file->hdr.hdr_size < MAIL_TRANSACTION_LOG_HEADER_MIN_SIZE) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	552 mail_transaction_log_file_set_corrupted(file,
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	553 "Header size too small");
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	554 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	555 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	556 if (file->hdr.hdr_size < sizeof(file->hdr)) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	557 /* @UNSAFE: smaller than we expected - zero out the fields we
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	558 shouldn't have filled */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	559 memset(PTR_OFFSET(&file->hdr, file->hdr.hdr_size), 0,
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	560 sizeof(file->hdr) - file->hdr.hdr_size);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	561 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	562
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	563 if (file->hdr.indexid == 0) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	564 /* corrupted */
5819 4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	565 file->corrupted = TRUE;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	566 mail_index_set_error(file->log->index,
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	567 "Transaction log file %s: marked corrupted",
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	568 file->filepath);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	569 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	570 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	571 if (file->hdr.indexid != file->log->index->indexid) {
7573 de08cc81da73 Fixes to handling races in initial index creation. Timo Sirainen <tss@iki.fi> parents: 7546 diff changeset	572 if (file->log->index->indexid != 0 &&
de08cc81da73 Fixes to handling races in initial index creation. Timo Sirainen <tss@iki.fi> parents: 7546 diff changeset	573 !file->log->index->initial_create) {
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	574 /* index file was probably just rebuilt and we don't
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	575 know about it yet */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	576 mail_transaction_log_file_set_corrupted(file,
5819 4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	577 "indexid changed %u -> %u",
4ea31bf18a56 Handle indexid changes better. Timo Sirainen <tss@iki.fi> parents: 5796 diff changeset	578 file->log->index->indexid, file->hdr.indexid);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	579 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	580 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	581
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	582 /* creating index file. since transaction log is created
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	583 first, use the indexid in it to create the main index
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	584 to avoid races. */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	585 file->log->index->indexid = file->hdr.indexid;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	586 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	587
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	588 /* make sure we already don't have a file with the same sequence
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	589 opened. it shouldn't happen unless the old log file was
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	590 corrupted. */
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	591 for (f = file->log->files; f != NULL; f = f->next) {
13247 d2890dd972ae lib-index: Avoid crashing when dealing with duplicate transaction log sequences. Timo Sirainen <tss@iki.fi> parents: 12985 diff changeset	592 if (f->hdr.file_seq == file->hdr.file_seq) {
10289 38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	593 if (strcmp(f->filepath, f->log->head->filepath) != 0) {
15316 be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	594 /* old "f" is the .log.2 */
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	595 return mail_transaction_log_file_fail_dupe(f);
10289 38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	596 } else {
15316 be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	597 /* new "file" is probably the .log.2 */
be50d12be960 lib-index: Fixed handling of finding a duplicate dovecot.index.log file_seq Timo Sirainen <tss@iki.fi> parents: 15315 diff changeset	598 return mail_transaction_log_file_fail_dupe(file);
10289 38d3e668dc1f index: If we see duplicate transaction log files, avoid corrupting the newer one. Timo Sirainen <tss@iki.fi> parents: 10204 diff changeset	599 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	600 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	601 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	602
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	603 file->sync_highest_modseq = file->hdr.initial_modseq;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	604 return 1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	605 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	606
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	607 static int
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	608 mail_transaction_log_file_stat(struct mail_transaction_log_file *file,
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	609 bool ignore_estale)
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	610 {
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	611 struct stat st;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	612
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	613 if (fstat(file->fd, &st) < 0) {
12936 d14b0fd0a423 Linux NFS: fstat() may return ENOENT instead of ESTALE in some kernel versions. Timo Sirainen <tss@iki.fi> parents: 12782 diff changeset	614 if (!ESTALE_FSTAT(errno) \|\| !ignore_estale)
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	615 log_file_set_syscall_error(file, "fstat()");
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	616 return -1;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	617 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	618
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	619 file->st_dev = st.st_dev;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	620 file->st_ino = st.st_ino;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	621 file->last_mtime = st.st_mtime;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	622 file->last_size = st.st_size;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	623 return 0;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	624 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	625
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	626 static bool
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	627 mail_transaction_log_file_is_dupe(struct mail_transaction_log_file *file)
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	628 {
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	629 struct mail_transaction_log_file *tmp;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	630
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	631 for (tmp = file->log->files; tmp != NULL; tmp = tmp->next) {
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	632 if (tmp->st_ino == file->st_ino &&
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	633 CMP_DEV_T(tmp->st_dev, file->st_dev))
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	634 return TRUE;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	635 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	636 return FALSE;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	637 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	638
14079 2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	639 static void log_write_ext_hdr_init_data(struct mail_index index, buffer_t buf)
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	640 {
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	641 const struct mail_index_registered_ext *rext;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	642 struct mail_transaction_header *hdr;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	643 struct mail_transaction_ext_intro *intro;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	644 struct mail_transaction_ext_hdr_update *ext_hdr;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	645 unsigned int hdr_offset;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	646
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	647 rext = array_idx(&index->extensions, index->ext_hdr_init_id);
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	648
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	649 /* introduce the extension */
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	650 hdr_offset = buf->used;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	651 hdr = buffer_append_space_unsafe(buf, sizeof(*hdr));
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	652 hdr->type = MAIL_TRANSACTION_EXT_INTRO;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	653
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	654 intro = buffer_append_space_unsafe(buf, sizeof(*intro));
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	655 intro->ext_id = (uint32_t)-1;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	656 intro->hdr_size = rext->hdr_size;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	657 intro->record_size = rext->record_size;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	658 intro->record_align = rext->record_align;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	659 intro->name_size = strlen(rext->name);
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	660 buffer_append(buf, rext->name, intro->name_size);
15097 d2dd659b4a4a lib-index: Transaction log creation crashed if required extension name length wasn't divisible by 4. Timo Sirainen <tss@iki.fi> parents: 15034 diff changeset	661 if (buf->used % 4 != 0)
d2dd659b4a4a lib-index: Transaction log creation crashed if required extension name length wasn't divisible by 4. Timo Sirainen <tss@iki.fi> parents: 15034 diff changeset	662 buffer_append_zero(buf, 4 - buf->used % 4);
14079 2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	663
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	664 hdr = buffer_get_space_unsafe(buf, hdr_offset, sizeof(*hdr));
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	665 hdr->size = mail_index_uint32_to_offset(buf->used - hdr_offset);
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	666
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	667 /* add the extension header data */
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	668 hdr_offset = buf->used;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	669 hdr = buffer_append_space_unsafe(buf, sizeof(*hdr));
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	670 hdr->type = MAIL_TRANSACTION_EXT_HDR_UPDATE;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	671
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	672 ext_hdr = buffer_append_space_unsafe(buf, sizeof(*ext_hdr));
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	673 ext_hdr->size = rext->hdr_size;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	674 buffer_append(buf, index->ext_hdr_init_data, rext->hdr_size);
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	675
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	676 hdr = buffer_get_space_unsafe(buf, hdr_offset, sizeof(*hdr));
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	677 hdr->size = mail_index_uint32_to_offset(buf->used - hdr_offset);
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	678 }
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	679
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	680 static int
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	681 mail_transaction_log_file_create2(struct mail_transaction_log_file *file,
5849 a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	682 int new_fd, bool reset,
a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	683 struct dotlock **dotlock)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	684 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	685 struct mail_index *index = file->log->index;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	686 struct stat st;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	687 const char *path2;
14079 2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	688 buffer_t *writebuf;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	689 int fd, ret;
18276 e7c65fa3ffa9 lib-index: Added an assert. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	690 bool rename_existing, need_lock;
e7c65fa3ffa9 lib-index: Added an assert. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	691
e7c65fa3ffa9 lib-index: Added an assert. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	692 need_lock = file->log->head != NULL && file->log->head->locked;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	693
15367 751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	694 if (fcntl(new_fd, F_SETFL, O_APPEND) < 0) {
751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	695 log_file_set_syscall_error(file, "fcntl(O_APPEND)");
751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	696 return -1;
751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	697 }
751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	698
11634 7f19062f58fd Renamed fsync_disable to mail_fsync=optimized\|always\|never. Timo Sirainen <tss@iki.fi> parents: 11623 diff changeset	699 if (file->log->nfs_flush) {
6857 41911abe6fa7 NFS cache flushing updates. Timo Sirainen <tss@iki.fi> parents: 6829 diff changeset	700 /* although we check also mtime and file size below, it's done
41911abe6fa7 NFS cache flushing updates. Timo Sirainen <tss@iki.fi> parents: 6829 diff changeset	701 only to fix broken log files. we don't bother flushing
41911abe6fa7 NFS cache flushing updates. Timo Sirainen <tss@iki.fi> parents: 6829 diff changeset	702 attribute cache just for that. */
41911abe6fa7 NFS cache flushing updates. Timo Sirainen <tss@iki.fi> parents: 6829 diff changeset	703 nfs_flush_file_handle_cache(file->filepath);
41911abe6fa7 NFS cache flushing updates. Timo Sirainen <tss@iki.fi> parents: 6829 diff changeset	704 }
5972 9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5953 diff changeset	705
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	706 /* log creation is locked now - see if someone already created it.
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	707 note that if we're rotating, we need to keep the log locked until
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	708 the file has been rewritten. and because fcntl() locks are stupid,
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	709 if we go and open()+close() the file and we had it already opened,
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	710 its locks are lost. so we use stat() to check if the file has been
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	711 recreated, although it almost never is. */
5849 a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	712 if (reset)
a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	713 rename_existing = FALSE;
a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	714 else if (nfs_safe_stat(file->filepath, &st) < 0) {
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	715 if (errno != ENOENT) {
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	716 log_file_set_syscall_error(file, "stat()");
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	717 return -1;
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	718 }
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	719 rename_existing = FALSE;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	720 } else if (st.st_ino == file->st_ino &&
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	721 CMP_DEV_T(st.st_dev, file->st_dev) &&
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	722 /* inode/dev checks are enough when we're rotating the file,
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	723 but not when we're replacing a broken log file */
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	724 st.st_mtime == file->last_mtime &&
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	725 (uoff_t)st.st_size == file->last_size) {
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	726 /* no-one else recreated the file */
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	727 rename_existing = TRUE;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	728 } else {
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	729 /* recreated. use the file if its header is ok */
15367 751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	730 fd = nfs_safe_open(file->filepath, O_RDWR \| O_APPEND);
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	731 if (fd == -1) {
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	732 if (errno != ENOENT) {
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	733 log_file_set_syscall_error(file, "open()");
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	734 return -1;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	735 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	736 } else {
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	737 file->fd = fd;
12204 062da0b83c01 lib-index: If transaction log file is small enough, just read all of it to memory immediately. Timo Sirainen <tss@iki.fi> parents: 12154 diff changeset	738 file->last_size = 0;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	739 if (mail_transaction_log_file_read_hdr(file,
5996 00be53d8f076 Fixed "duplicate transaction log sequence" errors when we noticed that log Timo Sirainen <tss@iki.fi> parents: 5972 diff changeset	740 FALSE) > 0 &&
00be53d8f076 Fixed "duplicate transaction log sequence" errors when we noticed that log Timo Sirainen <tss@iki.fi> parents: 5972 diff changeset	741 mail_transaction_log_file_stat(file, FALSE) == 0) {
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	742 /* yes, it was ok */
14637 a77ad2346cf0 Added file_dotlock_delete_verified() and changed file_dotlock_delete() to return void. Timo Sirainen <tss@iki.fi> parents: 14435 diff changeset	743 file_dotlock_delete(dotlock);
5953 2b150b4b49df Fixes Timo Sirainen <tss@iki.fi> parents: 5867 diff changeset	744 mail_transaction_log_file_add_to_list(file);
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	745 return 0;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	746 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	747 file->fd = -1;
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	748 if (close(fd) < 0)
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	749 log_file_set_syscall_error(file, "close()");
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	750 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	751 rename_existing = FALSE;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	752 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	753
14079 2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	754 if (index->fd == -1 && !rename_existing) {
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	755 /* creating the initial index */
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	756 reset = TRUE;
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	757 }
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	758
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	759 if (mail_transaction_log_init_hdr(file->log, &file->hdr) < 0)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	760 return -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	761
5849 a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	762 if (reset) {
8972 8f82eb1f5033 Don't reset modseqs when reseting index. Timo Sirainen <tss@iki.fi> parents: 8811 diff changeset	763 /* don't reset modseqs. if we're reseting due to rebuilding
8f82eb1f5033 Don't reset modseqs when reseting index. Timo Sirainen <tss@iki.fi> parents: 8811 diff changeset	764 indexes we'll probably want to keep uidvalidity and in such
8f82eb1f5033 Don't reset modseqs when reseting index. Timo Sirainen <tss@iki.fi> parents: 8811 diff changeset	765 cases we really don't want to shrink modseqs. */
5849 a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	766 file->hdr.prev_file_seq = 0;
a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	767 file->hdr.prev_file_offset = 0;
a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	768 }
a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	769
14079 2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	770 writebuf = buffer_create_dynamic(pool_datastack_create(), 128);
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	771 buffer_append(writebuf, &file->hdr, sizeof(file->hdr));
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	772
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	773 if (index->ext_hdr_init_data != NULL && reset)
2ab26bb55346 lib-index: Added mail_index_set_ext_init_data() for adding data to index on creation. Timo Sirainen <tss@iki.fi> parents: 13248 diff changeset	774 log_write_ext_hdr_init_data(index, writebuf);
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	775 if (write_full(new_fd, writebuf->data, writebuf->used) < 0) {
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	776 log_file_set_syscall_error(file, "write_full()");
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	777 return -1;
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	778 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	779
11634 7f19062f58fd Renamed fsync_disable to mail_fsync=optimized\|always\|never. Timo Sirainen <tss@iki.fi> parents: 11623 diff changeset	780 if (file->log->index->fsync_mode == FSYNC_MODE_ALWAYS) {
5972 9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5953 diff changeset	781 /* the header isn't important, so don't bother calling
11634 7f19062f58fd Renamed fsync_disable to mail_fsync=optimized\|always\|never. Timo Sirainen <tss@iki.fi> parents: 11623 diff changeset	782 fdatasync() unless it's required */
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	783 if (fdatasync(new_fd) < 0) {
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	784 log_file_set_syscall_error(file, "fdatasync()");
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	785 return -1;
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	786 }
5972 9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5953 diff changeset	787 }
9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5953 diff changeset	788
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	789 file->fd = new_fd;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	790 ret = mail_transaction_log_file_stat(file, FALSE);
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	791
21354 30ef85789908 lib-index: Fix assert-crash if .log creation unexpectedly fails at the end Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21344 diff changeset	792 if (need_lock && ret == 0) {
6981 66b439e24c47 If we're rotating log file while we're locked, lock the newly created file Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	793 /* we'll need to preserve the lock */
66b439e24c47 If we're rotating log file while we're locked, lock the newly created file Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	794 if (mail_transaction_log_file_lock(file) < 0)
66b439e24c47 If we're rotating log file while we're locked, lock the newly created file Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	795 ret = -1;
66b439e24c47 If we're rotating log file while we're locked, lock the newly created file Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	796 }
66b439e24c47 If we're rotating log file while we're locked, lock the newly created file Timo Sirainen <tss@iki.fi> parents: 6940 diff changeset	797
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	798 /* if we return -1 the dotlock deletion code closes the fd */
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	799 file->fd = -1;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	800 if (ret < 0)
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	801 return -1;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	802
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	803 /* keep two log files */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	804 if (rename_existing) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	805 /* rename() would be nice and easy way to do this, except then
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	806 there's a race condition between the rename and
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	807 file_dotlock_replace(). during that time the log file
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	808 doesn't exist, which could cause problems. */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	809 path2 = t_strconcat(file->filepath, ".2", NULL);
19137 27a629e444b2 lib-index: Replaced some unlink()s with i_unlink()s where we could. Timo Sirainen <tss@iki.fi>* parents: 18619 diff changeset	810 if (i_unlink_if_exists(path2) < 0) {
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	811 /* try to link() anyway */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	812 }
6813 575235932194 nfs_safe_link(): Support linking files without original link count=1. Use it Timo Sirainen <tss@iki.fi> parents: 6812 diff changeset	813 if (nfs_safe_link(file->filepath, path2, FALSE) < 0 &&
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	814 errno != ENOENT && errno != EEXIST) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	815 mail_index_set_error(index, "link(%s, %s) failed: %m",
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	816 file->filepath, path2);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	817 /* ignore the error. we don't care that much about the
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	818 second log file and we're going to overwrite this
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	819 first one. */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	820 }
15315 bcef97fb1202 lib-index: Handle better race condition there dovecot.index.log and .log.2 are the same link. Timo Sirainen <tss@iki.fi> parents: 14673 diff changeset	821 /* NOTE: here's a race condition where both .log and .log.2
bcef97fb1202 lib-index: Handle better race condition there dovecot.index.log and .log.2 are the same link. Timo Sirainen <tss@iki.fi> parents: 14673 diff changeset	822 point to the same file. our reading code should ignore that
bcef97fb1202 lib-index: Handle better race condition there dovecot.index.log and .log.2 are the same link. Timo Sirainen <tss@iki.fi> parents: 14673 diff changeset	823 though by comparing the inodes. */
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	824 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	825
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	826 if (file_dotlock_replace(dotlock,
21354 30ef85789908 lib-index: Fix assert-crash if .log creation unexpectedly fails at the end Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21344 diff changeset	827 DOTLOCK_REPLACE_FLAG_DONT_CLOSE_FD) <= 0) {
30ef85789908 lib-index: Fix assert-crash if .log creation unexpectedly fails at the end Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21344 diff changeset	828 /* need to unlock to avoid assert-crash in
30ef85789908 lib-index: Fix assert-crash if .log creation unexpectedly fails at the end Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21344 diff changeset	829 mail_transaction_log_file_free() */
30ef85789908 lib-index: Fix assert-crash if .log creation unexpectedly fails at the end Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21344 diff changeset	830 mail_transaction_log_file_unlock(file, "creation failed");
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	831 return -1;
21354 30ef85789908 lib-index: Fix assert-crash if .log creation unexpectedly fails at the end Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21344 diff changeset	832 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	833
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	834 /* success */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	835 file->fd = new_fd;
18276 e7c65fa3ffa9 lib-index: Added an assert. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	836 mail_transaction_log_file_add_to_list(file);
e7c65fa3ffa9 lib-index: Added an assert. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	837
e7c65fa3ffa9 lib-index: Added an assert. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	838 i_assert(!need_lock \|\| file->locked);
18103 df1d0c0e7ae2 lib-index: If transaction log is unexpectedly recreated while it was locked, log a nice error. Timo Sirainen <tss@iki.fi> parents: 17983 diff changeset	839 return 1;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	840 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	841
5849 a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	842 int mail_transaction_log_file_create(struct mail_transaction_log_file *file,
a9df50952600 Added support for resetting index. Timo Sirainen <tss@iki.fi> parents: 5819 diff changeset	843 bool reset)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	844 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	845 struct mail_index *index = file->log->index;
12071 2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	846 struct dotlock_settings new_dotlock_set;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	847 struct dotlock *dotlock;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	848 mode_t old_mask;
18103 df1d0c0e7ae2 lib-index: If transaction log is unexpectedly recreated while it was locked, log a nice error. Timo Sirainen <tss@iki.fi> parents: 17983 diff changeset	849 int fd, ret;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	850
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	851 i_assert(!MAIL_INDEX_IS_IN_MEMORY(index));
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	852
10006 a425ba46a64c lib-index: Take MAIL_INDEX_OPEN_FLAG_READONLY more seriously. Timo Sirainen <tss@iki.fi> parents: 9887 diff changeset	853 if (file->log->index->readonly) {
a425ba46a64c lib-index: Take MAIL_INDEX_OPEN_FLAG_READONLY more seriously. Timo Sirainen <tss@iki.fi> parents: 9887 diff changeset	854 mail_index_set_error(index,
a425ba46a64c lib-index: Take MAIL_INDEX_OPEN_FLAG_READONLY more seriously. Timo Sirainen <tss@iki.fi> parents: 9887 diff changeset	855 "Can't create log file %s: Index is read-only",
a425ba46a64c lib-index: Take MAIL_INDEX_OPEN_FLAG_READONLY more seriously. Timo Sirainen <tss@iki.fi> parents: 9887 diff changeset	856 file->filepath);
a425ba46a64c lib-index: Take MAIL_INDEX_OPEN_FLAG_READONLY more seriously. Timo Sirainen <tss@iki.fi> parents: 9887 diff changeset	857 return -1;
a425ba46a64c lib-index: Take MAIL_INDEX_OPEN_FLAG_READONLY more seriously. Timo Sirainen <tss@iki.fi> parents: 9887 diff changeset	858 }
a425ba46a64c lib-index: Take MAIL_INDEX_OPEN_FLAG_READONLY more seriously. Timo Sirainen <tss@iki.fi> parents: 9887 diff changeset	859
21142 572ab8899bbc lib-index: Make sure an already-corrupted index/log isn't written Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20721 diff changeset	860 if (index->indexid == 0) {
572ab8899bbc lib-index: Make sure an already-corrupted index/log isn't written Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20721 diff changeset	861 mail_index_set_error(index,
572ab8899bbc lib-index: Make sure an already-corrupted index/log isn't written Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20721 diff changeset	862 "Can't create log file %s: Index is marked corrupted",
572ab8899bbc lib-index: Make sure an already-corrupted index/log isn't written Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20721 diff changeset	863 file->filepath);
572ab8899bbc lib-index: Make sure an already-corrupted index/log isn't written Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20721 diff changeset	864 return -1;
572ab8899bbc lib-index: Make sure an already-corrupted index/log isn't written Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20721 diff changeset	865 }
572ab8899bbc lib-index: Make sure an already-corrupted index/log isn't written Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20721 diff changeset	866
12071 2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	867 mail_transaction_log_get_dotlock_set(file->log, &new_dotlock_set);
2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	868 new_dotlock_set.lock_suffix = LOG_NEW_DOTLOCK_SUFFIX;
2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	869
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	870 /* With dotlocking we might already have path.lock created, so this
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	871 filename has to be different. */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	872 old_mask = umask(index->mode ^ 0666);
12071 2dca300aee5c lib-index: Lock timeout setting wasn't actually used in all places. Timo Sirainen <tss@iki.fi> parents: 11665 diff changeset	873 fd = file_dotlock_open(&new_dotlock_set, file->filepath, 0, &dotlock);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	874 umask(old_mask);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	875
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	876 if (fd == -1) {
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	877 log_file_set_syscall_error(file, "file_dotlock_open()");
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	878 return -1;
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	879 }
9420 c496f137c843 index: Handle better errors where a new file's group can't be changed. Timo Sirainen <tss@iki.fi> parents: 9097 diff changeset	880 mail_index_fchown(index, fd, file_dotlock_get_lock_path(dotlock));
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	881
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	882 /* either fd gets used or the dotlock gets deleted and returned fd
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	883 is for the existing file */
18103 df1d0c0e7ae2 lib-index: If transaction log is unexpectedly recreated while it was locked, log a nice error. Timo Sirainen <tss@iki.fi> parents: 17983 diff changeset	884 ret = mail_transaction_log_file_create2(file, fd, reset, &dotlock);
df1d0c0e7ae2 lib-index: If transaction log is unexpectedly recreated while it was locked, log a nice error. Timo Sirainen <tss@iki.fi> parents: 17983 diff changeset	885 if (ret < 0) {
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	886 if (dotlock != NULL)
14637 a77ad2346cf0 Added file_dotlock_delete_verified() and changed file_dotlock_delete() to return void. Timo Sirainen <tss@iki.fi> parents: 14435 diff changeset	887 file_dotlock_delete(&dotlock);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	888 return -1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	889 }
18103 df1d0c0e7ae2 lib-index: If transaction log is unexpectedly recreated while it was locked, log a nice error. Timo Sirainen <tss@iki.fi> parents: 17983 diff changeset	890 return ret;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	891 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	892
21172 5ef269356b27 lib-index: Improve errors - return reason in mail_transaction_log_file_open() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21168 diff changeset	893 int mail_transaction_log_file_open(struct mail_transaction_log_file *file,
5ef269356b27 lib-index: Improve errors - return reason in mail_transaction_log_file_open() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21168 diff changeset	894 const char **reason_r)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	895 {
12964 5429f41aa6c1 lib-index: If .log file can't be opened read-write, fallback to read-only indexes. Timo Sirainen <tss@iki.fi> parents: 12936 diff changeset	896 struct mail_index *index = file->log->index;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	897 unsigned int i;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	898 bool ignore_estale;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	899 int ret;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	900
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	901 for (i = 0;; i++) {
15367 751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	902 if (!index->readonly) {
751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	903 file->fd = nfs_safe_open(file->filepath,
751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	904 O_RDWR \| O_APPEND);
751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	905 } else {
751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	906 file->fd = nfs_safe_open(file->filepath, O_RDONLY);
751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	907 }
12964 5429f41aa6c1 lib-index: If .log file can't be opened read-write, fallback to read-only indexes. Timo Sirainen <tss@iki.fi> parents: 12936 diff changeset	908 if (file->fd == -1 && errno == EACCES) {
5429f41aa6c1 lib-index: If .log file can't be opened read-write, fallback to read-only indexes. Timo Sirainen <tss@iki.fi> parents: 12936 diff changeset	909 file->fd = nfs_safe_open(file->filepath, O_RDONLY);
5429f41aa6c1 lib-index: If .log file can't be opened read-write, fallback to read-only indexes. Timo Sirainen <tss@iki.fi> parents: 12936 diff changeset	910 index->readonly = TRUE;
5429f41aa6c1 lib-index: If .log file can't be opened read-write, fallback to read-only indexes. Timo Sirainen <tss@iki.fi> parents: 12936 diff changeset	911 }
5429f41aa6c1 lib-index: If .log file can't be opened read-write, fallback to read-only indexes. Timo Sirainen <tss@iki.fi> parents: 12936 diff changeset	912 if (file->fd == -1) {
21172 5ef269356b27 lib-index: Improve errors - return reason in mail_transaction_log_file_open() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21168 diff changeset	913 if (errno == ENOENT) {
5ef269356b27 lib-index: Improve errors - return reason in mail_transaction_log_file_open() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21168 diff changeset	914 *reason_r = "File doesn't exist";
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	915 return 0;
21172 5ef269356b27 lib-index: Improve errors - return reason in mail_transaction_log_file_open() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21168 diff changeset	916 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	917
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	918 log_file_set_syscall_error(file, "open()");
21172 5ef269356b27 lib-index: Improve errors - return reason in mail_transaction_log_file_open() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21168 diff changeset	919 *reason_r = t_strdup_printf("open() failed: %m");
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	920 return -1;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	921 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	922
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	923 ignore_estale = i < MAIL_INDEX_ESTALE_RETRY_COUNT;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	924 if (mail_transaction_log_file_stat(file, ignore_estale) < 0)
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	925 ret = -1;
15315 bcef97fb1202 lib-index: Handle better race condition there dovecot.index.log and .log.2 are the same link. Timo Sirainen <tss@iki.fi> parents: 14673 diff changeset	926 else if (mail_transaction_log_file_is_dupe(file)) {
bcef97fb1202 lib-index: Handle better race condition there dovecot.index.log and .log.2 are the same link. Timo Sirainen <tss@iki.fi> parents: 14673 diff changeset	927 /* probably our already opened .log file has been
bcef97fb1202 lib-index: Handle better race condition there dovecot.index.log and .log.2 are the same link. Timo Sirainen <tss@iki.fi> parents: 14673 diff changeset	928 renamed to .log.2 and we're trying to reopen it.
bcef97fb1202 lib-index: Handle better race condition there dovecot.index.log and .log.2 are the same link. Timo Sirainen <tss@iki.fi> parents: 14673 diff changeset	929 also possible that hit a race condition where .log
bcef97fb1202 lib-index: Handle better race condition there dovecot.index.log and .log.2 are the same link. Timo Sirainen <tss@iki.fi> parents: 14673 diff changeset	930 and .log.2 are linked. */
21172 5ef269356b27 lib-index: Improve errors - return reason in mail_transaction_log_file_open() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21168 diff changeset	931 *reason_r = "File is already open";
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	932 return 0;
15315 bcef97fb1202 lib-index: Handle better race condition there dovecot.index.log and .log.2 are the same link. Timo Sirainen <tss@iki.fi> parents: 14673 diff changeset	933 } else {
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	934 ret = mail_transaction_log_file_read_hdr(file,
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	935 ignore_estale);
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	936 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	937 if (ret > 0) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	938 /* success */
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	939 break;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	940 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	941
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	942 if (ret == 0) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	943 /* corrupted */
12964 5429f41aa6c1 lib-index: If .log file can't be opened read-write, fallback to read-only indexes. Timo Sirainen <tss@iki.fi> parents: 12936 diff changeset	944 if (index->readonly) {
10006 a425ba46a64c lib-index: Take MAIL_INDEX_OPEN_FLAG_READONLY more seriously. Timo Sirainen <tss@iki.fi> parents: 9887 diff changeset	945 /* don't delete */
19137 27a629e444b2 lib-index: Replaced some unlink()s with i_unlink()s where we could. Timo Sirainen <tss@iki.fi>* parents: 18619 diff changeset	946 } else {
27a629e444b2 lib-index: Replaced some unlink()s with i_unlink()s where we could. Timo Sirainen <tss@iki.fi>* parents: 18619 diff changeset	947 i_unlink_if_exists(file->filepath);
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	948 }
21172 5ef269356b27 lib-index: Improve errors - return reason in mail_transaction_log_file_open() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21168 diff changeset	949 *reason_r = "File is corrupted";
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	950 return 0;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	951 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	952 if (errno != ESTALE \|\|
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	953 i == MAIL_INDEX_ESTALE_RETRY_COUNT) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	954 /* syscall error */
21172 5ef269356b27 lib-index: Improve errors - return reason in mail_transaction_log_file_open() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21168 diff changeset	955 *reason_r = t_strdup_printf("fstat() failed: %m");
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	956 return -1;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	957 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	958
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	959 /* ESTALE - try again */
21344 31df6d9d548a lib-index: Make sure buffer is not null before freeing Aki Tuomi <aki.tuomi@dovecot.fi> parents: 21238 diff changeset	960 if (file->buffer != NULL)
31df6d9d548a lib-index: Make sure buffer is not null before freeing Aki Tuomi <aki.tuomi@dovecot.fi> parents: 21238 diff changeset	961 buffer_free(&file->buffer);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	962 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	963
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	964 mail_transaction_log_file_add_to_list(file);
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	965 return 1;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	966 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	967
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	968 static int
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	969 log_file_track_mailbox_sync_offset_hdr(struct mail_transaction_log_file *file,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	970 const void *data, unsigned int trans_size,
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	971 const char **error_r)
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	972 {
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	973 const struct mail_transaction_header_update *u = data;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	974 const struct mail_index_header *ihdr;
17983 575dcd85b4be lib-index: Detect broken log_file_tail_offset updates without assert-crashing. Timo Sirainen <tss@iki.fi> parents: 17927 diff changeset	975 const unsigned int size = trans_size - sizeof(struct mail_transaction_header);
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	976 const unsigned int offset_pos =
5749 420a386fa27a int/ext/mailbox sync offset changes: Combined mailbox and int offsets to Timo Sirainen <tss@iki.fi> parents: 5723 diff changeset	977 offsetof(struct mail_index_header, log_file_tail_offset);
420a386fa27a int/ext/mailbox sync offset changes: Combined mailbox and int offsets to Timo Sirainen <tss@iki.fi> parents: 5723 diff changeset	978 const unsigned int offset_size = sizeof(ihdr->log_file_tail_offset);
8811 69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	979 uint32_t tail_offset;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	980
8811 69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	981 i_assert(offset_size == sizeof(tail_offset));
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	982
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	983 if (size < sizeof(u) \|\| size < sizeof(u) + u->size) {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	984 *error_r = "header update extends beyond record size";
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	985 mail_transaction_log_file_set_corrupted(file, "%s", *error_r);
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	986 return -1;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	987 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	988
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	989 if (u->offset <= offset_pos &&
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	990 u->offset + u->size >= offset_pos + offset_size) {
8811 69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	991 memcpy(&tail_offset,
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	992 CONST_PTR_OFFSET(u + 1, offset_pos - u->offset),
8811 69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	993 sizeof(tail_offset));
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	994
8811 69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	995 if (tail_offset < file->saved_tail_offset) {
17609 16d4cf2c0d65 lib-index: Don't update log_file_tail_offset unnecessarily. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	996 /* ignore shrinking tail offsets */
16d4cf2c0d65 lib-index: Don't update log_file_tail_offset unnecessarily. Timo Sirainen <tss@iki.fi> parents: 17130 diff changeset	997 return 1;
17983 575dcd85b4be lib-index: Detect broken log_file_tail_offset updates without assert-crashing. Timo Sirainen <tss@iki.fi> parents: 17927 diff changeset	998 } else if (tail_offset > file->sync_offset + trans_size) {
575dcd85b4be lib-index: Detect broken log_file_tail_offset updates without assert-crashing. Timo Sirainen <tss@iki.fi> parents: 17927 diff changeset	999 mail_transaction_log_file_set_corrupted(file,
575dcd85b4be lib-index: Detect broken log_file_tail_offset updates without assert-crashing. Timo Sirainen <tss@iki.fi> parents: 17927 diff changeset	1000 "log_file_tail_offset %u goes past sync offset %"PRIuUOFF_T,
575dcd85b4be lib-index: Detect broken log_file_tail_offset updates without assert-crashing. Timo Sirainen <tss@iki.fi> parents: 17927 diff changeset	1001 tail_offset, file->sync_offset + trans_size);
8811 69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	1002 } else {
69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	1003 file->saved_tail_offset = tail_offset;
69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	1004 if (tail_offset > file->max_tail_offset)
69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	1005 file->max_tail_offset = tail_offset;
69ddd7c9c585 indexes: Fixes to handling shrinking tail offsets. Timo Sirainen <tss@iki.fi> parents: 8809 diff changeset	1006 return 1;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1007 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1008 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1009 return 0;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1010 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1011
21876 ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1012 static bool
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1013 flag_updates_have_non_internal(const struct mail_transaction_flag_update *u,
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1014 unsigned int count, unsigned int version)
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1015 {
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1016 /* Hide internal flags from modseqs if the log file's version
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1017 is new enough. This allows upgrading without the modseqs suddenly
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1018 shrinking. */
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1019 if (!MAIL_TRANSACTION_LOG_VERSION_HAVE(version, HIDE_INTERNAL_MODSEQS))
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1020 return TRUE;
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1021
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1022 for (unsigned int i = 0; i < count; i++) {
22422 b92e6fa512df lib-index: Fix modseq tracking with multiple flag updates Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22421 diff changeset	1023 if (!MAIL_TRANSACTION_FLAG_UPDATE_IS_INTERNAL(&u[i]))
21876 ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1024 return TRUE;
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1025 }
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1026 return FALSE;
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1027 }
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1028
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1029 void mail_transaction_update_modseq(const struct mail_transaction_header *hdr,
21876 ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1030 const void data, uint64_t cur_modseq,
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1031 unsigned int version)
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1032 {
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1033 uint32_t trans_size;
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1034
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1035 trans_size = mail_index_offset_to_uint32(hdr->size);
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1036 i_assert(trans_size != 0);
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1037
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1038 if (*cur_modseq != 0) {
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1039 /* tracking modseqs */
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1040 } else if ((hdr->type & MAIL_TRANSACTION_TYPE_MASK) ==
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1041 MAIL_TRANSACTION_EXT_INTRO) {
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1042 /* modseqs not tracked yet. see if this is a modseq
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1043 extension introduction. */
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1044 const struct mail_transaction_ext_intro *intro = data;
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1045 const unsigned int modseq_ext_len =
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1046 strlen(MAIL_INDEX_MODSEQ_EXT_NAME);
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1047
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1048 if (intro->name_size == modseq_ext_len &&
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1049 memcmp(intro + 1, MAIL_INDEX_MODSEQ_EXT_NAME,
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1050 modseq_ext_len) == 0) {
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1051 /* modseq tracking started */
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1052 *cur_modseq += 1;
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1053 }
14435 30f8ef8696f0 lib-index: Minor code cleanup. Timo Sirainen <tss@iki.fi> parents: 14133 diff changeset	1054 return;
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1055 } else {
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1056 /* not tracking modseqs */
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1057 return;
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1058 }
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1059
21146 52a7790352e8 lib-index: Added mail_index_transaction_get_highest_modseq() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21142 diff changeset	1060 /* NOTE: keep in sync with mail_index_transaction_get_highest_modseq() */
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1061 switch (hdr->type & MAIL_TRANSACTION_TYPE_MASK) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1062 case MAIL_TRANSACTION_EXPUNGE \| MAIL_TRANSACTION_EXPUNGE_PROT:
9624 2558ba736207 Keep track of expunged messages' GUIDs and expose them via mailbox_get_expunges(). Timo Sirainen <tss@iki.fi> parents: 9623 diff changeset	1063 case MAIL_TRANSACTION_EXPUNGE_GUID \| MAIL_TRANSACTION_EXPUNGE_PROT:
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1064 if ((hdr->type & MAIL_TRANSACTION_EXTERNAL) == 0) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1065 /* ignore expunge requests */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1066 break;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1067 }
22465 ff8873d5675a global: Add/change switch case "fall through" comments Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22430 diff changeset	1068 /* fall through */
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1069 case MAIL_TRANSACTION_APPEND:
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1070 case MAIL_TRANSACTION_KEYWORD_UPDATE:
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1071 case MAIL_TRANSACTION_KEYWORD_RESET:
16023 db0136374bb0 lib-index: mail_index_attribute_[un]set() adds changed attributes' keys to transaction log. Timo Sirainen <tss@iki.fi> parents: 15956 diff changeset	1072 case MAIL_TRANSACTION_ATTRIBUTE_UPDATE:
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1073 /* these changes increase modseq */
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1074 *cur_modseq += 1;
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1075 break;
21876 ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1076 case MAIL_TRANSACTION_FLAG_UPDATE: {
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1077 const struct mail_transaction_flag_update *rec = data;
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1078 unsigned int count;
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1079
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1080 count = (trans_size - sizeof(hdr)) / sizeof(rec);
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1081 if (flag_updates_have_non_internal(rec, count, version))
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1082 *cur_modseq += 1;
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1083 break;
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1084 }
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1085 case MAIL_TRANSACTION_MODSEQ_UPDATE: {
11039 0f98525e4567 Removed dead code. Timo Sirainen <tss@iki.fi> parents: 10674 diff changeset	1086 const struct mail_transaction_modseq_update rec, end;
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1087
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1088 end = CONST_PTR_OFFSET(data, trans_size - sizeof(*hdr));
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1089 for (rec = data; rec < end; rec++) {
16709 4d297ab881ef lib-index: Fixed modseq updates when modseq was over 32bits long Timo Sirainen <tss@iki.fi> parents: 16023 diff changeset	1090 uint64_t modseq = ((uint64_t)rec->modseq_high32 << 32) \|
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1091 rec->modseq_low32;
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1092 if (*cur_modseq < modseq)
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1093 *cur_modseq = modseq;
b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1094 }
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1095 }
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1096 }
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1097 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1098
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1099 static struct modseq_cache *
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1100 modseq_cache_hit(struct mail_transaction_log_file *file, unsigned int idx)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1101 {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1102 struct modseq_cache cache;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1103
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1104 if (idx > 0) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1105 /* @UNSAFE: move it to top */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1106 cache = file->modseq_cache[idx];
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1107 memmove(file->modseq_cache + 1, file->modseq_cache,
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1108 sizeof(file->modseq_cache) idx);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1109 file->modseq_cache[0] = cache;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1110 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1111 return &file->modseq_cache[0];
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1112 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1113
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1114 static struct modseq_cache *
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1115 modseq_cache_get_offset(struct mail_transaction_log_file *file, uoff_t offset)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1116 {
15904 d3cf06639864 Replaced all -1U and (unsigned int)-1 with UINT_MAX. Timo Sirainen <tss@iki.fi> parents: 15850 diff changeset	1117 unsigned int i, best = UINT_MAX;
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1118
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1119 for (i = 0; i < N_ELEMENTS(file->modseq_cache); i++) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1120 if (offset < file->modseq_cache[i].offset)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1121 continue;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1122
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1123 if (file->modseq_cache[i].offset == 0)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1124 return NULL;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1125
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1126 if (offset == file->modseq_cache[i].offset) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1127 /* exact cache hit */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1128 return modseq_cache_hit(file, i);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1129 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1130
15904 d3cf06639864 Replaced all -1U and (unsigned int)-1 with UINT_MAX. Timo Sirainen <tss@iki.fi> parents: 15850 diff changeset	1131 if (best == UINT_MAX \|\|
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1132 file->modseq_cache[i].offset <
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1133 file->modseq_cache[best].offset)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1134 best = i;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1135 }
15904 d3cf06639864 Replaced all -1U and (unsigned int)-1 with UINT_MAX. Timo Sirainen <tss@iki.fi> parents: 15850 diff changeset	1136 if (best == UINT_MAX)
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1137 return NULL;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1138 return &file->modseq_cache[best];
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1139 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1140
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1141 static struct modseq_cache *
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1142 modseq_cache_get_modseq(struct mail_transaction_log_file *file, uint64_t modseq)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1143 {
15904 d3cf06639864 Replaced all -1U and (unsigned int)-1 with UINT_MAX. Timo Sirainen <tss@iki.fi> parents: 15850 diff changeset	1144 unsigned int i, best = UINT_MAX;
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1145
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1146 for (i = 0; i < N_ELEMENTS(file->modseq_cache); i++) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1147 if (modseq < file->modseq_cache[i].highest_modseq)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1148 continue;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1149
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1150 if (file->modseq_cache[i].offset == 0)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1151 return NULL;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1152
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1153 if (modseq == file->modseq_cache[i].highest_modseq) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1154 /* exact cache hit */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1155 return modseq_cache_hit(file, i);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1156 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1157
15904 d3cf06639864 Replaced all -1U and (unsigned int)-1 with UINT_MAX. Timo Sirainen <tss@iki.fi> parents: 15850 diff changeset	1158 if (best == UINT_MAX \|\|
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1159 file->modseq_cache[i].highest_modseq <
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1160 file->modseq_cache[best].highest_modseq)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1161 best = i;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1162 }
15904 d3cf06639864 Replaced all -1U and (unsigned int)-1 with UINT_MAX. Timo Sirainen <tss@iki.fi> parents: 15850 diff changeset	1163 if (best == UINT_MAX)
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1164 return NULL;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1165 return &file->modseq_cache[best];
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1166 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1167
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1168 static int
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1169 log_get_synced_record(struct mail_transaction_log_file file, uoff_t offset,
22062 7d4ec76b7d94 lib-index: mail_transaction_log_file_get_highest_modseq_at() - return error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22060 diff changeset	1170 const struct mail_transaction_header **hdr_r,
7d4ec76b7d94 lib-index: mail_transaction_log_file_get_highest_modseq_at() - return error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22060 diff changeset	1171 const char **error_r)
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1172 {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1173 const struct mail_transaction_header *hdr;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1174 uint32_t trans_size;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1175
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1176 hdr = CONST_PTR_OFFSET(file->buffer->data,
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1177 *offset - file->buffer_offset);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1178
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1179 /* we've already synced this record at some point. it should
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1180 be valid. */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1181 trans_size = mail_index_offset_to_uint32(hdr->size);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1182 if (trans_size < sizeof(*hdr) \|\|
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1183 *offset - file->buffer_offset + trans_size > file->buffer->used) {
22062 7d4ec76b7d94 lib-index: mail_transaction_log_file_get_highest_modseq_at() - return error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22060 diff changeset	1184 *error_r = t_strdup_printf(
8613 37e74d43ff26 Improved "Transaction log corrupted unexpectedly" error message. Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	1185 "Transaction log corrupted unexpectedly at "
37e74d43ff26 Improved "Transaction log corrupted unexpectedly" error message. Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	1186 "%"PRIuUOFF_T": Invalid size %u (type=%x)",
37e74d43ff26 Improved "Transaction log corrupted unexpectedly" error message. Timo Sirainen <tss@iki.fi> parents: 8590 diff changeset	1187 *offset, trans_size, hdr->type);
22062 7d4ec76b7d94 lib-index: mail_transaction_log_file_get_highest_modseq_at() - return error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22060 diff changeset	1188 mail_transaction_log_file_set_corrupted(file, "%s", *error_r);
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1189 return -1;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1190 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1191 *offset += trans_size;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1192 *hdr_r = hdr;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1193 return 0;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1194 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1195
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1196 int mail_transaction_log_file_get_highest_modseq_at(
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1197 struct mail_transaction_log_file *file,
22062 7d4ec76b7d94 lib-index: mail_transaction_log_file_get_highest_modseq_at() - return error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22060 diff changeset	1198 uoff_t offset, uint64_t *highest_modseq_r,
7d4ec76b7d94 lib-index: mail_transaction_log_file_get_highest_modseq_at() - return error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22060 diff changeset	1199 const char **error_r)
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1200 {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1201 const struct mail_transaction_header *hdr;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1202 struct modseq_cache *cache;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1203 uoff_t cur_offset;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1204 uint64_t cur_modseq;
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1205 const char *reason;
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1206 int ret;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1207
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1208 i_assert(offset <= file->sync_offset);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1209
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1210 if (offset == file->sync_offset) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1211 *highest_modseq_r = file->sync_highest_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1212 return 0;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1213 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1214
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1215 cache = modseq_cache_get_offset(file, offset);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1216 if (cache == NULL) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1217 /* nothing usable in cache - scan from beginning */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1218 cur_offset = file->hdr.hdr_size;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1219 cur_modseq = file->hdr.initial_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1220 } else if (cache->offset == offset) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1221 /* exact cache hit */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1222 *highest_modseq_r = cache->highest_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1223 return 0;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1224 } else {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1225 /* use cache to skip over some records */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1226 cur_offset = cache->offset;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1227 cur_modseq = cache->highest_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1228 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1229
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1230 ret = mail_transaction_log_file_map(file, cur_offset, offset, &reason);
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1231 if (ret <= 0) {
22062 7d4ec76b7d94 lib-index: mail_transaction_log_file_get_highest_modseq_at() - return error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22060 diff changeset	1232 *error_r = t_strdup_printf(
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1233 "Failed to map transaction log %s for getting modseq "
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1234 "at offset=%"PRIuUOFF_T" with start_offset=%"PRIuUOFF_T": %s",
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1235 file->filepath, offset, cur_offset, reason);
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1236 return -1;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1237 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1238
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1239 i_assert(cur_offset >= file->buffer_offset);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1240 i_assert(cur_offset + file->buffer->used >= offset);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1241 while (cur_offset < offset) {
22062 7d4ec76b7d94 lib-index: mail_transaction_log_file_get_highest_modseq_at() - return error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22060 diff changeset	1242 if (log_get_synced_record(file, &cur_offset, &hdr, error_r) < 0)
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1243 return- 1;
21876 ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1244 mail_transaction_update_modseq(hdr, hdr + 1, &cur_modseq,
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1245 MAIL_TRANSACTION_LOG_HDR_VERSION(&file->hdr));
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1246 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1247
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1248 /* @UNSAFE: cache the value */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1249 memmove(file->modseq_cache + 1, file->modseq_cache,
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1250 sizeof(file->modseq_cache)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1251 (N_ELEMENTS(file->modseq_cache) - 1));
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1252 file->modseq_cache[0].offset = cur_offset;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1253 file->modseq_cache[0].highest_modseq = cur_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1254
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1255 *highest_modseq_r = cur_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1256 return 0;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1257 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1258
22429 3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1259 static int
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1260 get_modseq_next_offset_at(struct mail_transaction_log_file *file,
22430 2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1261 uint64_t modseq, bool use_highest,
22429 3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1262 uoff_t cur_offset, uint64_t cur_modseq,
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1263 uoff_t *next_offset_r)
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1264 {
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1265 const struct mail_transaction_header *hdr;
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1266 const char *reason;
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1267 int ret;
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1268
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1269 /* make sure we've read until end of file. this is especially important
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1270 with non-head logs which might only have been opened without being
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1271 synced. */
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1272 ret = mail_transaction_log_file_map(file, *cur_offset, (uoff_t)-1, &reason);
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1273 if (ret <= 0) {
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1274 mail_index_set_error(file->log->index,
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1275 "Failed to map transaction log %s for getting offset "
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1276 "for modseq=%llu with start_offset=%"PRIuUOFF_T": %s",
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1277 file->filepath, (unsigned long long)modseq,
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1278 *cur_offset, reason);
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1279 return -1;
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1280 }
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1281
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1282 /* check sync_highest_modseq again in case sync_offset was updated */
22430 2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1283 if (modseq >= file->sync_highest_modseq && use_highest) {
22429 3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1284 *next_offset_r = file->sync_offset;
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1285 return 0;
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1286 }
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1287
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1288 i_assert(*cur_offset >= file->buffer_offset);
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1289 while (*cur_offset < file->sync_offset) {
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1290 if (log_get_synced_record(file, cur_offset, &hdr, &reason) < 0) {
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1291 mail_index_set_error(file->log->index,
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1292 "%s: %s", file->filepath, reason);
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1293 return -1;
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1294 }
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1295 mail_transaction_update_modseq(hdr, hdr + 1, cur_modseq,
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1296 MAIL_TRANSACTION_LOG_HDR_VERSION(&file->hdr));
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1297 if (*cur_modseq >= modseq)
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1298 break;
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1299 }
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1300 return 1;
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1301 }
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1302
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1303 int mail_transaction_log_file_get_modseq_next_offset(
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1304 struct mail_transaction_log_file *file,
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1305 uint64_t modseq, uoff_t *next_offset_r)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1306 {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1307 struct modseq_cache *cache;
9887 69152c0d40a1 lib-index: Minor code cleanup. Timo Sirainen <tss@iki.fi> parents: 9724 diff changeset	1308 uoff_t cur_offset;
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1309 uint64_t cur_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1310 int ret;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1311
20721 6e339cbdeaf5 lib-index: Fixed mail_index_modseq_get_next_log_offset() when accessing .log.2 Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20720 diff changeset	1312 if (modseq == file->sync_highest_modseq) {
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1313 *next_offset_r = file->sync_offset;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1314 return 0;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1315 }
10081 1120a013da33 lib-index: looking up offset for transaction log's initial modseq failed. Timo Sirainen <tss@iki.fi> parents: 10006 diff changeset	1316 if (modseq == file->hdr.initial_modseq) {
1120a013da33 lib-index: looking up offset for transaction log's initial modseq failed. Timo Sirainen <tss@iki.fi> parents: 10006 diff changeset	1317 *next_offset_r = file->hdr.hdr_size;
1120a013da33 lib-index: looking up offset for transaction log's initial modseq failed. Timo Sirainen <tss@iki.fi> parents: 10006 diff changeset	1318 return 0;
1120a013da33 lib-index: looking up offset for transaction log's initial modseq failed. Timo Sirainen <tss@iki.fi> parents: 10006 diff changeset	1319 }
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1320
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1321 cache = modseq_cache_get_modseq(file, modseq);
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1322 if (cache == NULL) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1323 /* nothing usable in cache - scan from beginning */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1324 cur_offset = file->hdr.hdr_size;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1325 cur_modseq = file->hdr.initial_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1326 } else if (cache->highest_modseq == modseq) {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1327 /* exact cache hit */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1328 *next_offset_r = cache->offset;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1329 return 0;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1330 } else {
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1331 /* use cache to skip over some records */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1332 cur_offset = cache->offset;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1333 cur_modseq = cache->highest_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1334 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1335
22430 2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1336 if ((ret = get_modseq_next_offset_at(file, modseq, TRUE, &cur_offset,
22429 3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1337 &cur_modseq, next_offset_r)) <= 0)
3ba9331e880d lib-index: Code cleanup - move code to get_modseq_next_offset_at() Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22422 diff changeset	1338 return ret;
9691 b09d9350a2d9 Added ability to specify message's minimum modseq value. Timo Sirainen <tss@iki.fi> parents: 9624 diff changeset	1339 if (cur_offset == file->sync_offset) {
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1340 /* if we got to sync_offset, cur_modseq should be
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1341 sync_highest_modseq */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1342 mail_index_set_error(file->log->index,
22430 2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1343 "%s: Transaction log modseq tracking is corrupted - fixing",
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1344 file->filepath);
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1345 /* retry getting the offset by reading from the beginning
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1346 of the file */
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1347 cur_offset = file->hdr.hdr_size;
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1348 cur_modseq = file->hdr.initial_modseq;
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1349 ret = get_modseq_next_offset_at(file, modseq, FALSE,
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1350 &cur_offset, &cur_modseq,
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1351 next_offset_r);
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1352 if (ret < 0)
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1353 return -1;
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1354 i_assert(ret != 0);
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1355 /* get it fixed on the next sync */
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1356 file->log->index->need_recreate = TRUE;
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1357 file->need_rotate = TRUE;
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1358 /* clear cache, since it's unreliable */
2d9175af5c99 lib-index: Fix wrong mail_index_modseq_header automatically Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22429 diff changeset	1359 memset(file->modseq_cache, 0, sizeof(file->modseq_cache));
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1360 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1361
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1362 /* @UNSAFE: cache the value */
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1363 memmove(file->modseq_cache + 1, file->modseq_cache,
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1364 sizeof(file->modseq_cache)
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1365 (N_ELEMENTS(file->modseq_cache) - 1));
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1366 file->modseq_cache[0].offset = cur_offset;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1367 file->modseq_cache[0].highest_modseq = cur_modseq;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1368
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1369 *next_offset_r = cur_offset;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1370 return 0;
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1371 }
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1372
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1373 static int
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1374 log_file_track_sync(struct mail_transaction_log_file *file,
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1375 const struct mail_transaction_header *hdr,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1376 unsigned int trans_size, const char **error_r)
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1377 {
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1378 const void *data = hdr + 1;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1379 int ret;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1380
21876 ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1381 mail_transaction_update_modseq(hdr, hdr + 1, &file->sync_highest_modseq,
ccbdafa83dbc lib-index: Don't increase modseq for backend/dirty flag changes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21875 diff changeset	1382 MAIL_TRANSACTION_LOG_HDR_VERSION(&file->hdr));
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1383 if ((hdr->type & MAIL_TRANSACTION_EXTERNAL) == 0)
12493 b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1384 return 1;
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1385
d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1386 /* external transactions: */
10655 fc0ac73f0b36 Added support for marking mailbox index deleted. Don't allow any changes after that. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	1387 switch (hdr->type & MAIL_TRANSACTION_TYPE_MASK) {
fc0ac73f0b36 Added support for marking mailbox index deleted. Don't allow any changes after that. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	1388 case MAIL_TRANSACTION_HEADER_UPDATE:
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1389 /* see if this updates mailbox_sync_offset */
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1390 ret = log_file_track_mailbox_sync_offset_hdr(file, data,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1391 trans_size, error_r);
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1392 if (ret != 0)
12493 b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1393 return ret < 0 ? -1 : 1;
10655 fc0ac73f0b36 Added support for marking mailbox index deleted. Don't allow any changes after that. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	1394 break;
fc0ac73f0b36 Added support for marking mailbox index deleted. Don't allow any changes after that. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	1395 case MAIL_TRANSACTION_INDEX_DELETED:
10674 22354f505277 lib-index: Added support for undeleting a deleted index. Timo Sirainen <tss@iki.fi> parents: 10655 diff changeset	1396 if (file->sync_offset < file->index_undeleted_offset)
22354f505277 lib-index: Added support for undeleting a deleted index. Timo Sirainen <tss@iki.fi> parents: 10655 diff changeset	1397 break;
10655 fc0ac73f0b36 Added support for marking mailbox index deleted. Don't allow any changes after that. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	1398 file->log->index->index_deleted = TRUE;
17758 6980c53bf7d7 lib-index, lib-storage: Fixed race conditions with deleting mailbox. Timo Sirainen <tss@iki.fi> parents: 17609 diff changeset	1399 file->log->index->index_delete_requested = FALSE;
10655 fc0ac73f0b36 Added support for marking mailbox index deleted. Don't allow any changes after that. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	1400 file->index_deleted_offset = file->sync_offset + trans_size;
fc0ac73f0b36 Added support for marking mailbox index deleted. Don't allow any changes after that. Timo Sirainen <tss@iki.fi> parents: 10582 diff changeset	1401 break;
10674 22354f505277 lib-index: Added support for undeleting a deleted index. Timo Sirainen <tss@iki.fi> parents: 10655 diff changeset	1402 case MAIL_TRANSACTION_INDEX_UNDELETED:
22354f505277 lib-index: Added support for undeleting a deleted index. Timo Sirainen <tss@iki.fi> parents: 10655 diff changeset	1403 if (file->sync_offset < file->index_deleted_offset)
22354f505277 lib-index: Added support for undeleting a deleted index. Timo Sirainen <tss@iki.fi> parents: 10655 diff changeset	1404 break;
22354f505277 lib-index: Added support for undeleting a deleted index. Timo Sirainen <tss@iki.fi> parents: 10655 diff changeset	1405 file->log->index->index_deleted = FALSE;
22354f505277 lib-index: Added support for undeleting a deleted index. Timo Sirainen <tss@iki.fi> parents: 10655 diff changeset	1406 file->log->index->index_delete_requested = FALSE;
22354f505277 lib-index: Added support for undeleting a deleted index. Timo Sirainen <tss@iki.fi> parents: 10655 diff changeset	1407 file->index_undeleted_offset = file->sync_offset + trans_size;
22354f505277 lib-index: Added support for undeleting a deleted index. Timo Sirainen <tss@iki.fi> parents: 10655 diff changeset	1408 break;
12493 b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1409 case MAIL_TRANSACTION_BOUNDARY: {
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1410 const struct mail_transaction_boundary *boundary =
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1411 (const void *)(hdr + 1);
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1412 size_t wanted_buffer_size;
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1413
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1414 wanted_buffer_size = file->sync_offset - file->buffer_offset +
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1415 boundary->size;
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1416 if (wanted_buffer_size > file->buffer->used) {
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1417 /* the full transaction hasn't been written yet */
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1418 return 0;
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1419 }
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1420 break;
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1421 }
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1422 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1423
5749 420a386fa27a int/ext/mailbox sync offset changes: Combined mailbox and int offsets to Timo Sirainen <tss@iki.fi> parents: 5723 diff changeset	1424 if (file->max_tail_offset == file->sync_offset) {
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1425 /* external transactions aren't synced to mailbox. we can
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1426 update mailbox sync offset to skip this transaction to
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1427 avoid re-reading it at the next sync. */
5749 420a386fa27a int/ext/mailbox sync offset changes: Combined mailbox and int offsets to Timo Sirainen <tss@iki.fi> parents: 5723 diff changeset	1428 file->max_tail_offset += trans_size;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1429 }
12493 b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1430 return 1;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1431 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1432
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1433 static int
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1434 mail_transaction_log_file_sync(struct mail_transaction_log_file *file,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1435 bool retry_r, const char *reason_r)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1436 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1437 const struct mail_transaction_header *hdr;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1438 const void *data;
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1439 struct stat st;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1440 size_t size, avail;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1441 uint32_t trans_size = 0;
12493 b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1442 int ret;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1443
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1444 i_assert(file->sync_offset >= file->buffer_offset);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1445
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1446 *retry_r = FALSE;
bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1447
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1448 data = buffer_get_data(file->buffer, &size);
11313 361157623f2a lib-index: Give a better error message if transction log file shrinks unexpectedly. Timo Sirainen <tss@iki.fi> parents: 11310 diff changeset	1449 if (file->buffer_offset + size < file->sync_offset) {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1450 *reason_r = t_strdup_printf(
11313 361157623f2a lib-index: Give a better error message if transction log file shrinks unexpectedly. Timo Sirainen <tss@iki.fi> parents: 11310 diff changeset	1451 "log file shrank (%"PRIuUOFF_T" < %"PRIuUOFF_T")",
361157623f2a lib-index: Give a better error message if transction log file shrinks unexpectedly. Timo Sirainen <tss@iki.fi> parents: 11310 diff changeset	1452 file->buffer_offset + (uoff_t)size, file->sync_offset);
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1453 mail_transaction_log_file_set_corrupted(file, "%s", *reason_r);
21237 a0be55f275c5 lib-index: Fix assert-crash after "log file shrank" error. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21187 diff changeset	1454 /* fix the sync_offset to avoid crashes later on */
a0be55f275c5 lib-index: Fix assert-crash after "log file shrank" error. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21187 diff changeset	1455 file->sync_offset = file->buffer_offset + size;
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1456 return 0;
11313 361157623f2a lib-index: Give a better error message if transction log file shrinks unexpectedly. Timo Sirainen <tss@iki.fi> parents: 11310 diff changeset	1457 }
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1458 while (file->sync_offset - file->buffer_offset + sizeof(*hdr) <= size) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1459 hdr = CONST_PTR_OFFSET(data, file->sync_offset -
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1460 file->buffer_offset);
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1461 trans_size = mail_index_offset_to_uint32(hdr->size);
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1462 if (trans_size == 0) {
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1463 /* unfinished */
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1464 return 1;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1465 }
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1466 if (trans_size < sizeof(*hdr)) {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1467 *reason_r = t_strdup_printf(
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1468 "hdr.size too small (%u)", trans_size);
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1469 mail_transaction_log_file_set_corrupted(file, "%s", *reason_r);
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1470 return 0;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1471 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1472
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1473 if (file->sync_offset - file->buffer_offset + trans_size > size)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1474 break;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1475
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1476 /* transaction has been fully written */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1477 if ((ret = log_file_track_sync(file, hdr, trans_size, reason_r)) <= 0) {
12493 b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1478 if (ret < 0)
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1479 return 0;
12493 b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1480 break;
b7d13ee51aa4 lib-index: Added "transaction boundary" record for future compatibility. Timo Sirainen <tss@iki.fi> parents: 12395 diff changeset	1481 }
7812 d10cb44ab446 Modseqs are no longer calculated from transaction log sequence + offset. Timo Sirainen <tss@iki.fi> parents: 7573 diff changeset	1482
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1483 file->sync_offset += trans_size;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1484 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1485
5796 e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1486 if (file->mmap_base != NULL && !file->locked) {
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1487 /* Now that all the mmaped pages have page faulted, check if
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1488 the file had changed while doing that. Only after the last
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1489 page has faulted, the size returned by fstat() can be
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1490 trusted. Otherwise it might point to a page boundary while
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1491 the next page is still being written.
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1492
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1493 Without this check we might see partial transactions,
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1494 sometimes causing "Extension record updated without intro
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1495 prefix" errors. */
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1496 if (fstat(file->fd, &st) < 0) {
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1497 log_file_set_syscall_error(file, "fstat()");
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1498 *reason_r = t_strdup_printf("fstat() failed: %m");
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1499 return -1;
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1500 }
5796 e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1501 if ((uoff_t)st.st_size != file->last_size) {
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1502 file->last_size = st.st_size;
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1503 *retry_r = TRUE;
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1504 *reason_r = "File size changed - retrying";
5796 e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1505 return 0;
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1506 }
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1507 }
e5da155f44c3 Better race condition fix for mmaped log files Timo Sirainen <tss@iki.fi> parents: 5795 diff changeset	1508
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1509 avail = file->sync_offset - file->buffer_offset;
5791 2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1510 if (avail != size) {
2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1511 /* There's more data than we could sync at the moment. If the
2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1512 last record's size wasn't valid, we can't know if it will
6761 d0a2e137f044 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6645 diff changeset	1513 be updated unless we've locked the log. */
15367 751181168b72 lib-index: Write to transaction log using O_APPEND flag. Timo Sirainen <tss@iki.fi> parents: 15337 diff changeset	1514 if (file->locked) {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1515 *reason_r = "Unexpected garbage at EOF";
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1516 mail_transaction_log_file_set_corrupted(file, "%s", *reason_r);
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1517 return 0;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1518 }
6829 dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	1519 /* The size field will be updated soon */
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	1520 mail_index_flush_read_cache(file->log->index, file->filepath,
dbab5e592577 NFS cache flushing APIs changed and backend implements it a bit differently. Timo Sirainen <tss@iki.fi> parents: 6819 diff changeset	1521 file->fd, file->locked);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1522 }
5792 a250c1a48471 Log an error if non-head log file couldn't be synced to the next file's Timo Sirainen <tss@iki.fi> parents: 5791 diff changeset	1523
a250c1a48471 Log an error if non-head log file couldn't be synced to the next file's Timo Sirainen <tss@iki.fi> parents: 5791 diff changeset	1524 if (file->next != NULL &&
a250c1a48471 Log an error if non-head log file couldn't be synced to the next file's Timo Sirainen <tss@iki.fi> parents: 5791 diff changeset	1525 file->hdr.file_seq == file->next->hdr.prev_file_seq &&
a250c1a48471 Log an error if non-head log file couldn't be synced to the next file's Timo Sirainen <tss@iki.fi> parents: 5791 diff changeset	1526 file->next->hdr.prev_file_offset != file->sync_offset) {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1527 *reason_r = t_strdup_printf(
5792 a250c1a48471 Log an error if non-head log file couldn't be synced to the next file's Timo Sirainen <tss@iki.fi> parents: 5791 diff changeset	1528 "Invalid transaction log size "
a250c1a48471 Log an error if non-head log file couldn't be synced to the next file's Timo Sirainen <tss@iki.fi> parents: 5791 diff changeset	1529 "(%"PRIuUOFF_T" vs %u): %s", file->sync_offset,
a250c1a48471 Log an error if non-head log file couldn't be synced to the next file's Timo Sirainen <tss@iki.fi> parents: 5791 diff changeset	1530 file->log->head->hdr.prev_file_offset, file->filepath);
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1531 mail_transaction_log_file_set_corrupted(file, "%s", *reason_r);
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1532 return 0;
5792 a250c1a48471 Log an error if non-head log file couldn't be synced to the next file's Timo Sirainen <tss@iki.fi> parents: 5791 diff changeset	1533 }
a250c1a48471 Log an error if non-head log file couldn't be synced to the next file's Timo Sirainen <tss@iki.fi> parents: 5791 diff changeset	1534
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1535 return 1;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1536 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1537
5714 dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1538 static int
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1539 mail_transaction_log_file_insert_read(struct mail_transaction_log_file *file,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1540 uoff_t offset, const char **reason_r)
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1541 {
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1542 void *data;
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1543 size_t size;
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1544 ssize_t ret;
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1545
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1546 size = file->buffer_offset - offset;
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1547 buffer_copy(file->buffer, size, file->buffer, 0, (size_t)-1);
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1548
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1549 data = buffer_get_space_unsafe(file->buffer, 0, size);
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1550 ret = pread_full(file->fd, data, size, offset);
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1551 if (ret > 0) {
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1552 /* success */
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1553 file->buffer_offset -= size;
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1554 return 1;
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1555 }
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1556
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1557 /* failure. don't leave ourself to inconsistent state */
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1558 buffer_copy(file->buffer, 0, file->buffer, size, (size_t)-1);
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1559 buffer_set_used_size(file->buffer, file->buffer->used - size);
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1560
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1561 if (ret == 0) {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1562 *reason_r = "file shrank unexpectedly";
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1563 mail_transaction_log_file_set_corrupted(file, "%s", *reason_r);
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1564 return 0;
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1565 } else if (errno == ESTALE) {
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1566 /* log file was deleted in NFS server, fail silently */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1567 *reason_r = t_strdup_printf("read() failed: %m");
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1568 return 0;
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1569 } else {
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1570 log_file_set_syscall_error(file, "pread()");
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1571 *reason_r = t_strdup_printf("read() failed: %m");
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1572 return -1;
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1573 }
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1574 }
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1575
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1576 static int
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1577 mail_transaction_log_file_read_more(struct mail_transaction_log_file *file,
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1578 const char **reason_r)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1579 {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1580 void *data;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1581 size_t size;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1582 uint32_t read_offset;
6002 16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1583 ssize_t ret;
5972 9fb9dc4d8df8 Flush NFS caches when needed if MAIL_INDEX_OPEN_FLAG_NFS_FLUSH is enabled. Timo Sirainen <tss@iki.fi> parents: 5953 diff changeset	1584
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1585 read_offset = file->buffer_offset + buffer_get_used_size(file->buffer);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1586
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1587 do {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1588 data = buffer_append_space_unsafe(file->buffer, LOG_PREFETCH);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1589 ret = pread(file->fd, data, LOG_PREFETCH, read_offset);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1590 if (ret > 0)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1591 read_offset += ret;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1592
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1593 size = read_offset - file->buffer_offset;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1594 buffer_set_used_size(file->buffer, size);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1595 } while (ret > 0 \|\| (ret < 0 && errno == EINTR));
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1596
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1597 file->last_size = read_offset;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1598
5791 2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1599 if (ret < 0) {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1600 *reason_r = t_strdup_printf("pread() failed: %m");
5791 2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1601 if (errno == ESTALE) {
2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1602 /* log file was deleted in NFS server, fail silently */
2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1603 return 0;
2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1604 }
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1605 log_file_set_syscall_error(file, "pread()");
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1606 return -1;
5791 2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1607 }
6002 16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1608 return 1;
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1609 }
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1610
6802 ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1611 static bool
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1612 mail_transaction_log_file_need_nfs_flush(struct mail_transaction_log_file *file)
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1613 {
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1614 const struct mail_index_header *hdr = &file->log->index->map->hdr;
6807 5382964fc01d Avoid flushing attribute cache when checking to see if there's new data in Timo Sirainen <tss@iki.fi> parents: 6802 diff changeset	1615 uoff_t max_offset = file->last_size;
6802 ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1616
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1617 if (file->next != NULL &&
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1618 file->hdr.file_seq == file->next->hdr.prev_file_seq &&
6807 5382964fc01d Avoid flushing attribute cache when checking to see if there's new data in Timo Sirainen <tss@iki.fi> parents: 6802 diff changeset	1619 file->next->hdr.prev_file_offset != max_offset) {
6802 ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1620 /* we already have a newer log file which says that we haven't
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1621 synced the entire file. */
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1622 return TRUE;
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1623 }
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1624
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1625 if (file->hdr.file_seq == hdr->log_file_seq &&
6807 5382964fc01d Avoid flushing attribute cache when checking to see if there's new data in Timo Sirainen <tss@iki.fi> parents: 6802 diff changeset	1626 max_offset < hdr->log_file_head_offset)
6802 ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1627 return TRUE;
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1628
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1629 return FALSE;
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1630 }
ba58eaf06763 Keep better track of when we need to flush NFS attribute caches. Timo Sirainen <tss@iki.fi> parents: 6798 diff changeset	1631
6002 16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1632 static int
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1633 mail_transaction_log_file_read(struct mail_transaction_log_file *file,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1634 uoff_t start_offset, bool nfs_flush,
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1635 const char **reason_r)
6002 16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1636 {
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1637 bool retry;
6002 16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1638 int ret;
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1639
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1640 i_assert(file->mmap_base == NULL);
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1641
6808 937faba78a68 Don't flush attribute cache twice after locking transaction log. Timo Sirainen <tss@iki.fi> parents: 6807 diff changeset	1642 /* NFS: if file isn't locked, we're optimistic that we can read enough
937faba78a68 Don't flush attribute cache twice after locking transaction log. Timo Sirainen <tss@iki.fi> parents: 6807 diff changeset	1643 data without flushing attribute cache. if after reading we notice
937faba78a68 Don't flush attribute cache twice after locking transaction log. Timo Sirainen <tss@iki.fi> parents: 6807 diff changeset	1644 that we really should have read more, flush the cache and try again.
937faba78a68 Don't flush attribute cache twice after locking transaction log. Timo Sirainen <tss@iki.fi> parents: 6807 diff changeset	1645 if file is locked, the attribute cache was already flushed when
937faba78a68 Don't flush attribute cache twice after locking transaction log. Timo Sirainen <tss@iki.fi> parents: 6807 diff changeset	1646 refreshing the log. */
11634 7f19062f58fd Renamed fsync_disable to mail_fsync=optimized\|always\|never. Timo Sirainen <tss@iki.fi> parents: 11623 diff changeset	1647 if (file->log->nfs_flush && nfs_flush) {
6928 fe777302d928 NFS attribute cache flushing fix Timo Sirainen <tss@iki.fi> parents: 6921 diff changeset	1648 if (!file->locked)
fe777302d928 NFS attribute cache flushing fix Timo Sirainen <tss@iki.fi> parents: 6921 diff changeset	1649 nfs_flush_attr_cache_unlocked(file->filepath);
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1650 else
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1651 nfs_flush_attr_cache_fd_locked(file->filepath, file->fd);
6928 fe777302d928 NFS attribute cache flushing fix Timo Sirainen <tss@iki.fi> parents: 6921 diff changeset	1652 }
6002 16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1653
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1654 if (file->buffer != NULL && file->buffer_offset > start_offset) {
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1655 /* we have to insert missing data to beginning of buffer */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1656 ret = mail_transaction_log_file_insert_read(file, start_offset, reason_r);
6002 16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1657 if (ret <= 0)
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1658 return ret;
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1659 }
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1660
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1661 if (file->buffer == NULL) {
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1662 file->buffer =
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1663 buffer_create_dynamic(default_pool, LOG_PREFETCH);
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1664 file->buffer_offset = start_offset;
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1665 }
16f91815c7ed Code cleanup Timo Sirainen <tss@iki.fi> parents: 5996 diff changeset	1666
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1667 if ((ret = mail_transaction_log_file_read_more(file, reason_r)) <= 0)
12395 878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	1668 ;
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	1669 else if (file->log->nfs_flush && !nfs_flush &&
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	1670 mail_transaction_log_file_need_nfs_flush(file)) {
6807 5382964fc01d Avoid flushing attribute cache when checking to see if there's new data in Timo Sirainen <tss@iki.fi> parents: 6802 diff changeset	1671 /* we didn't read enough data. flush and try again. */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1672 return mail_transaction_log_file_read(file, start_offset, TRUE, reason_r);
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1673 } else if ((ret = mail_transaction_log_file_sync(file, &retry, reason_r)) == 0) {
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1674 i_assert(!retry); /* retry happens only with mmap */
6807 5382964fc01d Avoid flushing attribute cache when checking to see if there's new data in Timo Sirainen <tss@iki.fi> parents: 6802 diff changeset	1675 }
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1676 i_assert(file->sync_offset >= file->buffer_offset);
5791 2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1677 buffer_set_used_size(file->buffer,
2a05cff7eca4 Handle partial reads with mmap_disable=no correctly. Timo Sirainen <tss@iki.fi> parents: 5787 diff changeset	1678 file->sync_offset - file->buffer_offset);
12395 878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	1679 return ret;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1680 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1681
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1682 static bool
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1683 log_file_map_check_offsets(struct mail_transaction_log_file *file,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1684 uoff_t start_offset, uoff_t end_offset,
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1685 const char **reason_r)
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1686 {
22063 de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1687 struct stat st, st2;
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1688
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1689 if (start_offset > file->sync_offset) {
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1690 /* broken start offset */
22073 ade8986248dd lib-index: Fix start_offset > sync_offset error handling for in-memory indexes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22063 diff changeset	1691 if (MAIL_TRANSACTION_LOG_FILE_IN_MEMORY(file)) {
ade8986248dd lib-index: Fix start_offset > sync_offset error handling for in-memory indexes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22063 diff changeset	1692 *reason_r = t_strdup_printf(
ade8986248dd lib-index: Fix start_offset > sync_offset error handling for in-memory indexes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22063 diff changeset	1693 "%s: start_offset (%"PRIuUOFF_T") > "
ade8986248dd lib-index: Fix start_offset > sync_offset error handling for in-memory indexes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22063 diff changeset	1694 "current sync_offset (%"PRIuUOFF_T")",
ade8986248dd lib-index: Fix start_offset > sync_offset error handling for in-memory indexes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22063 diff changeset	1695 file->filepath, start_offset, file->sync_offset);
ade8986248dd lib-index: Fix start_offset > sync_offset error handling for in-memory indexes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22063 diff changeset	1696 return FALSE;
ade8986248dd lib-index: Fix start_offset > sync_offset error handling for in-memory indexes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22063 diff changeset	1697 }
ade8986248dd lib-index: Fix start_offset > sync_offset error handling for in-memory indexes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22063 diff changeset	1698
22063 de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1699 if (fstat(file->fd, &st) < 0) {
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1700 log_file_set_syscall_error(file, "fstat()");
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1701 st.st_size = -1;
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1702 }
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1703 *reason_r = t_strdup_printf(
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1704 "%s: start_offset (%"PRIuUOFF_T") > "
22063 de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1705 "current sync_offset (%"PRIuUOFF_T"), file size=%"PRIuUOFF_T,
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1706 file->filepath, start_offset, file->sync_offset,
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1707 st.st_size);
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1708 if (stat(file->filepath, &st2) == 0) {
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1709 if (st.st_ino != st2.st_ino) {
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1710 *reason_r = t_strdup_printf(
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1711 "%s, file unexpectedly replaced", *reason_r);
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1712 }
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1713 } else if (errno == ENOENT) {
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1714 *reason_r = t_strdup_printf(
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1715 "%s, file unexpectedly deleted", *reason_r);
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1716 } else {
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1717 log_file_set_syscall_error(file, "stat()");
de0cf622817b lib-index: Improve "start_offset > current sync_offset" error message. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22062 diff changeset	1718 }
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1719 return FALSE;
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1720 }
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1721 if (end_offset != (uoff_t)-1 && end_offset > file->sync_offset) {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1722 *reason_r = t_strdup_printf(
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1723 "%s: end_offset (%"PRIuUOFF_T") > "
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1724 "current sync_offset (%"PRIuUOFF_T")",
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1725 file->filepath, start_offset, file->sync_offset);
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1726 return FALSE;
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1727 }
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1728
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1729 return TRUE;
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1730 }
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1731
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1732 static int
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1733 mail_transaction_log_file_mmap(struct mail_transaction_log_file *file,
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1734 const char **reason_r)
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1735 {
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1736 if (file->buffer != NULL) {
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1737 /* in case we just switched to mmaping */
6414 a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6360 diff changeset	1738 buffer_free(&file->buffer);
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1739 }
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1740 file->mmap_size = file->last_size;
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1741 file->mmap_base = mmap(NULL, file->mmap_size, PROT_READ, MAP_SHARED,
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1742 file->fd, 0);
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1743 if (file->mmap_base == MAP_FAILED) {
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1744 file->mmap_base = NULL;
21187 8117c1c2d3f6 lib-index: Limit mmap errors in txn log file to 1/s Aki Tuomi <aki.tuomi@dovecot.fi> parents: 21172 diff changeset	1745 if (ioloop_time != file->last_mmap_error_time) {
21524 c14dda2ca252 lib-index: don't reset ioloop_time on mmap error Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21390 diff changeset	1746 file->last_mmap_error_time = ioloop_time;
21187 8117c1c2d3f6 lib-index: Limit mmap errors in txn log file to 1/s Aki Tuomi <aki.tuomi@dovecot.fi> parents: 21172 diff changeset	1747 log_file_set_syscall_error(file, t_strdup_printf(
8117c1c2d3f6 lib-index: Limit mmap errors in txn log file to 1/s Aki Tuomi <aki.tuomi@dovecot.fi> parents: 21172 diff changeset	1748 "mmap(size=%"PRIuSIZE_T")", file->mmap_size));
8117c1c2d3f6 lib-index: Limit mmap errors in txn log file to 1/s Aki Tuomi <aki.tuomi@dovecot.fi> parents: 21172 diff changeset	1749 }
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1750 *reason_r = t_strdup_printf("mmap(size=%"PRIuSIZE_T") failed: %m",
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1751 file->mmap_size);
21525 cea8065bd571 lib-index: clear file->mmap_size only after logging it Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21524 diff changeset	1752 file->mmap_size = 0;
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1753 return -1;
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1754 }
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1755
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1756 if (file->mmap_size > mmap_get_page_size()) {
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1757 if (madvise(file->mmap_base, file->mmap_size,
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	1758 MADV_SEQUENTIAL) < 0)
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	1759 log_file_set_syscall_error(file, "madvise()");
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1760 }
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1761
15034 7efef678bca8 Renamed buffer_create_data() to buffer_create_from_data() for consistency. Timo Sirainen <tss@iki.fi> parents: 14688 diff changeset	1762 buffer_create_from_const_data(&file->mmap_buffer,
7efef678bca8 Renamed buffer_create_data() to buffer_create_from_data() for consistency. Timo Sirainen <tss@iki.fi> parents: 14688 diff changeset	1763 file->mmap_base, file->mmap_size);
9623 3da42dafa798 buffer_create_[const_]data() API change: Take buffer_t as parameter instead of allocating it. Timo Sirainen <tss@iki.fi> parents: 9420 diff changeset	1764 file->buffer = &file->mmap_buffer;
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1765 file->buffer_offset = 0;
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1766 return 0;
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1767 }
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1768
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1769 static void
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1770 mail_transaction_log_file_munmap(struct mail_transaction_log_file *file)
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1771 {
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1772 if (file->mmap_base == NULL)
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1773 return;
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1774
18619 49f5131a502e Make Coverity happier. Timo Sirainen <tss@iki.fi> parents: 18276 diff changeset	1775 i_assert(file->buffer != NULL);
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	1776 if (munmap(file->mmap_base, file->mmap_size) < 0)
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	1777 log_file_set_syscall_error(file, "munmap()");
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1778 file->mmap_base = NULL;
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1779 file->mmap_size = 0;
6414 a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6360 diff changeset	1780 buffer_free(&file->buffer);
5795 0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1781 }
0c0a829a9a63 Handle race conditions in mmap()ed transaction logs better. Timo Sirainen <tss@iki.fi> parents: 5792 diff changeset	1782
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1783 static int
6929 8c80093fbab4 Don't mmap() transaction log file if we're reading less than a page. Timo Sirainen <tss@iki.fi> parents: 6928 diff changeset	1784 mail_transaction_log_file_map_mmap(struct mail_transaction_log_file *file,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1785 uoff_t start_offset, const char **reason_r)
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1786 {
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1787 struct stat st;
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1788 bool retry;
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1789 int ret;
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1790
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1791 /* we are going to mmap() this file, but it's not necessarily
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1792 mmaped currently. */
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1793 i_assert(file->buffer_offset == 0 \|\| file->mmap_base == NULL);
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1794 i_assert(file->mmap_size == 0 \|\| file->mmap_base != NULL);
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1795
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1796 if (fstat(file->fd, &st) < 0) {
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1797 log_file_set_syscall_error(file, "fstat()");
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1798 *reason_r = t_strdup_printf("fstat() failed: %m");
14682 d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1799 return -1;
d0d7b810646b Make sure we check all the functions' return values. Minor API changes to simplify this. Timo Sirainen <tss@iki.fi> parents: 14674 diff changeset	1800 }
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1801 file->last_size = st.st_size;
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1802
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1803 if ((uoff_t)st.st_size < file->sync_offset) {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1804 *reason_r = t_strdup_printf(
12389 16abe905f897 lib-index: Minor error message improvement. Timo Sirainen <tss@iki.fi> parents: 12249 diff changeset	1805 "file size shrank (%"PRIuUOFF_T" < %"PRIuUOFF_T")",
16abe905f897 lib-index: Minor error message improvement. Timo Sirainen <tss@iki.fi> parents: 12249 diff changeset	1806 (uoff_t)st.st_size, file->sync_offset);
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1807 mail_transaction_log_file_set_corrupted(file, "%s", *reason_r);
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1808 return 0;
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1809 }
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1810
7439 dc9c098a7d26 Fix to previous optimization: Don't crash if we want to read older data from Timo Sirainen <tss@iki.fi> parents: 7435 diff changeset	1811 if (file->buffer != NULL && file->buffer_offset <= start_offset &&
7435 6983dfc231d7 Small optimization: Don't try to pread() log file if we already know we've Timo Sirainen <tss@iki.fi> parents: 7359 diff changeset	1812 (uoff_t)st.st_size == file->buffer_offset + file->buffer->used) {
6983dfc231d7 Small optimization: Don't try to pread() log file if we already know we've Timo Sirainen <tss@iki.fi> parents: 7359 diff changeset	1813 /* we already have the whole file mapped */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1814 if ((ret = mail_transaction_log_file_sync(file, &retry, reason_r)) != 0 \|\|
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1815 !retry)
bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1816 return ret;
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1817 /* size changed, re-mmap */
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1818 }
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1819
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1820 do {
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1821 mail_transaction_log_file_munmap(file);
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1822
6929 8c80093fbab4 Don't mmap() transaction log file if we're reading less than a page. Timo Sirainen <tss@iki.fi> parents: 6928 diff changeset	1823 if (file->last_size - start_offset < mmap_get_page_size()) {
8c80093fbab4 Don't mmap() transaction log file if we're reading less than a page. Timo Sirainen <tss@iki.fi> parents: 6928 diff changeset	1824 /* just reading the file is probably faster */
8c80093fbab4 Don't mmap() transaction log file if we're reading less than a page. Timo Sirainen <tss@iki.fi> parents: 6928 diff changeset	1825 return mail_transaction_log_file_read(file,
8c80093fbab4 Don't mmap() transaction log file if we're reading less than a page. Timo Sirainen <tss@iki.fi> parents: 6928 diff changeset	1826 start_offset,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1827 FALSE, reason_r);
6929 8c80093fbab4 Don't mmap() transaction log file if we're reading less than a page. Timo Sirainen <tss@iki.fi> parents: 6928 diff changeset	1828 }
8c80093fbab4 Don't mmap() transaction log file if we're reading less than a page. Timo Sirainen <tss@iki.fi> parents: 6928 diff changeset	1829
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1830 if (mail_transaction_log_file_mmap(file, reason_r) < 0)
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1831 return -1;
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1832 ret = mail_transaction_log_file_sync(file, &retry, reason_r);
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1833 } while (retry);
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1834
21238 bfbe1f6b8ab6 lib-index: mail_transaction_log_file_sync(): Don't mix I/O errors and corruption Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21237 diff changeset	1835 return ret;
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1836 }
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1837
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1838 int mail_transaction_log_file_map(struct mail_transaction_log_file *file,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1839 uoff_t start_offset, uoff_t end_offset,
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1840 const char **reason_r)
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1841 {
19457 a47d273e144f lib-index: mail_transaction_log_file_map() didn't properly check start_offset Timo Sirainen <tss@iki.fi> parents: 19137 diff changeset	1842 uoff_t map_start_offset = start_offset;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1843 size_t size;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1844 int ret;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1845
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1846 if (file->hdr.indexid == 0) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1847 /* corrupted */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1848 *reason_r = "corrupted, indexid=0";
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1849 return 0;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1850 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1851
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1852 i_assert(start_offset >= file->hdr.hdr_size);
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1853 i_assert(start_offset <= end_offset);
12395 878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	1854 i_assert(file->buffer == NULL \|\| file->mmap_base != NULL \|\|
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	1855 file->sync_offset >= file->buffer_offset + file->buffer->used);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1856
12249 41e99ee5c1dd lib-index: Avoid unnecessary pread()s at end of transaction log file. Timo Sirainen <tss@iki.fi> parents: 12204 diff changeset	1857 if (file->locked_sync_offset_updated && file == file->log->head &&
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1858 end_offset == (uoff_t)-1) {
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1859 /* we're not interested of going further than sync_offset */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1860 if (!log_file_map_check_offsets(file, start_offset,
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1861 end_offset, reason_r))
5786 b6cf05796f6d If log offsets are broken, don't assert-crash. Timo Sirainen <tss@iki.fi> parents: 5772 diff changeset	1862 return 0;
5689 c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1863 i_assert(start_offset <= file->sync_offset);
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1864 end_offset = file->sync_offset;
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1865 }
c2362f144f15 Initial commit for major index file code cleanup. Timo Sirainen <tss@iki.fi> parents: 5687 diff changeset	1866
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1867 if (file->buffer != NULL && file->buffer_offset <= start_offset) {
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1868 /* see if we already have it */
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1869 size = buffer_get_used_size(file->buffer);
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1870 if (file->buffer_offset + size >= end_offset)
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1871 return 1;
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1872 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1873
12394 7f8f73bed7f9 lib-index: Fix to transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12389 diff changeset	1874 if (file->locked) {
7f8f73bed7f9 lib-index: Fix to transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12389 diff changeset	1875 /* set this only when we've synced to end of file while locked
7f8f73bed7f9 lib-index: Fix to transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12389 diff changeset	1876 (either end_offset=(uoff_t)-1 or we had to read anyway) */
7f8f73bed7f9 lib-index: Fix to transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12389 diff changeset	1877 file->locked_sync_offset_updated = TRUE;
7f8f73bed7f9 lib-index: Fix to transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12389 diff changeset	1878 }
7f8f73bed7f9 lib-index: Fix to transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12389 diff changeset	1879
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1880 if (MAIL_TRANSACTION_LOG_FILE_IN_MEMORY(file)) {
15951 e33fe1a7bb89 lib-index: Fixed assert-crash on some error conditions. Timo Sirainen <tss@iki.fi> parents: 15316 diff changeset	1881 if (start_offset < file->buffer_offset \|\| file->buffer == NULL) {
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1882 /* we had moved the log to memory but failed to read
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1883 the beginning of the log file */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1884 *reason_r = "Beginning of the log isn't available";
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1885 return 0;
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1886 }
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1887 return log_file_map_check_offsets(file, start_offset,
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1888 end_offset, reason_r) ? 1 : 0;
5716 4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1889 }
4a08705e97f5 Fixes to mail_transaction_log_file_map() error handling. Timo Sirainen <tss@iki.fi> parents: 5714 diff changeset	1890
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1891 if (start_offset > file->sync_offset)
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1892 mail_transaction_log_file_skip_to_head(file);
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1893 if (start_offset > file->sync_offset) {
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1894 /* although we could just skip over the unwanted data, we have
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1895 to sync everything so that modseqs are calculated
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1896 correctly */
19457 a47d273e144f lib-index: mail_transaction_log_file_map() didn't properly check start_offset Timo Sirainen <tss@iki.fi> parents: 19137 diff changeset	1897 map_start_offset = file->sync_offset;
7931 502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1898 }
502cfdcc5650 Keep modseqs as 1 until the first modseq ext intro record enables them. Timo Sirainen <tss@iki.fi> parents: 7842 diff changeset	1899
11634 7f19062f58fd Renamed fsync_disable to mail_fsync=optimized\|always\|never. Timo Sirainen <tss@iki.fi> parents: 11623 diff changeset	1900 if ((file->log->index->flags & MAIL_INDEX_OPEN_FLAG_MMAP_DISABLE) == 0)
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1901 ret = mail_transaction_log_file_map_mmap(file, map_start_offset, reason_r);
6921 4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1902 else {
4aa0a3a2d3f8 Code cleanup Timo Sirainen <tss@iki.fi> parents: 6857 diff changeset	1903 mail_transaction_log_file_munmap(file);
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1904 ret = mail_transaction_log_file_read(file, map_start_offset, FALSE, reason_r);
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1905 }
d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1906
12395 878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	1907 i_assert(file->buffer == NULL \|\| file->mmap_base != NULL \|\|
878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	1908 file->sync_offset >= file->buffer_offset + file->buffer->used);
14673 c257b9c19915 Make static analyzer happier. Timo Sirainen <tss@iki.fi> parents: 14435 diff changeset	1909 if (ret <= 0)
c257b9c19915 Make static analyzer happier. Timo Sirainen <tss@iki.fi> parents: 14435 diff changeset	1910 return ret;
12395 878746da7992 lib-index: Fixed another transaction log read optimization. Timo Sirainen <tss@iki.fi> parents: 12394 diff changeset	1911
14673 c257b9c19915 Make static analyzer happier. Timo Sirainen <tss@iki.fi> parents: 14435 diff changeset	1912 i_assert(file->buffer != NULL);
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1913 return log_file_map_check_offsets(file, start_offset, end_offset,
dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1914 reason_r) ? 1 : 0;
5687 d28185a3131a Moved mail transaction log file related code to its own file. Timo Sirainen <tss@iki.fi> parents: diff changeset	1915 }
5714 dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1916
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1917 void mail_transaction_log_file_move_to_memory(struct mail_transaction_log_file
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1918 *file)
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1919 {
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1920 const char *error;
5714 dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1921 buffer_t *buf;
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1922
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1923 if (MAIL_TRANSACTION_LOG_FILE_IN_MEMORY(file))
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1924 return;
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1925
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1926 if (file->mmap_base != NULL) {
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1927 /* just copy to memory */
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1928 i_assert(file->buffer_offset == 0);
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1929
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1930 buf = buffer_create_dynamic(default_pool, file->mmap_size);
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1931 buffer_append(buf, file->mmap_base, file->mmap_size);
6414 a6a49d5efc59 Changed buffer_free() and buffer_free_without_data() APIs to take ** pointer Timo Sirainen <tss@iki.fi> parents: 6360 diff changeset	1932 buffer_free(&file->buffer);
5714 dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1933 file->buffer = buf;
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1934
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1935 /* and lose the mmap */
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	1936 if (munmap(file->mmap_base, file->mmap_size) < 0)
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	1937 log_file_set_syscall_error(file, "munmap()");
5714 dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1938 file->mmap_base = NULL;
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1939 } else if (file->buffer_offset != 0) {
dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1940 /* we don't have the full log in the memory. read it. */
22060 dc04a912eb35 lib-index: mail_transaction_log_file_map() - return reason/error string Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21876 diff changeset	1941 (void)mail_transaction_log_file_read(file, 0, FALSE, &error);
5714 dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1942 }
9097 da5f1b89d4e1 indexes: Error handling fixes. Timo Sirainen <tss@iki.fi> parents: 8972 diff changeset	1943 file->last_size = 0;
5714 dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1944
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	1945 if (close(file->fd) < 0)
6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	1946 log_file_set_syscall_error(file, "close()");
5714 dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1947 file->fd = -1;
8292 9ed4ecd4a866 Fixes to handling "out of disk space/quota" write failures. Timo Sirainen <tss@iki.fi> parents: 8136 diff changeset	1948
9ed4ecd4a866 Fixes to handling "out of disk space/quota" write failures. Timo Sirainen <tss@iki.fi> parents: 8136 diff changeset	1949 i_free(file->filepath);
9697 6e4b79ca75cc index: Code cleanups. Timo Sirainen <tss@iki.fi> parents: 9696 diff changeset	1950 file->filepath = i_strdup(file->log->filepath);
5714 dcf5dbb6a76a mail_transaction_log_move_to_memory() can't fail anymore. Moved the code to Timo Sirainen <tss@iki.fi> parents: 5689 diff changeset	1951 }

Mercurial > dovecot > core-2.2

annotate src/lib-index/mail-transaction-log-file.c @ 22691:dca05b22217b