dovecot/core-2.2: src/plugins/pop3-migration/pop3-migration-plugin.c annotate

annotate src/plugins/pop3-migration/pop3-migration-plugin.c @ 22195:c2a33eba3615

pop3-migration: Strip trailing spaces from headers when calculating hash Fixes matching mails with Zimbra.

author	Timo Sirainen <timo.sirainen@dovecot.fi>
date	Mon, 12 Jun 2017 14:24:59 +0300
parents	e78a42ead488
children	096f0c8b9cb1

rev	line source
21390 2e2563132d5f Updated copyright notices to include the year 2017. Stephan Bosch <stephan.bosch@dovecot.fi> parents: 21389 diff changeset	1 /* Copyright (c) 2007-2017 Dovecot authors, see the included COPYING file */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	2
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	3 #include "lib.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	4 #include "array.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	5 #include "istream.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	6 #include "istream-header-filter.h"
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	7 #include "str.h"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	8 #include "sha1.h"
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	9 #include "message-size.h"
19691 69502e2013f3 pop3-migration: Use message_header_hash_more() instead of duplicating the code Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19672 diff changeset	10 #include "message-header-hash.h"
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	11 #include "message-header-parser.h"
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	12 #include "mail-cache.h"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	13 #include "mail-namespace.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	14 #include "mail-search-build.h"
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	15 #include "index-storage.h"
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	16 #include "index-mail.h"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	17 #include "pop3-migration-plugin.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	18
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	19 #define POP3_MIGRATION_CONTEXT(obj) \
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	20 MODULE_CONTEXT(obj, pop3_migration_storage_module)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	21 #define POP3_MIGRATION_MAIL_CONTEXT(obj) \
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	22 MODULE_CONTEXT(obj, pop3_migration_mail_module)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	23
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	24 struct msg_map_common {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	25 /* sha1(header) - set only when needed */
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	26 unsigned char hdr_sha1[SHA1_RESULTLEN];
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	27 unsigned int hdr_sha1_set:1;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	28 };
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	29
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	30 struct pop3_uidl_map {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	31 struct msg_map_common common;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	32
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	33 uint32_t pop3_seq;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	34 uint32_t imap_uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	35
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	36 /* UIDL */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	37 const char *pop3_uidl;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	38 /* LIST size */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	39 uoff_t size;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	40 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	41
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	42 struct imap_msg_map {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	43 struct msg_map_common common;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	44
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	45 uint32_t uid, pop3_seq;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	46 uoff_t psize;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	47 const char *pop3_uidl;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	48 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	49
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	50 struct pop3_migration_mail_storage {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	51 union mail_storage_module_context module_ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	52
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	53 const char *pop3_box_vname;
14920 a097ef0a9d6d Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Timo Sirainen <tss@iki.fi> parents: 14835 diff changeset	54 ARRAY(struct pop3_uidl_map) pop3_uidl_map;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	55
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	56 unsigned int all_mailboxes:1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	57 unsigned int pop3_all_hdr_sha1_set:1;
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	58 unsigned int ignore_missing_uidls:1;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	59 unsigned int skip_size_check:1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	60 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	61
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	62 struct pop3_migration_mailbox {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	63 union mailbox_module_context module_ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	64
14920 a097ef0a9d6d Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Timo Sirainen <tss@iki.fi> parents: 14835 diff changeset	65 ARRAY(struct imap_msg_map) imap_msg_map;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	66 unsigned int first_unfound_idx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	67
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	68 struct mail_cache_field cache_field;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	69
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	70 unsigned int cache_field_registered:1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	71 unsigned int uidl_synced:1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	72 unsigned int uidl_sync_failed:1;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	73 unsigned int uidl_ordered:1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	74 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	75
19614 87d9089d2c76 pop3-migration: Ignore Return-Path: header when matching UIDLs by header hashes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	76 /* NOTE: these headers must be sorted */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	77 static const char *hdr_hash_skip_headers[] = {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	78 "Content-Length",
19614 87d9089d2c76 pop3-migration: Ignore Return-Path: header when matching UIDLs by header hashes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	79 "Return-Path", /* Yahoo IMAP has Return-Path, Yahoo POP3 doesn't */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	80 "Status",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	81 "X-IMAP",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	82 "X-IMAPbase",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	83 "X-Keywords",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	84 "X-Message-Flag",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	85 "X-Status",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	86 "X-UID",
19618 591fa1448786 pop3-migration: Ignore X-Yahoo-Newman-Property: header Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19617 diff changeset	87 "X-UIDL",
591fa1448786 pop3-migration: Ignore X-Yahoo-Newman-Property: header Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19617 diff changeset	88 "X-Yahoo-Newman-Property"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	89 };
15199 73916b7be94e Plugin ABI version checking improvements. Timo Sirainen <tss@iki.fi> parents: 14921 diff changeset	90 const char *pop3_migration_plugin_version = DOVECOT_ABI_VERSION;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	91
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	92 static MODULE_CONTEXT_DEFINE_INIT(pop3_migration_storage_module,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	93 &mail_storage_module_register);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	94 static MODULE_CONTEXT_DEFINE_INIT(pop3_migration_mail_module,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	95 &mail_module_register);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	96
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	97 static int imap_msg_map_uid_cmp(const struct imap_msg_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	98 const struct imap_msg_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	99 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	100 if (map1->uid < map2->uid)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	101 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	102 if (map1->uid > map2->uid)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	103 return 1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	104 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	105 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	106
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	107 static int pop3_uidl_map_pop3_seq_cmp(const struct pop3_uidl_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	108 const struct pop3_uidl_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	109 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	110 if (map1->pop3_seq < map2->pop3_seq)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	111 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	112 if (map1->pop3_seq > map2->pop3_seq)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	113 return 1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	114 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	115 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	116
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	117 static int pop3_uidl_map_hdr_cmp(const struct pop3_uidl_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	118 const struct pop3_uidl_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	119 {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	120 return memcmp(map1->common.hdr_sha1, map2->common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	121 sizeof(map1->common.hdr_sha1));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	122 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	123
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	124 static int imap_msg_map_hdr_cmp(const struct imap_msg_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	125 const struct imap_msg_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	126 {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	127 return memcmp(map1->common.hdr_sha1, map2->common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	128 sizeof(map1->common.hdr_sha1));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	129 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	130
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	131 struct pop3_hdr_context {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	132 bool have_eoh;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	133 bool stop;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	134 };
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	135
19624 259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	136 static bool header_name_is_valid(const char *name)
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	137 {
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	138 unsigned int i;
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	139
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	140 for (i = 0; name[i] != '\0'; i++) {
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	141 if ((uint8_t)name[i] <= 0x20 \|\| name[i] >= 0x7f)
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	142 return FALSE;
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	143 }
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	144 return TRUE;
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	145 }
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	146
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	147 static void
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	148 pop3_header_filter_callback(struct header_filter_istream *input ATTR_UNUSED,
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	149 struct message_header_line *hdr,
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	150 bool matched, struct pop3_hdr_context ctx)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	151 {
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	152 if (hdr == NULL)
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	153 return;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	154 if (hdr->eoh) {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	155 ctx->have_eoh = TRUE;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	156 if (ctx->stop) {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	157 /* matched is handled differently for eoh by
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	158 istream-header-filter. a design bug I guess.. */
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	159 *matched = FALSE;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	160 }
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	161 } else {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	162 if (strspn(hdr->name, "\r") == hdr->name_len) {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	163 /* CR+CR+LF - some servers stop the header processing
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	164 here while others don't. To make sure they can be
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	165 matched correctly we want to stop here entirely. */
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	166 ctx->stop = TRUE;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	167 }
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	168 if (ctx->stop)
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	169 *matched = TRUE;
19624 259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	170 else if (!header_name_is_valid(hdr->name)) {
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	171 /* Yahoo IMAP drops headers with invalid names, while
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	172 Yahoo POP3 preserves them. Drop them all. */
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	173 *matched = TRUE;
22195 c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	174 } else if (hdr->value_len > 0 &&
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	175 hdr->value[hdr->value_len-1] == ' ') T_BEGIN {
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	176 /* Delete trailing whitespace. Zimbra is stripping it
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	177 away with BODY[HEADER]. */
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	178 struct message_header_line new_hdr = *hdr;
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	179 while (new_hdr.value_len > 0 &&
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	180 new_hdr.value[new_hdr.value_len-1] == ' ')
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	181 new_hdr.value_len--;
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	182 new_hdr.crlf_newline = FALSE; /* CRs are stripped */
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	183 string_t *new_line = t_str_new(128);
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	184 message_header_line_write(new_line, &new_hdr);
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	185 i_stream_header_filter_add(input, str_data(new_line),
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	186 str_len(new_line));
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	187 *matched = TRUE;
c2a33eba3615 pop3-migration: Strip trailing spaces from headers when calculating hash Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21878 diff changeset	188 } T_END;
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	189 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	190 }
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	191
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	192 int pop3_migration_get_hdr_sha1(uint32_t mail_seq, struct istream *input,
20328 7e016f5e8cb4 [LEN] to [static LEN] on some function parameters Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 19693 diff changeset	193 unsigned char sha1_r[STATIC_ARRAY SHA1_RESULTLEN],
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	194 bool *have_eoh_r)
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	195 {
19623 7e08d7ae86be pop3-migration: Convert all non-ASCII in headers to '?' Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19619 diff changeset	196 const unsigned char *data;
19691 69502e2013f3 pop3-migration: Use message_header_hash_more() instead of duplicating the code Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19672 diff changeset	197 size_t size;
20769 efadeff4ab8d lib-mail: message_header_hash_more() now allows input in any slices. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20767 diff changeset	198 struct message_header_hash_context hash_ctx;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	199 struct sha1_ctxt sha1_ctx;
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	200 struct pop3_hdr_context hdr_ctx;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	201
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20769 diff changeset	202 i_zero(&hdr_ctx);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	203 /* hide headers that might change or be different in IMAP vs. POP3 */
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	204 input = i_stream_create_header_filter(input, HEADER_FILTER_HIDE_BODY \|
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	205 HEADER_FILTER_EXCLUDE \| HEADER_FILTER_NO_CR,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	206 hdr_hash_skip_headers,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	207 N_ELEMENTS(hdr_hash_skip_headers),
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	208 pop3_header_filter_callback, &hdr_ctx);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	209
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	210 sha1_init(&sha1_ctx);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	211 while (i_stream_read_data(input, &data, &size, 0) > 0) {
20769 efadeff4ab8d lib-mail: message_header_hash_more() now allows input in any slices. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20767 diff changeset	212 message_header_hash_more(&hash_ctx, &hash_method_sha1, &sha1_ctx, 2,
19691 69502e2013f3 pop3-migration: Use message_header_hash_more() instead of duplicating the code Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19672 diff changeset	213 data, size);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	214 i_stream_skip(input, size);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	215 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	216 if (input->stream_errno != 0) {
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	217 i_error("pop3_migration: Failed to read header for msg %u: %s",
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	218 mail_seq, i_stream_get_error(input));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	219 i_stream_unref(&input);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	220 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	221 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	222 sha1_result(&sha1_ctx, sha1_r);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	223 i_stream_unref(&input);
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	224
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	225 *have_eoh_r = hdr_ctx.have_eoh;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	226 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	227 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	228
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	229 static unsigned int get_cache_idx(struct mail *mail)
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	230 {
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	231 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(mail->box);
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	232
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	233 if (mbox->cache_field_registered)
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	234 return mbox->cache_field.idx;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	235
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	236 mbox->cache_field.name = "pop3-migration.hdr";
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	237 mbox->cache_field.type = MAIL_CACHE_FIELD_FIXED_SIZE;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	238 mbox->cache_field.field_size = SHA1_RESULTLEN;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	239 mail_cache_register_fields(mail->box->cache, &mbox->cache_field, 1);
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	240 mbox->cache_field_registered = TRUE;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	241 return mbox->cache_field.idx;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	242 }
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	243
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	244 static int
20328 7e016f5e8cb4 [LEN] to [static LEN] on some function parameters Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 19693 diff changeset	245 get_hdr_sha1(struct mail *mail, unsigned char sha1_r[STATIC_ARRAY SHA1_RESULTLEN])
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	246 {
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	247 struct istream *input;
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	248 const char *errstr;
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	249 enum mail_error error;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	250 bool have_eoh;
20730 a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	251 int ret;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	252
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	253 if (mail_get_hdr_stream(mail, NULL, &input) < 0) {
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	254 errstr = mailbox_get_last_internal_error(mail->box, &error);
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	255 i_error("pop3_migration: Failed to get header for msg %u: %s",
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	256 mail->seq, errstr);
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	257 return error == MAIL_ERROR_EXPUNGED ? 0 : -1;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	258 }
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	259 if (pop3_migration_get_hdr_sha1(mail->seq, input, sha1_r, &have_eoh) < 0)
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	260 return -1;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	261 if (have_eoh) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	262 struct index_mail imail = (struct index_mail )mail;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	263
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	264 index_mail_cache_add_idx(imail, get_cache_idx(mail),
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	265 sha1_r, SHA1_RESULTLEN);
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	266 return 1;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	267 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	268
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	269 /* The empty "end of headers" line is missing. Either this means that
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	270 the headers ended unexpectedly (which is ok) or that the remote
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	271 server is buggy. Some servers have problems with
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	272
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	273 1) header line continuations that contain only whitespace and
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	274 2) headers that have no ":". The header gets truncated when such
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	275 line is reached.
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	276
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	277 At least Oracle IMS IMAP FETCH BODY[HEADER] handles 1) by not
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	278 returning the whitespace line and 2) by returning the line but
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	279 truncating the rest. POP3 TOP instead returns the entire header.
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	280 This causes the IMAP and POP3 hashes not to match.
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	281
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	282 If there's LF+CR+CR+LF in the middle of headers, Courier IMAP's
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	283 FETCH BODY[HEADER] stops after that, but Courier POP3's TOP doesn't.
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	284
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	285 So we'll try to avoid this by falling back to full FETCH BODY[]
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	286 (and/or RETR) and we'll parse the header ourself from it. This
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	287 should work around any similar bugs in all IMAP/POP3 servers. */
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	288 if (mail_get_stream_because(mail, NULL, NULL, "pop3-migration", &input) < 0) {
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	289 errstr = mailbox_get_last_internal_error(mail->box, &error);
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	290 i_error("pop3_migration: Failed to get body for msg %u: %s",
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	291 mail->seq, errstr);
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	292 return error == MAIL_ERROR_EXPUNGED ? 0 : -1;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	293 }
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	294 ret = pop3_migration_get_hdr_sha1(mail->seq, input, sha1_r, &have_eoh);
20730 a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	295 if (ret == 0) {
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	296 if (!have_eoh)
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	297 i_warning("pop3_migration: Truncated email with UID %u stored as truncated", mail->uid);
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	298 struct index_mail imail = (struct index_mail )mail;
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	299 index_mail_cache_add_idx(imail, get_cache_idx(mail),
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	300 sha1_r, SHA1_RESULTLEN);
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	301 return 1;
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	302 } else {
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	303 return -1;
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	304 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	305 }
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	306
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	307 static bool
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	308 get_cached_hdr_sha1(struct mail mail, buffer_t cache_buf,
20328 7e016f5e8cb4 [LEN] to [static LEN] on some function parameters Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 19693 diff changeset	309 unsigned char sha1_r[STATIC_ARRAY SHA1_RESULTLEN])
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	310 {
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	311 struct index_mail imail = (struct index_mail )mail;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	312
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	313 buffer_set_used_size(cache_buf, 0);
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	314 if (index_mail_cache_lookup_field(imail, cache_buf,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	315 get_cache_idx(mail)) > 0 &&
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	316 cache_buf->used == SHA1_RESULTLEN) {
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	317 memcpy(sha1_r, cache_buf->data, cache_buf->used);
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	318 return TRUE;
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	319 }
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	320 return FALSE;
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	321 }
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	322
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	323 static struct mailbox pop3_mailbox_alloc(struct mail_storage storage)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	324 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	325 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	326 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	327 struct mail_namespace *ns;
21878 e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	328 struct mailbox *box;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	329
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	330 ns = mail_namespace_find(storage->user->namespaces,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	331 mstorage->pop3_box_vname);
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	332 i_assert(ns != NULL);
21878 e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	333 box = mailbox_alloc(ns->list, mstorage->pop3_box_vname,
e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	334 MAILBOX_FLAG_READONLY \| MAILBOX_FLAG_POP3_SESSION);
e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	335 mailbox_set_reason(box, "pop3_migration");
e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	336 return box;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	337 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	338
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	339 static int pop3_map_read(struct mail_storage storage, struct mailbox pop3_box)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	340 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	341 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	342 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	343 struct mailbox_transaction_context *t;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	344 struct mail_search_args *search_args;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	345 struct mail_search_context *ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	346 struct mail *mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	347 struct pop3_uidl_map *map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	348 const char *uidl;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	349 uoff_t size = (uoff_t)-1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	350 int ret = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	351
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	352 if (array_is_created(&mstorage->pop3_uidl_map)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	353 /* already read these, just reset the imap_uids */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	354 array_foreach_modifiable(&mstorage->pop3_uidl_map, map)
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	355 map->imap_uid = 0;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	356 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	357 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	358 i_array_init(&mstorage->pop3_uidl_map, 128);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	359
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	360 if (mailbox_sync(pop3_box, 0) < 0) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	361 i_error("pop3_migration: Couldn't sync mailbox %s: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	362 pop3_box->vname, mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	363 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	364 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	365
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	366 t = mailbox_transaction_begin(pop3_box, 0);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	367 search_args = mail_search_build_init();
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	368 mail_search_build_add_all(search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	369 ctx = mailbox_search_init(t, search_args, NULL,
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	370 mstorage->skip_size_check ? 0 :
18912 64568a033fc8 pop3-migration: Fetch physical sizes instead of virtual sizes so pop3c uses LIST Timo Sirainen <tss@iki.fi> parents: 18911 diff changeset	371 MAIL_FETCH_PHYSICAL_SIZE, NULL);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	372 mail_search_args_unref(&search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	373
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	374 while (mailbox_search_next(ctx, &mail)) {
18911 4bebfbb32410 pop3-migration: Use LIST instead of RETRs to get the messages' sizes. Timo Sirainen <tss@iki.fi> parents: 18909 diff changeset	375 /* get the size with LIST instead of RETR */
4bebfbb32410 pop3-migration: Use LIST instead of RETRs to get the messages' sizes. Timo Sirainen <tss@iki.fi> parents: 18909 diff changeset	376 mail->lookup_abort = MAIL_LOOKUP_ABORT_READ_MAIL;
4bebfbb32410 pop3-migration: Use LIST instead of RETRs to get the messages' sizes. Timo Sirainen <tss@iki.fi> parents: 18909 diff changeset	377
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	378 if (mstorage->skip_size_check)
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	379 ;
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	380 else if (mail_get_physical_size(mail, &size) < 0) {
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	381 i_error("pop3_migration: Failed to get size for msg %u: %s",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	382 mail->seq,
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	383 mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	384 ret = -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	385 break;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	386 }
21649 78f38a28086e lib-storage, pop3-migration: Reset lookup_abort before continuing to search Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21390 diff changeset	387 mail->lookup_abort = MAIL_LOOKUP_ABORT_NEVER;
78f38a28086e lib-storage, pop3-migration: Reset lookup_abort before continuing to search Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21390 diff changeset	388
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	389 if (mail_get_special(mail, MAIL_FETCH_UIDL_BACKEND, &uidl) < 0) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	390 i_error("pop3_migration: Failed to get UIDL for msg %u: %s",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	391 mail->seq,
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	392 mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	393 ret = -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	394 break;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	395 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	396 if (*uidl == '\0') {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	397 i_warning("pop3_migration: UIDL for msg %u is empty",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	398 mail->seq);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	399 continue;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	400 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	401
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	402 map = array_append_space(&mstorage->pop3_uidl_map);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	403 map->pop3_seq = mail->seq;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	404 map->pop3_uidl = p_strdup(storage->pool, uidl);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	405 map->size = size;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	406 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	407
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	408 if (mailbox_search_deinit(&ctx) < 0) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	409 i_error("pop3_migration: Failed to search all POP3 mails: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	410 mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	411 ret = -1;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	412 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	413 (void)mailbox_transaction_commit(&t);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	414 return ret;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	415 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	416
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	417 static void
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	418 pop3_map_read_cached_hdr_hashes(struct mailbox_transaction_context *t,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	419 struct mail_search_args *search_args,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	420 struct array *msg_map)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	421 {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	422 struct mail_search_context *ctx;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	423 struct mail *mail;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	424 struct msg_map_common *map;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	425 buffer_t *cache_buf;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	426
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	427 ctx = mailbox_search_init(t, search_args, NULL, 0, NULL);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	428 cache_buf = buffer_create_dynamic(pool_datastack_create(), SHA1_RESULTLEN);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	429
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	430 while (mailbox_search_next(ctx, &mail)) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	431 map = array_idx_modifiable_i(msg_map, mail->seq-1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	432
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	433 if (get_cached_hdr_sha1(mail, cache_buf, map->hdr_sha1))
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	434 map->hdr_sha1_set = TRUE;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	435 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	436
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	437 if (mailbox_search_deinit(&ctx) < 0) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	438 i_warning("pop3_migration: Failed to search all cached POP3 header hashes: %s - ignoring",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	439 mailbox_get_last_internal_error(t->box, NULL));
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	440 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	441 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	442
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	443 static void map_remove_found_seqs(struct mail_search_arg *search_arg,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	444 struct array *msg_map, uint32_t seq1)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	445 {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	446 const struct msg_map_common *map;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	447 uint32_t seq, count = array_count_i(msg_map);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	448
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	449 i_assert(search_arg->type == SEARCH_SEQSET);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	450
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	451 for (seq = seq1; seq <= count; seq++) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	452 map = array_idx_i(msg_map, seq-1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	453 if (map->hdr_sha1_set)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	454 seq_range_array_remove(&search_arg->value.seqset, seq);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	455 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	456 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	457
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	458 static int
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	459 map_read_hdr_hashes(struct mailbox box, struct array msg_map, uint32_t seq1)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	460 {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	461 struct mailbox_transaction_context *t;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	462 struct mail_search_args *search_args;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	463 struct mail_search_context *ctx;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	464 struct mail *mail;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	465 struct msg_map_common *map;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	466 int ret = 0;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	467
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	468 t = mailbox_transaction_begin(box, 0);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	469 /* get all the cached hashes */
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	470 search_args = mail_search_build_init();
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	471 mail_search_build_add_seqset(search_args, seq1, array_count_i(msg_map));
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	472 pop3_map_read_cached_hdr_hashes(t, search_args, msg_map);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	473 /* read all the non-cached hashes. doing this in two passes allows
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	474 us to set wanted_fields=MAIL_FETCH_STREAM_HEADER, which allows
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	475 prefetching to work without downloading all the headers even
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	476 for mails that already are cached. */
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	477 map_remove_found_seqs(search_args->args, msg_map, seq1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	478 ctx = mailbox_search_init(t, search_args, NULL,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	479 MAIL_FETCH_STREAM_HEADER, NULL);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	480 mail_search_args_unref(&search_args);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	481
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	482 while (mailbox_search_next(ctx, &mail)) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	483 map = array_idx_modifiable_i(msg_map, mail->seq-1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	484
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	485 if ((ret = get_hdr_sha1(mail, map->hdr_sha1)) < 0) {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	486 ret = -1;
19647 8f2ba9ebc463 pop3-migration: If reading message header for hashing fails, fail immediately. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19646 diff changeset	487 break;
8f2ba9ebc463 pop3-migration: If reading message header for hashing fails, fail immediately. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19646 diff changeset	488 }
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	489 if (ret > 0)
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	490 map->hdr_sha1_set = TRUE;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	491 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	492
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	493 if (mailbox_search_deinit(&ctx) < 0) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	494 i_error("pop3_migration: Failed to search all mail headers: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	495 mailbox_get_last_internal_error(box, NULL));
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	496 ret = -1;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	497 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	498 (void)mailbox_transaction_commit(&t);
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	499 return ret < 0 ? -1 : 0;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	500 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	501
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	502 static int
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	503 pop3_map_read_hdr_hashes(struct mail_storage storage, struct mailbox pop3_box,
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	504 unsigned first_seq)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	505 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	506 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	507 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	508
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	509 if (mstorage->pop3_all_hdr_sha1_set)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	510 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	511 if (mstorage->all_mailboxes) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	512 /* we may be matching against multiple mailboxes.
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	513 read all the hashes only once. */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	514 first_seq = 1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	515 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	516
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	517 if (map_read_hdr_hashes(pop3_box, &mstorage->pop3_uidl_map.arr,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	518 first_seq) < 0)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	519 return -1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	520
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	521 if (first_seq == 1)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	522 mstorage->pop3_all_hdr_sha1_set = TRUE;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	523 return 0;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	524 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	525
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	526 static int imap_map_read(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	527 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	528 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	529 struct pop3_migration_mail_storage *mstorage =
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	530 POP3_MIGRATION_CONTEXT(box->storage);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	531 struct mailbox_status status;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	532 struct mailbox_transaction_context *t;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	533 struct mail_search_args *search_args;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	534 struct mail_search_context *ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	535 struct mail *mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	536 struct imap_msg_map *map;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	537 uoff_t psize = (uoff_t)-1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	538 int ret = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	539
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	540 mailbox_get_open_status(box, STATUS_MESSAGES, &status);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	541
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	542 i_assert(!array_is_created(&mbox->imap_msg_map));
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	543 p_array_init(&mbox->imap_msg_map, box->pool, status.messages);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	544
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	545 t = mailbox_transaction_begin(box, 0);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	546 search_args = mail_search_build_init();
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	547 mail_search_build_add_all(search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	548 ctx = mailbox_search_init(t, search_args, NULL,
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	549 mstorage->skip_size_check ? 0 :
14402 87cd04f35f82 pop3-replication: Don't request virtual size from IMAP mailbox, we don't need it. Timo Sirainen <tss@iki.fi> parents: 14401 diff changeset	550 MAIL_FETCH_PHYSICAL_SIZE, NULL);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	551 mail_search_args_unref(&search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	552
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	553 while (mailbox_search_next(ctx, &mail)) {
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	554 if (mstorage->skip_size_check)
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	555 ;
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	556 else if (mail_get_physical_size(mail, &psize) < 0) {
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	557 i_error("pop3_migration: Failed to get psize for imap uid %u: %s",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	558 mail->uid,
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	559 mailbox_get_last_internal_error(box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	560 ret = -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	561 break;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	562 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	563
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	564 map = array_append_space(&mbox->imap_msg_map);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	565 map->uid = mail->uid;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	566 map->psize = psize;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	567 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	568
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	569 if (mailbox_search_deinit(&ctx) < 0) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	570 i_error("pop3_migration: Failed to search all IMAP mails: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	571 mailbox_get_last_internal_error(box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	572 ret = -1;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	573 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	574 (void)mailbox_transaction_commit(&t);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	575 return ret;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	576 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	577
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	578 static int imap_map_read_hdr_hashes(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	579 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	580 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	581
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	582 return map_read_hdr_hashes(box, &mbox->imap_msg_map.arr,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	583 mbox->first_unfound_idx+1);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	584 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	585
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	586 static bool pop3_uidl_assign_by_size(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	587 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	588 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	589 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	590 POP3_MIGRATION_CONTEXT(box->storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	591 struct pop3_uidl_map *pop3_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	592 struct imap_msg_map *imap_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	593 unsigned int i, pop3_count, imap_count, count;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	594
19644 defab5b9af2a pop3-migration: Don't even try to match message sizes with pop3_migration_skip_size_check=yes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19640 diff changeset	595 if (mstorage->skip_size_check)
defab5b9af2a pop3-migration: Don't even try to match message sizes with pop3_migration_skip_size_check=yes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19640 diff changeset	596 return FALSE;
defab5b9af2a pop3-migration: Don't even try to match message sizes with pop3_migration_skip_size_check=yes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19640 diff changeset	597
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	598 pop3_map = array_get_modifiable(&mstorage->pop3_uidl_map, &pop3_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	599 imap_map = array_get_modifiable(&mbox->imap_msg_map, &imap_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	600 count = I_MIN(pop3_count, imap_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	601
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	602 /* see if we can match the messages using sizes */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	603 for (i = 0; i < count; i++) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	604 if (pop3_map[i].size != imap_map[i].psize)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	605 break;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	606 if (i+1 < count && pop3_map[i].size == pop3_map[i+1].size) {
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	607 /* two messages with same size, don't trust them */
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	608 break;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	609 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	610
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	611 pop3_map[i].imap_uid = imap_map[i].uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	612 imap_map[i].pop3_uidl = pop3_map[i].pop3_uidl;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	613 imap_map[i].pop3_seq = pop3_map[i].pop3_seq;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	614 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	615 mbox->first_unfound_idx = i;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	616 if (box->storage->user->mail_debug)
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	617 i_debug("pop3_migration: %u/%u mails matched by size", i, count);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	618 return i == count;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	619 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	620
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	621 static int
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	622 pop3_uidl_assign_by_hdr_hash(struct mailbox box, struct mailbox pop3_box)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	623 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	624 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	625 POP3_MIGRATION_CONTEXT(box->storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	626 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	627 struct pop3_uidl_map *pop3_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	628 struct imap_msg_map *imap_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	629 unsigned int pop3_idx, imap_idx, pop3_count, imap_count;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	630 unsigned int first_seq, missing_uids_count;
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	631 uint32_t first_missing_idx = 0, first_missing_seq = (uint32_t)-1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	632 int ret;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	633
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	634 first_seq = mbox->first_unfound_idx+1;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	635 if (pop3_map_read_hdr_hashes(box->storage, pop3_box, first_seq) < 0 \|\|
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	636 imap_map_read_hdr_hashes(box) < 0)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	637 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	638
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	639 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_hdr_cmp);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	640 array_sort(&mbox->imap_msg_map, imap_msg_map_hdr_cmp);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	641
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	642 pop3_map = array_get_modifiable(&mstorage->pop3_uidl_map, &pop3_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	643 imap_map = array_get_modifiable(&mbox->imap_msg_map, &imap_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	644
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	645 pop3_idx = imap_idx = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	646 while (pop3_idx < pop3_count && imap_idx < imap_count) {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	647 if (!pop3_map[pop3_idx].common.hdr_sha1_set \|\|
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	648 pop3_map[pop3_idx].imap_uid != 0) {
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	649 pop3_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	650 continue;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	651 }
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	652 if (!imap_map[imap_idx].common.hdr_sha1_set \|\|
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	653 imap_map[imap_idx].pop3_uidl != NULL) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	654 imap_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	655 continue;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	656 }
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	657 ret = memcmp(pop3_map[pop3_idx].common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	658 imap_map[imap_idx].common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	659 sizeof(pop3_map[pop3_idx].common.hdr_sha1));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	660 if (ret < 0)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	661 pop3_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	662 else if (ret > 0)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	663 imap_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	664 else {
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	665 pop3_map[pop3_idx].imap_uid = imap_map[imap_idx].uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	666 imap_map[imap_idx].pop3_uidl =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	667 pop3_map[pop3_idx].pop3_uidl;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	668 imap_map[imap_idx].pop3_seq =
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	669 pop3_map[pop3_idx].pop3_seq;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	670 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	671 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	672 missing_uids_count = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	673 for (pop3_idx = 0; pop3_idx < pop3_count; pop3_idx++) {
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	674 if (pop3_map[pop3_idx].imap_uid != 0) {
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	675 /* matched */
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	676 } else if (!pop3_map[pop3_idx].common.hdr_sha1_set) {
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	677 /* we treated this mail as expunged - ignore */
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	678 } else {
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	679 uint32_t seq = pop3_map[pop3_idx].pop3_seq;
6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	680 if (first_missing_seq > seq) {
6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	681 first_missing_seq = seq;
19617 7b0a093df334 pop3-migration: When logging the first missing POP3 mail, it logged a wrong mail Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19616 diff changeset	682 first_missing_idx = pop3_idx;
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	683 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	684 missing_uids_count++;
18908 b51dfee18fd2 pop3-migration: Show the first message's number and UIDL which wasn't found from IMAP. Timo Sirainen <tss@iki.fi> parents: 18624 diff changeset	685 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	686 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	687 if (missing_uids_count > 0 && !mstorage->all_mailboxes) {
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	688 string_t *str = t_str_new(128);
af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	689
af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	690 str_printfa(str, "pop3_migration: %u POP3 messages have no "
af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	691 "matching IMAP messages (first POP3 msg %u UIDL %s)",
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	692 missing_uids_count, first_missing_seq,
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	693 pop3_map[first_missing_idx].pop3_uidl);
19619 c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	694 if (imap_count + missing_uids_count == pop3_count) {
c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	695 str_append(str, " - all IMAP messages were found "
c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	696 "(POP3 contains more than IMAP INBOX - you may want to set pop3_migration_all_mailboxes=yes)");
c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	697 }
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	698 if (!mstorage->ignore_missing_uidls) {
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	699 i_error("%s - set pop3_migration_ignore_missing_uidls=yes to continue anyway",
af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	700 str_c(str));
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	701 return -1;
1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	702 }
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	703 i_warning("%s", str_c(str));
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	704 } else if (box->storage->user->mail_debug) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	705 i_debug("pop3_migration: %u mails matched by headers", pop3_count);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	706 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	707 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_pop3_seq_cmp);
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	708 array_sort(&mbox->imap_msg_map, imap_msg_map_uid_cmp);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	709 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	710 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	711
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	712 static int pop3_migration_uidl_sync(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	713 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	714 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	715 struct pop3_migration_mail_storage *mstorage =
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	716 POP3_MIGRATION_CONTEXT(box->storage);
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	717 struct mailbox *pop3_box;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	718 const struct pop3_uidl_map *pop3_map;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	719 unsigned int i, count;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	720 uint32_t prev_uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	721
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	722 pop3_box = pop3_mailbox_alloc(box->storage);
14835 ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	723 /* the POP3 server isn't connected to yet. handle all IMAP traffic
ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	724 first before connecting, so POP3 server won't disconnect us due to
ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	725 idling. */
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	726 if (imap_map_read(box) < 0 \|\|
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	727 pop3_map_read(box->storage, pop3_box) < 0) {
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	728 mailbox_free(&pop3_box);
14835 ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	729 return -1;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	730 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	731
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	732 if (!pop3_uidl_assign_by_size(box)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	733 /* everything wasn't assigned, figure out the rest with
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	734 header hashes */
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	735 if (pop3_uidl_assign_by_hdr_hash(box, pop3_box) < 0) {
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	736 mailbox_free(&pop3_box);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	737 return -1;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	738 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	739 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	740
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	741 /* see if the POP3 UIDL order is the same as IMAP UID order */
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	742 mbox->uidl_ordered = TRUE;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	743 pop3_map = array_get(&mstorage->pop3_uidl_map, &count);
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	744 prev_uid = 0;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	745 for (i = 0; i < count; i++) {
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	746 if (pop3_map[i].imap_uid == 0)
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	747 continue;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	748
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	749 if (prev_uid > pop3_map[i].imap_uid) {
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	750 mbox->uidl_ordered = FALSE;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	751 break;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	752 }
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	753 prev_uid = pop3_map[i].imap_uid;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	754 }
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	755
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	756 mbox->uidl_synced = TRUE;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	757 mailbox_free(&pop3_box);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	758 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	759 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	760
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	761 static int pop3_migration_uidl_sync_if_needed(struct mailbox *box)
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	762 {
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	763 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	764
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	765 if (mbox->uidl_synced)
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	766 return 0;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	767
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	768 if (mbox->uidl_sync_failed \|\|
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	769 pop3_migration_uidl_sync(box) < 0) {
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	770 mbox->uidl_sync_failed = TRUE;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	771 mail_storage_set_error(box->storage, MAIL_ERROR_TEMP,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	772 "POP3 UIDLs couldn't be synced");
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	773 return -1;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	774 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	775 return 0;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	776 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	777
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	778 static int
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	779 pop3_migration_get_special(struct mail *_mail, enum mail_fetch_field field,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	780 const char **value_r)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	781 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	782 struct mail_private mail = (struct mail_private )_mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	783 union mail_module_context *mmail = POP3_MIGRATION_MAIL_CONTEXT(mail);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	784 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(_mail->box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	785 struct imap_msg_map map_key, *map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	786
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	787 if (field == MAIL_FETCH_UIDL_BACKEND \|\|
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	788 field == MAIL_FETCH_POP3_ORDER) {
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	789 if (pop3_migration_uidl_sync_if_needed(_mail->box) < 0)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	790 return -1;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	791
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20769 diff changeset	792 i_zero(&map_key);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	793 map_key.uid = _mail->uid;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	794 map = array_bsearch(&mbox->imap_msg_map, &map_key,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	795 imap_msg_map_uid_cmp);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	796 if (map != NULL && map->pop3_uidl != NULL) {
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	797 if (field == MAIL_FETCH_UIDL_BACKEND)
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	798 *value_r = map->pop3_uidl;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	799 else
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	800 *value_r = t_strdup_printf("%u", map->pop3_seq);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	801 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	802 }
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	803 /* not found from POP3 server, fallback to default */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	804 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	805 return mmail->super.get_special(_mail, field, value_r);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	806 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	807
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	808 static void pop3_migration_mail_allocated(struct mail *_mail)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	809 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	810 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	811 POP3_MIGRATION_CONTEXT(_mail->box->storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	812 struct mail_private mail = (struct mail_private )_mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	813 struct mail_vfuncs *v = mail->vlast;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	814 union mail_module_context *mmail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	815 struct mail_namespace *ns;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	816
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	817 if (mstorage == NULL \|\|
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	818 (!mstorage->all_mailboxes && !_mail->box->inbox_user)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	819 /* assigns UIDLs only for INBOX */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	820 return;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	821 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	822
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	823 ns = mail_namespace_find(_mail->box->storage->user->namespaces,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	824 mstorage->pop3_box_vname);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	825 if (ns == mailbox_get_namespace(_mail->box)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	826 /* we're accessing the pop3-migration namespace itself */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	827 return;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	828 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	829
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	830 mmail = p_new(mail->pool, union mail_module_context, 1);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	831 mmail->super = *v;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	832 mail->vlast = &mmail->super;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	833
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	834 v->get_special = pop3_migration_get_special;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	835 MODULE_CONTEXT_SET_SELF(mail, pop3_migration_mail_module, mmail);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	836 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	837
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	838 static struct mail_search_context *
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	839 pop3_migration_mailbox_search_init(struct mailbox_transaction_context *t,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	840 struct mail_search_args *args,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	841 const enum mail_sort_type *sort_program,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	842 enum mail_fetch_field wanted_fields,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	843 struct mailbox_header_lookup_ctx *wanted_headers)
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	844 {
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	845 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(t->box);
19672 e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	846 struct pop3_migration_mail_storage *mstorage =
e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	847 POP3_MIGRATION_CONTEXT(t->box->storage);
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	848
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	849 if ((wanted_fields & (MAIL_FETCH_UIDL_BACKEND \|
19672 e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	850 MAIL_FETCH_POP3_ORDER)) != 0 &&
e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	851 (mstorage->all_mailboxes \|\| t->box->inbox_user)) {
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	852 /* Start POP3 UIDL syncing before the search, so we'll do it
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	853 before we start sending any FETCH BODY[]s to IMAP. It
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	854 shouldn't matter much, except this works around a bug in
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	855 Yahoo IMAP where it sometimes breaks its state when doing
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	856 a FETCH BODY[] followed by FETCH BODY[HEADER].. */
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	857 (void)pop3_migration_uidl_sync_if_needed(t->box);
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	858 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	859
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	860 return mbox->module_ctx.super.search_init(t, args, sort_program,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	861 wanted_fields, wanted_headers);
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	862 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	863
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	864 static void pop3_migration_mailbox_allocated(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	865 {
20746 6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	866 struct pop3_migration_mail_storage *mstorage =
6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	867 POP3_MIGRATION_CONTEXT(box->storage);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	868 struct mailbox_vfuncs *v = box->vlast;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	869 struct pop3_migration_mailbox *mbox;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	870
20746 6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	871 if (mstorage == NULL)
6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	872 return;
6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	873
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	874 mbox = p_new(box->pool, struct pop3_migration_mailbox, 1);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	875 mbox->module_ctx.super = *v;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	876 box->vlast = &mbox->module_ctx.super;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	877
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	878 v->search_init = pop3_migration_mailbox_search_init;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	879
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	880 MODULE_CONTEXT_SET(box, pop3_migration_storage_module, mbox);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	881 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	882
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	883 static void pop3_migration_mail_storage_destroy(struct mail_storage *storage)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	884 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	885 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	886 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	887
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	888 if (array_is_created(&mstorage->pop3_uidl_map))
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	889 array_free(&mstorage->pop3_uidl_map);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	890
15272 ca10d2e8d8e3 lib-storage: Moved mail attribute dict to struct mail_storage. Timo Sirainen <tss@iki.fi> parents: 15199 diff changeset	891 mstorage->module_ctx.super.destroy(storage);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	892 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	893
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	894 static void pop3_migration_mail_storage_created(struct mail_storage *storage)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	895 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	896 struct pop3_migration_mail_storage *mstorage;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	897 struct mail_storage_vfuncs *v = storage->vlast;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	898 const char *pop3_box_vname;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	899
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	900 pop3_box_vname = mail_user_plugin_getenv(storage->user,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	901 "pop3_migration_mailbox");
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	902 if (pop3_box_vname == NULL) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	903 if (storage->user->mail_debug)
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	904 i_debug("pop3_migration: No pop3_migration_mailbox setting - disabled");
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	905 return;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	906 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	907
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	908 mstorage = p_new(storage->pool, struct pop3_migration_mail_storage, 1);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	909 mstorage->module_ctx.super = *v;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	910 storage->vlast = &mstorage->module_ctx.super;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	911 v->destroy = pop3_migration_mail_storage_destroy;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	912
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	913 mstorage->pop3_box_vname = p_strdup(storage->pool, pop3_box_vname);
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	914 mstorage->all_mailboxes =
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	915 mail_user_plugin_getenv(storage->user,
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	916 "pop3_migration_all_mailboxes") != NULL;
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	917 mstorage->ignore_missing_uidls =
1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	918 mail_user_plugin_getenv(storage->user,
1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	919 "pop3_migration_ignore_missing_uidls") != NULL;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	920 mstorage->skip_size_check =
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	921 mail_user_plugin_getenv(storage->user,
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	922 "pop3_migration_skip_size_check") != NULL;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	923
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	924 MODULE_CONTEXT_SET(storage, pop3_migration_storage_module, mstorage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	925 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	926
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	927 static struct mail_storage_hooks pop3_migration_mail_storage_hooks = {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	928 .mail_allocated = pop3_migration_mail_allocated,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	929 .mailbox_allocated = pop3_migration_mailbox_allocated,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	930 .mail_storage_created = pop3_migration_mail_storage_created
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	931 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	932
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	933 void pop3_migration_plugin_init(struct module *module)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	934 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	935 mail_storage_hooks_add(module, &pop3_migration_mail_storage_hooks);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	936 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	937
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	938 void pop3_migration_plugin_deinit(void)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	939 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	940 mail_storage_hooks_remove(&pop3_migration_mail_storage_hooks);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	941 }

Mercurial > dovecot > core-2.2

annotate src/plugins/pop3-migration/pop3-migration-plugin.c @ 22195:c2a33eba3615