dovecot/core-2.2: src/plugins/pop3-migration/pop3-migration-plugin.c annotate

annotate src/plugins/pop3-migration/pop3-migration-plugin.c @ 22453:6eb5a6930614

pop3-migration: Fail if all IMAP mails were matched by size, but POP3 had extra mails This makes the handling same as what happens when the same situation happens and the last mails were matched by headers. That's an error, unless pop3_migration_ignore_*_uidls=yes.

author	Timo Sirainen <timo.sirainen@dovecot.fi>
date	Mon, 04 Sep 2017 19:37:12 +0300
parents	566bd6b3b348
children	cb108f786fb4

rev	line source
21390 2e2563132d5f Updated copyright notices to include the year 2017. Stephan Bosch <stephan.bosch@dovecot.fi> parents: 21389 diff changeset	1 /* Copyright (c) 2007-2017 Dovecot authors, see the included COPYING file */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	2
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	3 #include "lib.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	4 #include "array.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	5 #include "istream.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	6 #include "istream-header-filter.h"
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	7 #include "str.h"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	8 #include "sha1.h"
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	9 #include "message-size.h"
19691 69502e2013f3 pop3-migration: Use message_header_hash_more() instead of duplicating the code Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19672 diff changeset	10 #include "message-header-hash.h"
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	11 #include "message-header-parser.h"
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	12 #include "mail-cache.h"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	13 #include "mail-namespace.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	14 #include "mail-search-build.h"
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	15 #include "index-storage.h"
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	16 #include "index-mail.h"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	17 #include "pop3-migration-plugin.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	18
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	19 #define POP3_MIGRATION_CONTEXT(obj) \
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	20 MODULE_CONTEXT(obj, pop3_migration_storage_module)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	21 #define POP3_MIGRATION_MAIL_CONTEXT(obj) \
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	22 MODULE_CONTEXT(obj, pop3_migration_mail_module)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	23
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	24 struct msg_map_common {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	25 /* sha1(header) - set only when needed */
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	26 unsigned char hdr_sha1[SHA1_RESULTLEN];
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	27 unsigned int hdr_sha1_set:1;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	28 };
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	29
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	30 struct pop3_uidl_map {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	31 struct msg_map_common common;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	32
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	33 uint32_t pop3_seq;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	34 uint32_t imap_uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	35
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	36 /* UIDL */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	37 const char *pop3_uidl;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	38 /* LIST size */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	39 uoff_t size;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	40 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	41
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	42 struct imap_msg_map {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	43 struct msg_map_common common;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	44
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	45 uint32_t uid, pop3_seq;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	46 uoff_t psize;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	47 const char *pop3_uidl;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	48 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	49
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	50 struct pop3_migration_mail_storage {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	51 union mail_storage_module_context module_ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	52
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	53 const char *pop3_box_vname;
14920 a097ef0a9d6d Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Timo Sirainen <tss@iki.fi> parents: 14835 diff changeset	54 ARRAY(struct pop3_uidl_map) pop3_uidl_map;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	55
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	56 unsigned int all_mailboxes:1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	57 unsigned int pop3_all_hdr_sha1_set:1;
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	58 unsigned int ignore_missing_uidls:1;
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	59 unsigned int ignore_extra_uidls:1;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	60 unsigned int skip_size_check:1;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	61 unsigned int skip_uidl_cache:1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	62 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	63
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	64 struct pop3_migration_mailbox {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	65 union mailbox_module_context module_ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	66
14920 a097ef0a9d6d Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Timo Sirainen <tss@iki.fi> parents: 14835 diff changeset	67 ARRAY(struct imap_msg_map) imap_msg_map;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	68 unsigned int first_unfound_idx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	69
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	70 struct mail_cache_field cache_field;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	71
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	72 unsigned int cache_field_registered:1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	73 unsigned int uidl_synced:1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	74 unsigned int uidl_sync_failed:1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	75 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	76
19614 87d9089d2c76 pop3-migration: Ignore Return-Path: header when matching UIDLs by header hashes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	77 /* NOTE: these headers must be sorted */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	78 static const char *hdr_hash_skip_headers[] = {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	79 "Content-Length",
19614 87d9089d2c76 pop3-migration: Ignore Return-Path: header when matching UIDLs by header hashes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	80 "Return-Path", /* Yahoo IMAP has Return-Path, Yahoo POP3 doesn't */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	81 "Status",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	82 "X-IMAP",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	83 "X-IMAPbase",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	84 "X-Keywords",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	85 "X-Message-Flag",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	86 "X-Status",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	87 "X-UID",
19618 591fa1448786 pop3-migration: Ignore X-Yahoo-Newman-Property: header Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19617 diff changeset	88 "X-UIDL",
591fa1448786 pop3-migration: Ignore X-Yahoo-Newman-Property: header Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19617 diff changeset	89 "X-Yahoo-Newman-Property"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	90 };
15199 73916b7be94e Plugin ABI version checking improvements. Timo Sirainen <tss@iki.fi> parents: 14921 diff changeset	91 const char *pop3_migration_plugin_version = DOVECOT_ABI_VERSION;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	92
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	93 static MODULE_CONTEXT_DEFINE_INIT(pop3_migration_storage_module,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	94 &mail_storage_module_register);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	95 static MODULE_CONTEXT_DEFINE_INIT(pop3_migration_mail_module,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	96 &mail_module_register);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	97
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	98 static int imap_msg_map_uid_cmp(const struct imap_msg_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	99 const struct imap_msg_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	100 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	101 if (map1->uid < map2->uid)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	102 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	103 if (map1->uid > map2->uid)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	104 return 1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	105 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	106 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	107
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	108 static int pop3_uidl_map_pop3_seq_cmp(const struct pop3_uidl_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	109 const struct pop3_uidl_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	110 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	111 if (map1->pop3_seq < map2->pop3_seq)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	112 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	113 if (map1->pop3_seq > map2->pop3_seq)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	114 return 1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	115 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	116 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	117
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	118 static int pop3_uidl_map_uidl_cmp(const struct pop3_uidl_map *map1,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	119 const struct pop3_uidl_map *map2)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	120 {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	121 return strcmp(map1->pop3_uidl, map2->pop3_uidl);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	122 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	123
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	124 static int imap_msg_map_uidl_cmp(const struct imap_msg_map *map1,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	125 const struct imap_msg_map *map2)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	126 {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	127 return null_strcmp(map1->pop3_uidl, map2->pop3_uidl);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	128 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	129
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	130 static int pop3_uidl_map_hdr_cmp(const struct pop3_uidl_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	131 const struct pop3_uidl_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	132 {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	133 return memcmp(map1->common.hdr_sha1, map2->common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	134 sizeof(map1->common.hdr_sha1));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	135 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	136
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	137 static int imap_msg_map_hdr_cmp(const struct imap_msg_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	138 const struct imap_msg_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	139 {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	140 return memcmp(map1->common.hdr_sha1, map2->common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	141 sizeof(map1->common.hdr_sha1));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	142 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	143
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	144 struct pop3_hdr_context {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	145 bool have_eoh;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	146 bool stop;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	147 };
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	148
19624 259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	149 static bool header_name_is_valid(const char *name)
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	150 {
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	151 unsigned int i;
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	152
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	153 for (i = 0; name[i] != '\0'; i++) {
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	154 if ((uint8_t)name[i] <= 0x20 \|\| name[i] >= 0x7f)
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	155 return FALSE;
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	156 }
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	157 return TRUE;
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	158 }
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	159
22265 66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	160 static bool header_value_want_skip(const struct message_header_line *hdr)
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	161 {
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	162 for (size_t i = 0; i < hdr->value_len; i++) {
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	163 if (hdr->value[i] != ' ' && hdr->value[i] != '\t')
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	164 return FALSE;
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	165 }
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	166 /* "header: \r\n \r\n" - Zimbra's BODY[HEADER] strips this line away. */
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	167 return TRUE;
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	168 }
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	169
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	170 static void
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	171 pop3_header_filter_callback(struct header_filter_istream *input ATTR_UNUSED,
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	172 struct message_header_line *hdr,
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	173 bool matched, struct pop3_hdr_context ctx)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	174 {
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	175 if (hdr == NULL)
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	176 return;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	177 if (hdr->eoh) {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	178 ctx->have_eoh = TRUE;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	179 if (ctx->stop) {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	180 /* matched is handled differently for eoh by
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	181 istream-header-filter. a design bug I guess.. */
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	182 *matched = FALSE;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	183 }
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	184 } else {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	185 if (strspn(hdr->name, "\r") == hdr->name_len) {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	186 /* CR+CR+LF - some servers stop the header processing
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	187 here while others don't. To make sure they can be
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	188 matched correctly we want to stop here entirely. */
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	189 ctx->stop = TRUE;
22451 e2b9cadf91e4 pop3-migration: Strip away invalid header lines. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22450 diff changeset	190 } else if (!hdr->continued && hdr->middle_len == 0) {
e2b9cadf91e4 pop3-migration: Strip away invalid header lines. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22450 diff changeset	191 /* not a valid "key: value" header -
e2b9cadf91e4 pop3-migration: Strip away invalid header lines. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22450 diff changeset	192 Zimbra's BODY[HEADER] strips this line away. */
e2b9cadf91e4 pop3-migration: Strip away invalid header lines. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22450 diff changeset	193 *matched = TRUE;
22265 66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	194 } else if (hdr->continued && header_value_want_skip(hdr)) {
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	195 *matched = TRUE;
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	196 }
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	197 if (ctx->stop)
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	198 *matched = TRUE;
19624 259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	199 else if (!header_name_is_valid(hdr->name)) {
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	200 /* Yahoo IMAP drops headers with invalid names, while
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	201 Yahoo POP3 preserves them. Drop them all. */
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	202 *matched = TRUE;
22263 096f0c8b9cb1 pop3-migration: Replace trailing whitespace removal with new header hashing version Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22195 diff changeset	203 }
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	204 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	205 }
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	206
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	207 int pop3_migration_get_hdr_sha1(uint32_t mail_seq, struct istream *input,
20328 7e016f5e8cb4 [LEN] to [static LEN] on some function parameters Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 19693 diff changeset	208 unsigned char sha1_r[STATIC_ARRAY SHA1_RESULTLEN],
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	209 bool *have_eoh_r)
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	210 {
19623 7e08d7ae86be pop3-migration: Convert all non-ASCII in headers to '?' Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19619 diff changeset	211 const unsigned char *data;
19691 69502e2013f3 pop3-migration: Use message_header_hash_more() instead of duplicating the code Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19672 diff changeset	212 size_t size;
20769 efadeff4ab8d lib-mail: message_header_hash_more() now allows input in any slices. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20767 diff changeset	213 struct message_header_hash_context hash_ctx;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	214 struct sha1_ctxt sha1_ctx;
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	215 struct pop3_hdr_context hdr_ctx;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	216
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20769 diff changeset	217 i_zero(&hdr_ctx);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	218 /* hide headers that might change or be different in IMAP vs. POP3 */
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	219 input = i_stream_create_header_filter(input, HEADER_FILTER_HIDE_BODY \|
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	220 HEADER_FILTER_EXCLUDE \| HEADER_FILTER_NO_CR,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	221 hdr_hash_skip_headers,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	222 N_ELEMENTS(hdr_hash_skip_headers),
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	223 pop3_header_filter_callback, &hdr_ctx);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	224
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	225 sha1_init(&sha1_ctx);
22263 096f0c8b9cb1 pop3-migration: Replace trailing whitespace removal with new header hashing version Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22195 diff changeset	226 i_zero(&hash_ctx);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	227 while (i_stream_read_data(input, &data, &size, 0) > 0) {
22263 096f0c8b9cb1 pop3-migration: Replace trailing whitespace removal with new header hashing version Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22195 diff changeset	228 message_header_hash_more(&hash_ctx, &hash_method_sha1, &sha1_ctx,
096f0c8b9cb1 pop3-migration: Replace trailing whitespace removal with new header hashing version Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22195 diff changeset	229 MESSAGE_HEADER_HASH_MAX_VERSION,
19691 69502e2013f3 pop3-migration: Use message_header_hash_more() instead of duplicating the code Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19672 diff changeset	230 data, size);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	231 i_stream_skip(input, size);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	232 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	233 if (input->stream_errno != 0) {
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	234 i_error("pop3_migration: Failed to read header for msg %u: %s",
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	235 mail_seq, i_stream_get_error(input));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	236 i_stream_unref(&input);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	237 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	238 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	239 sha1_result(&sha1_ctx, sha1_r);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	240 i_stream_unref(&input);
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	241
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	242 *have_eoh_r = hdr_ctx.have_eoh;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	243 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	244 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	245
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	246 static unsigned int get_cache_idx(struct mail *mail)
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	247 {
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	248 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(mail->box);
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	249
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	250 if (mbox->cache_field_registered)
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	251 return mbox->cache_field.idx;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	252
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	253 mbox->cache_field.name = "pop3-migration.hdr";
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	254 mbox->cache_field.type = MAIL_CACHE_FIELD_FIXED_SIZE;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	255 mbox->cache_field.field_size = SHA1_RESULTLEN;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	256 mail_cache_register_fields(mail->box->cache, &mbox->cache_field, 1);
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	257 mbox->cache_field_registered = TRUE;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	258 return mbox->cache_field.idx;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	259 }
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	260
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	261 static int
20328 7e016f5e8cb4 [LEN] to [static LEN] on some function parameters Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 19693 diff changeset	262 get_hdr_sha1(struct mail *mail, unsigned char sha1_r[STATIC_ARRAY SHA1_RESULTLEN])
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	263 {
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	264 struct istream *input;
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	265 const char *errstr;
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	266 enum mail_error error;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	267 bool have_eoh;
20730 a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	268 int ret;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	269
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	270 if (mail_get_hdr_stream(mail, NULL, &input) < 0) {
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	271 errstr = mailbox_get_last_internal_error(mail->box, &error);
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	272 i_error("pop3_migration: Failed to get header for msg %u: %s",
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	273 mail->seq, errstr);
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	274 return error == MAIL_ERROR_EXPUNGED ? 0 : -1;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	275 }
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	276 if (pop3_migration_get_hdr_sha1(mail->seq, input, sha1_r, &have_eoh) < 0)
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	277 return -1;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	278 if (have_eoh) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	279 struct index_mail imail = (struct index_mail )mail;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	280
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	281 index_mail_cache_add_idx(imail, get_cache_idx(mail),
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	282 sha1_r, SHA1_RESULTLEN);
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	283 return 1;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	284 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	285
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	286 /* The empty "end of headers" line is missing. Either this means that
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	287 the headers ended unexpectedly (which is ok) or that the remote
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	288 server is buggy. Some servers have problems with
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	289
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	290 1) header line continuations that contain only whitespace and
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	291 2) headers that have no ":". The header gets truncated when such
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	292 line is reached.
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	293
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	294 At least Oracle IMS IMAP FETCH BODY[HEADER] handles 1) by not
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	295 returning the whitespace line and 2) by returning the line but
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	296 truncating the rest. POP3 TOP instead returns the entire header.
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	297 This causes the IMAP and POP3 hashes not to match.
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	298
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	299 If there's LF+CR+CR+LF in the middle of headers, Courier IMAP's
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	300 FETCH BODY[HEADER] stops after that, but Courier POP3's TOP doesn't.
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	301
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	302 So we'll try to avoid this by falling back to full FETCH BODY[]
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	303 (and/or RETR) and we'll parse the header ourself from it. This
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	304 should work around any similar bugs in all IMAP/POP3 servers. */
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	305 if (mail_get_stream_because(mail, NULL, NULL, "pop3-migration", &input) < 0) {
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	306 errstr = mailbox_get_last_internal_error(mail->box, &error);
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	307 i_error("pop3_migration: Failed to get body for msg %u: %s",
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	308 mail->seq, errstr);
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	309 return error == MAIL_ERROR_EXPUNGED ? 0 : -1;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	310 }
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	311 ret = pop3_migration_get_hdr_sha1(mail->seq, input, sha1_r, &have_eoh);
20730 a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	312 if (ret == 0) {
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	313 if (!have_eoh)
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	314 i_warning("pop3_migration: Truncated email with UID %u stored as truncated", mail->uid);
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	315 struct index_mail imail = (struct index_mail )mail;
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	316 index_mail_cache_add_idx(imail, get_cache_idx(mail),
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	317 sha1_r, SHA1_RESULTLEN);
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	318 return 1;
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	319 } else {
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	320 return -1;
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	321 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	322 }
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	323
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	324 static bool
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	325 get_cached_hdr_sha1(struct mail mail, buffer_t cache_buf,
20328 7e016f5e8cb4 [LEN] to [static LEN] on some function parameters Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 19693 diff changeset	326 unsigned char sha1_r[STATIC_ARRAY SHA1_RESULTLEN])
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	327 {
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	328 struct index_mail imail = (struct index_mail )mail;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	329
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	330 buffer_set_used_size(cache_buf, 0);
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	331 if (index_mail_cache_lookup_field(imail, cache_buf,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	332 get_cache_idx(mail)) > 0 &&
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	333 cache_buf->used == SHA1_RESULTLEN) {
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	334 memcpy(sha1_r, cache_buf->data, cache_buf->used);
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	335 return TRUE;
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	336 }
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	337 return FALSE;
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	338 }
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	339
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	340 static struct mailbox pop3_mailbox_alloc(struct mail_storage storage)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	341 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	342 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	343 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	344 struct mail_namespace *ns;
21878 e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	345 struct mailbox *box;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	346
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	347 ns = mail_namespace_find(storage->user->namespaces,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	348 mstorage->pop3_box_vname);
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	349 i_assert(ns != NULL);
21878 e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	350 box = mailbox_alloc(ns->list, mstorage->pop3_box_vname,
e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	351 MAILBOX_FLAG_READONLY \| MAILBOX_FLAG_POP3_SESSION);
e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	352 mailbox_set_reason(box, "pop3_migration");
e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	353 return box;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	354 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	355
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	356 static int pop3_map_read(struct mail_storage storage, struct mailbox pop3_box)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	357 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	358 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	359 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	360 struct mailbox_transaction_context *t;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	361 struct mail_search_args *search_args;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	362 struct mail_search_context *ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	363 struct mail *mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	364 struct pop3_uidl_map *map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	365 const char *uidl;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	366 uoff_t size = (uoff_t)-1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	367 int ret = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	368
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	369 if (array_is_created(&mstorage->pop3_uidl_map)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	370 /* already read these, just reset the imap_uids */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	371 array_foreach_modifiable(&mstorage->pop3_uidl_map, map)
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	372 map->imap_uid = 0;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	373 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	374 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	375 i_array_init(&mstorage->pop3_uidl_map, 128);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	376
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	377 if (mailbox_sync(pop3_box, 0) < 0) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	378 i_error("pop3_migration: Couldn't sync mailbox %s: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	379 pop3_box->vname, mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	380 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	381 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	382
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	383 t = mailbox_transaction_begin(pop3_box, 0);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	384 search_args = mail_search_build_init();
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	385 mail_search_build_add_all(search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	386 ctx = mailbox_search_init(t, search_args, NULL,
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	387 mstorage->skip_size_check ? 0 :
18912 64568a033fc8 pop3-migration: Fetch physical sizes instead of virtual sizes so pop3c uses LIST Timo Sirainen <tss@iki.fi> parents: 18911 diff changeset	388 MAIL_FETCH_PHYSICAL_SIZE, NULL);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	389 mail_search_args_unref(&search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	390
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	391 while (mailbox_search_next(ctx, &mail)) {
18911 4bebfbb32410 pop3-migration: Use LIST instead of RETRs to get the messages' sizes. Timo Sirainen <tss@iki.fi> parents: 18909 diff changeset	392 /* get the size with LIST instead of RETR */
4bebfbb32410 pop3-migration: Use LIST instead of RETRs to get the messages' sizes. Timo Sirainen <tss@iki.fi> parents: 18909 diff changeset	393 mail->lookup_abort = MAIL_LOOKUP_ABORT_READ_MAIL;
4bebfbb32410 pop3-migration: Use LIST instead of RETRs to get the messages' sizes. Timo Sirainen <tss@iki.fi> parents: 18909 diff changeset	394
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	395 if (mstorage->skip_size_check)
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	396 ;
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	397 else if (mail_get_physical_size(mail, &size) < 0) {
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	398 i_error("pop3_migration: Failed to get size for msg %u: %s",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	399 mail->seq,
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	400 mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	401 ret = -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	402 break;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	403 }
21649 78f38a28086e lib-storage, pop3-migration: Reset lookup_abort before continuing to search Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21390 diff changeset	404 mail->lookup_abort = MAIL_LOOKUP_ABORT_NEVER;
78f38a28086e lib-storage, pop3-migration: Reset lookup_abort before continuing to search Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21390 diff changeset	405
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	406 if (mail_get_special(mail, MAIL_FETCH_UIDL_BACKEND, &uidl) < 0) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	407 i_error("pop3_migration: Failed to get UIDL for msg %u: %s",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	408 mail->seq,
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	409 mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	410 ret = -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	411 break;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	412 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	413 if (*uidl == '\0') {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	414 i_warning("pop3_migration: UIDL for msg %u is empty",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	415 mail->seq);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	416 continue;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	417 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	418
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	419 map = array_append_space(&mstorage->pop3_uidl_map);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	420 map->pop3_seq = mail->seq;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	421 map->pop3_uidl = p_strdup(storage->pool, uidl);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	422 map->size = size;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	423 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	424
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	425 if (mailbox_search_deinit(&ctx) < 0) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	426 i_error("pop3_migration: Failed to search all POP3 mails: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	427 mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	428 ret = -1;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	429 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	430 (void)mailbox_transaction_commit(&t);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	431 return ret;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	432 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	433
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	434 static void
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	435 pop3_map_read_cached_hdr_hashes(struct mailbox_transaction_context *t,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	436 struct mail_search_args *search_args,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	437 struct array *msg_map)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	438 {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	439 struct mail_search_context *ctx;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	440 struct mail *mail;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	441 struct msg_map_common *map;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	442 buffer_t *cache_buf;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	443
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	444 ctx = mailbox_search_init(t, search_args, NULL, 0, NULL);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	445 cache_buf = buffer_create_dynamic(pool_datastack_create(), SHA1_RESULTLEN);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	446
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	447 while (mailbox_search_next(ctx, &mail)) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	448 map = array_idx_modifiable_i(msg_map, mail->seq-1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	449
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	450 if (get_cached_hdr_sha1(mail, cache_buf, map->hdr_sha1))
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	451 map->hdr_sha1_set = TRUE;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	452 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	453
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	454 if (mailbox_search_deinit(&ctx) < 0) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	455 i_warning("pop3_migration: Failed to search all cached POP3 header hashes: %s - ignoring",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	456 mailbox_get_last_internal_error(t->box, NULL));
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	457 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	458 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	459
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	460 static void map_remove_found_seqs(struct mail_search_arg *search_arg,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	461 struct array *msg_map, uint32_t seq1)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	462 {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	463 const struct msg_map_common *map;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	464 uint32_t seq, count = array_count_i(msg_map);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	465
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	466 i_assert(search_arg->type == SEARCH_SEQSET);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	467
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	468 for (seq = seq1; seq <= count; seq++) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	469 map = array_idx_i(msg_map, seq-1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	470 if (map->hdr_sha1_set)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	471 seq_range_array_remove(&search_arg->value.seqset, seq);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	472 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	473 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	474
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	475 static int
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	476 map_read_hdr_hashes(struct mailbox box, struct array msg_map, uint32_t seq1)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	477 {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	478 struct mailbox_transaction_context *t;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	479 struct mail_search_args *search_args;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	480 struct mail_search_context *ctx;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	481 struct mail *mail;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	482 struct msg_map_common *map;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	483 int ret = 0;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	484
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	485 t = mailbox_transaction_begin(box, 0);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	486 /* get all the cached hashes */
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	487 search_args = mail_search_build_init();
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	488 mail_search_build_add_seqset(search_args, seq1, array_count_i(msg_map));
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	489 pop3_map_read_cached_hdr_hashes(t, search_args, msg_map);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	490 /* read all the non-cached hashes. doing this in two passes allows
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	491 us to set wanted_fields=MAIL_FETCH_STREAM_HEADER, which allows
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	492 prefetching to work without downloading all the headers even
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	493 for mails that already are cached. */
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	494 map_remove_found_seqs(search_args->args, msg_map, seq1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	495 ctx = mailbox_search_init(t, search_args, NULL,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	496 MAIL_FETCH_STREAM_HEADER, NULL);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	497 mail_search_args_unref(&search_args);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	498
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	499 while (mailbox_search_next(ctx, &mail)) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	500 map = array_idx_modifiable_i(msg_map, mail->seq-1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	501
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	502 if ((ret = get_hdr_sha1(mail, map->hdr_sha1)) < 0) {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	503 ret = -1;
19647 8f2ba9ebc463 pop3-migration: If reading message header for hashing fails, fail immediately. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19646 diff changeset	504 break;
8f2ba9ebc463 pop3-migration: If reading message header for hashing fails, fail immediately. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19646 diff changeset	505 }
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	506 if (ret > 0)
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	507 map->hdr_sha1_set = TRUE;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	508 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	509
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	510 if (mailbox_search_deinit(&ctx) < 0) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	511 i_error("pop3_migration: Failed to search all mail headers: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	512 mailbox_get_last_internal_error(box, NULL));
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	513 ret = -1;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	514 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	515 (void)mailbox_transaction_commit(&t);
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	516 return ret < 0 ? -1 : 0;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	517 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	518
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	519 static int
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	520 pop3_map_read_hdr_hashes(struct mail_storage storage, struct mailbox pop3_box,
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	521 unsigned first_seq)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	522 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	523 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	524 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	525
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	526 if (mstorage->pop3_all_hdr_sha1_set)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	527 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	528 if (mstorage->all_mailboxes) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	529 /* we may be matching against multiple mailboxes.
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	530 read all the hashes only once. */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	531 first_seq = 1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	532 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	533
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	534 if (map_read_hdr_hashes(pop3_box, &mstorage->pop3_uidl_map.arr,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	535 first_seq) < 0)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	536 return -1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	537
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	538 if (first_seq == 1)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	539 mstorage->pop3_all_hdr_sha1_set = TRUE;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	540 return 0;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	541 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	542
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	543 static int imap_map_read(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	544 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	545 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	546 struct index_mailbox_context *ibox = INDEX_STORAGE_CONTEXT(box);
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	547 struct pop3_migration_mail_storage *mstorage =
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	548 POP3_MIGRATION_CONTEXT(box->storage);
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	549 const unsigned int uidl_cache_idx =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	550 ibox->cache_fields[MAIL_CACHE_POP3_UIDL].idx;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	551 struct mailbox_status status;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	552 struct mailbox_transaction_context *t;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	553 struct mail_search_args *search_args;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	554 struct mail_search_context *ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	555 struct mail *mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	556 struct imap_msg_map *map;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	557 uoff_t psize = (uoff_t)-1;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	558 string_t *uidl;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	559 int ret = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	560
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	561 mailbox_get_open_status(box, STATUS_MESSAGES, &status);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	562
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	563 i_assert(!array_is_created(&mbox->imap_msg_map));
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	564 p_array_init(&mbox->imap_msg_map, box->pool, status.messages);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	565
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	566 t = mailbox_transaction_begin(box, 0);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	567 search_args = mail_search_build_init();
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	568 mail_search_build_add_all(search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	569 ctx = mailbox_search_init(t, search_args, NULL,
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	570 mstorage->skip_size_check ? 0 :
14402 87cd04f35f82 pop3-replication: Don't request virtual size from IMAP mailbox, we don't need it. Timo Sirainen <tss@iki.fi> parents: 14401 diff changeset	571 MAIL_FETCH_PHYSICAL_SIZE, NULL);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	572 mail_search_args_unref(&search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	573
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	574 uidl = t_str_new(64);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	575 while (mailbox_search_next(ctx, &mail)) {
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	576 if (mstorage->skip_size_check)
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	577 ;
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	578 else if (mail_get_physical_size(mail, &psize) < 0) {
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	579 i_error("pop3_migration: Failed to get psize for imap uid %u: %s",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	580 mail->uid,
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	581 mailbox_get_last_internal_error(box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	582 ret = -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	583 break;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	584 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	585
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	586 if (!mstorage->skip_uidl_cache) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	587 str_truncate(uidl, 0);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	588 (void)mail_cache_lookup_field(mail->transaction->cache_view,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	589 uidl, mail->seq, uidl_cache_idx);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	590 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	591
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	592 map = array_append_space(&mbox->imap_msg_map);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	593 map->uid = mail->uid;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	594 map->psize = psize;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	595 map->pop3_uidl = p_strdup_empty(box->pool, str_c(uidl));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	596 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	597
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	598 if (mailbox_search_deinit(&ctx) < 0) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	599 i_error("pop3_migration: Failed to search all IMAP mails: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	600 mailbox_get_last_internal_error(box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	601 ret = -1;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	602 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	603 (void)mailbox_transaction_commit(&t);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	604 return ret;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	605 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	606
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	607 static int imap_map_read_hdr_hashes(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	608 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	609 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	610
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	611 return map_read_hdr_hashes(box, &mbox->imap_msg_map.arr,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	612 mbox->first_unfound_idx+1);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	613 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	614
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	615 static void pop3_uidl_assign_cached(struct mailbox *box)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	616 {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	617 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	618 struct pop3_migration_mail_storage *mstorage =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	619 POP3_MIGRATION_CONTEXT(box->storage);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	620 struct pop3_uidl_map *pop3_map;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	621 struct imap_msg_map *imap_map;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	622 unsigned int imap_idx, pop3_idx, pop3_count, imap_count;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	623 int ret;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	624
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	625 if (mstorage->skip_uidl_cache)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	626 return;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	627
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	628 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_uidl_cmp);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	629 array_sort(&mbox->imap_msg_map, imap_msg_map_uidl_cmp);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	630
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	631 pop3_map = array_get_modifiable(&mstorage->pop3_uidl_map, &pop3_count);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	632 imap_map = array_get_modifiable(&mbox->imap_msg_map, &imap_count);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	633
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	634 /* see if we can match the messages using sizes */
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	635 for (imap_idx = pop3_idx = 0; imap_idx < imap_count; imap_idx++) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	636 if (imap_map[imap_idx].pop3_uidl == NULL)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	637 continue;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	638
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	639 ret = 1;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	640 for (; pop3_idx < pop3_count; pop3_idx++) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	641 ret = strcmp(imap_map[imap_idx].pop3_uidl,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	642 pop3_map[pop3_idx].pop3_uidl);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	643 if (ret >= 0)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	644 break;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	645 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	646 if (ret == 0) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	647 imap_map[imap_idx].pop3_seq =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	648 pop3_map[pop3_idx].pop3_seq;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	649 pop3_map[pop3_idx].imap_uid = imap_map[imap_idx].uid;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	650 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	651 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	652 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	653
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	654 static bool pop3_uidl_assign_by_size(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	655 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	656 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	657 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	658 POP3_MIGRATION_CONTEXT(box->storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	659 struct pop3_uidl_map *pop3_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	660 struct imap_msg_map *imap_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	661 unsigned int i, pop3_count, imap_count, count;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	662 unsigned int size_match = 0, uidl_match = 0;
19644 defab5b9af2a pop3-migration: Don't even try to match message sizes with pop3_migration_skip_size_check=yes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19640 diff changeset	663
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	664 pop3_map = array_get_modifiable(&mstorage->pop3_uidl_map, &pop3_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	665 imap_map = array_get_modifiable(&mbox->imap_msg_map, &imap_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	666 count = I_MIN(pop3_count, imap_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	667
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	668 /* see if we can match the messages using sizes */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	669 for (i = 0; i < count; i++) {
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	670 if (imap_map[i].pop3_uidl != NULL) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	671 /* some of the UIDLs were already found cached. */
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	672 if (strcmp(pop3_map[i].pop3_uidl, imap_map[i].pop3_uidl) == 0) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	673 uidl_match++;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	674 continue;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	675 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	676 /* mismatch - can't trust the sizes */
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	677 break;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	678 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	679
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	680 if (pop3_map[i].size != imap_map[i].psize \|\|
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	681 mstorage->skip_size_check)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	682 break;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	683 if (i+1 < count && pop3_map[i].size == pop3_map[i+1].size) {
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	684 /* two messages with same size, don't trust them */
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	685 break;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	686 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	687
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	688 size_match++;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	689 pop3_map[i].imap_uid = imap_map[i].uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	690 imap_map[i].pop3_uidl = pop3_map[i].pop3_uidl;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	691 imap_map[i].pop3_seq = pop3_map[i].pop3_seq;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	692 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	693 mbox->first_unfound_idx = i;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	694 if (box->storage->user->mail_debug) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	695 i_debug("pop3_migration: cached uidls=%u, size matches=%u, total=%u",
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	696 uidl_match, size_match, count);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	697 }
22453 6eb5a6930614 pop3-migration: Fail if all IMAP mails were matched by size, but POP3 had extra mails Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22452 diff changeset	698 return i == count && imap_count == pop3_count;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	699 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	700
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	701 static int
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	702 pop3_uidl_assign_by_hdr_hash(struct mailbox box, struct mailbox pop3_box)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	703 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	704 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	705 POP3_MIGRATION_CONTEXT(box->storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	706 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	707 struct pop3_uidl_map *pop3_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	708 struct imap_msg_map *imap_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	709 unsigned int pop3_idx, imap_idx, pop3_count, imap_count;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	710 unsigned int first_seq, missing_uids_count;
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	711 uint32_t first_missing_idx = 0, first_missing_seq = (uint32_t)-1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	712 int ret;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	713
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	714 first_seq = mbox->first_unfound_idx+1;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	715 if (pop3_map_read_hdr_hashes(box->storage, pop3_box, first_seq) < 0 \|\|
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	716 imap_map_read_hdr_hashes(box) < 0)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	717 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	718
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	719 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_hdr_cmp);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	720 array_sort(&mbox->imap_msg_map, imap_msg_map_hdr_cmp);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	721
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	722 pop3_map = array_get_modifiable(&mstorage->pop3_uidl_map, &pop3_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	723 imap_map = array_get_modifiable(&mbox->imap_msg_map, &imap_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	724
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	725 pop3_idx = imap_idx = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	726 while (pop3_idx < pop3_count && imap_idx < imap_count) {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	727 if (!pop3_map[pop3_idx].common.hdr_sha1_set \|\|
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	728 pop3_map[pop3_idx].imap_uid != 0) {
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	729 pop3_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	730 continue;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	731 }
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	732 if (!imap_map[imap_idx].common.hdr_sha1_set \|\|
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	733 imap_map[imap_idx].pop3_uidl != NULL) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	734 imap_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	735 continue;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	736 }
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	737 ret = memcmp(pop3_map[pop3_idx].common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	738 imap_map[imap_idx].common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	739 sizeof(pop3_map[pop3_idx].common.hdr_sha1));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	740 if (ret < 0)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	741 pop3_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	742 else if (ret > 0)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	743 imap_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	744 else {
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	745 pop3_map[pop3_idx].imap_uid = imap_map[imap_idx].uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	746 imap_map[imap_idx].pop3_uidl =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	747 pop3_map[pop3_idx].pop3_uidl;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	748 imap_map[imap_idx].pop3_seq =
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	749 pop3_map[pop3_idx].pop3_seq;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	750 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	751 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	752 missing_uids_count = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	753 for (pop3_idx = 0; pop3_idx < pop3_count; pop3_idx++) {
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	754 if (pop3_map[pop3_idx].imap_uid != 0) {
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	755 /* matched */
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	756 } else if (!pop3_map[pop3_idx].common.hdr_sha1_set) {
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	757 /* we treated this mail as expunged - ignore */
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	758 } else {
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	759 uint32_t seq = pop3_map[pop3_idx].pop3_seq;
6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	760 if (first_missing_seq > seq) {
6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	761 first_missing_seq = seq;
19617 7b0a093df334 pop3-migration: When logging the first missing POP3 mail, it logged a wrong mail Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19616 diff changeset	762 first_missing_idx = pop3_idx;
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	763 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	764 missing_uids_count++;
18908 b51dfee18fd2 pop3-migration: Show the first message's number and UIDL which wasn't found from IMAP. Timo Sirainen <tss@iki.fi> parents: 18624 diff changeset	765 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	766 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	767 if (missing_uids_count > 0 && !mstorage->all_mailboxes) {
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	768 string_t *str = t_str_new(128);
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	769 bool all_imap_mails_found = FALSE;
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	770
af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	771 str_printfa(str, "pop3_migration: %u POP3 messages have no "
af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	772 "matching IMAP messages (first POP3 msg %u UIDL %s)",
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	773 missing_uids_count, first_missing_seq,
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	774 pop3_map[first_missing_idx].pop3_uidl);
19619 c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	775 if (imap_count + missing_uids_count == pop3_count) {
c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	776 str_append(str, " - all IMAP messages were found "
c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	777 "(POP3 contains more than IMAP INBOX - you may want to set pop3_migration_all_mailboxes=yes)");
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	778 all_imap_mails_found = TRUE;
19619 c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	779 }
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	780 if (all_imap_mails_found && mstorage->ignore_extra_uidls) {
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	781 /* pop3 had more mails than imap. maybe it was just
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	782 that a new mail was just delivered. */
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	783 } else if (!mstorage->ignore_missing_uidls) {
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	784 str_append(str, " - set pop3_migration_ignore_missing_uidls=yes");
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	785 if (all_imap_mails_found)
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	786 str_append(str, " or pop3_migration_ignore_extra_uidls=yes");
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	787 i_error("%s to continue anyway", str_c(str));
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	788 return -1;
1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	789 }
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	790 i_warning("%s", str_c(str));
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	791 } else if (box->storage->user->mail_debug) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	792 i_debug("pop3_migration: %u mails matched by headers", pop3_count);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	793 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	794 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_pop3_seq_cmp);
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	795 array_sort(&mbox->imap_msg_map, imap_msg_map_uid_cmp);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	796 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	797 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	798
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	799 static void imap_uidls_add_to_cache(struct mailbox *box)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	800 {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	801 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	802 struct mailbox_transaction_context *t;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	803 struct mail *mail;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	804 struct index_mail *imail;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	805 struct imap_msg_map *imap_map;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	806 unsigned int i, count;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	807 unsigned int field_idx;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	808
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	809 t = mailbox_transaction_begin(box, 0);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	810 mail = mail_alloc(t, 0, NULL);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	811 imail = (struct index_mail *)mail;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	812 field_idx = imail->ibox->cache_fields[MAIL_CACHE_POP3_UIDL].idx;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	813
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	814 imap_map = array_get_modifiable(&mbox->imap_msg_map, &count);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	815 for (i = 0; i < count; i++) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	816 if (imap_map[i].pop3_uidl == NULL)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	817 continue;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	818
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	819 if (!mail_set_uid(mail, imap_map[i].uid))
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	820 i_unreached();
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	821 if (mail_cache_field_can_add(t->cache_trans, mail->seq, field_idx)) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	822 index_mail_cache_add_idx(imail, field_idx,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	823 imap_map[i].pop3_uidl, strlen(imap_map[i].pop3_uidl)+1);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	824 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	825 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	826 mail_free(&mail);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	827 (void)mailbox_transaction_commit(&t);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	828 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	829
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	830 static int pop3_migration_uidl_sync(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	831 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	832 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	833 struct pop3_migration_mail_storage *mstorage =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	834 POP3_MIGRATION_CONTEXT(box->storage);
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	835 struct mailbox *pop3_box;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	836
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	837 pop3_box = pop3_mailbox_alloc(box->storage);
14835 ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	838 /* the POP3 server isn't connected to yet. handle all IMAP traffic
ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	839 first before connecting, so POP3 server won't disconnect us due to
ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	840 idling. */
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	841 if (imap_map_read(box) < 0 \|\|
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	842 pop3_map_read(box->storage, pop3_box) < 0) {
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	843 mailbox_free(&pop3_box);
14835 ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	844 return -1;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	845 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	846
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	847 pop3_uidl_assign_cached(box);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	848
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	849 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_pop3_seq_cmp);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	850 array_sort(&mbox->imap_msg_map, imap_msg_map_uid_cmp);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	851
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	852 if (!pop3_uidl_assign_by_size(box)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	853 /* everything wasn't assigned, figure out the rest with
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	854 header hashes */
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	855 if (pop3_uidl_assign_by_hdr_hash(box, pop3_box) < 0) {
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	856 mailbox_free(&pop3_box);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	857 return -1;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	858 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	859 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	860
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	861 if (!mstorage->skip_uidl_cache)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	862 imap_uidls_add_to_cache(box);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	863
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	864 mbox->uidl_synced = TRUE;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	865 mailbox_free(&pop3_box);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	866 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	867 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	868
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	869 static int pop3_migration_uidl_sync_if_needed(struct mailbox *box)
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	870 {
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	871 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	872
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	873 if (mbox->uidl_synced)
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	874 return 0;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	875
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	876 if (mbox->uidl_sync_failed \|\|
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	877 pop3_migration_uidl_sync(box) < 0) {
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	878 mbox->uidl_sync_failed = TRUE;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	879 mail_storage_set_error(box->storage, MAIL_ERROR_TEMP,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	880 "POP3 UIDLs couldn't be synced");
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	881 return -1;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	882 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	883 return 0;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	884 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	885
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	886 static int
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	887 pop3_migration_get_special(struct mail *_mail, enum mail_fetch_field field,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	888 const char **value_r)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	889 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	890 struct mail_private mail = (struct mail_private )_mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	891 union mail_module_context *mmail = POP3_MIGRATION_MAIL_CONTEXT(mail);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	892 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(_mail->box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	893 struct imap_msg_map map_key, *map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	894
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	895 if (field == MAIL_FETCH_UIDL_BACKEND \|\|
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	896 field == MAIL_FETCH_POP3_ORDER) {
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	897 if (pop3_migration_uidl_sync_if_needed(_mail->box) < 0)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	898 return -1;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	899
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20769 diff changeset	900 i_zero(&map_key);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	901 map_key.uid = _mail->uid;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	902 map = array_bsearch(&mbox->imap_msg_map, &map_key,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	903 imap_msg_map_uid_cmp);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	904 if (map != NULL && map->pop3_uidl != NULL) {
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	905 if (field == MAIL_FETCH_UIDL_BACKEND)
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	906 *value_r = map->pop3_uidl;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	907 else
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	908 *value_r = t_strdup_printf("%u", map->pop3_seq);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	909 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	910 }
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	911 /* not found from POP3 server, fallback to default */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	912 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	913 return mmail->super.get_special(_mail, field, value_r);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	914 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	915
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	916 static void pop3_migration_mail_allocated(struct mail *_mail)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	917 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	918 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	919 POP3_MIGRATION_CONTEXT(_mail->box->storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	920 struct mail_private mail = (struct mail_private )_mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	921 struct mail_vfuncs *v = mail->vlast;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	922 union mail_module_context *mmail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	923 struct mail_namespace *ns;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	924
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	925 if (mstorage == NULL \|\|
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	926 (!mstorage->all_mailboxes && !_mail->box->inbox_user)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	927 /* assigns UIDLs only for INBOX */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	928 return;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	929 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	930
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	931 ns = mail_namespace_find(_mail->box->storage->user->namespaces,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	932 mstorage->pop3_box_vname);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	933 if (ns == mailbox_get_namespace(_mail->box)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	934 /* we're accessing the pop3-migration namespace itself */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	935 return;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	936 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	937
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	938 mmail = p_new(mail->pool, union mail_module_context, 1);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	939 mmail->super = *v;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	940 mail->vlast = &mmail->super;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	941
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	942 v->get_special = pop3_migration_get_special;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	943 MODULE_CONTEXT_SET_SELF(mail, pop3_migration_mail_module, mmail);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	944 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	945
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	946 static struct mail_search_context *
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	947 pop3_migration_mailbox_search_init(struct mailbox_transaction_context *t,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	948 struct mail_search_args *args,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	949 const enum mail_sort_type *sort_program,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	950 enum mail_fetch_field wanted_fields,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	951 struct mailbox_header_lookup_ctx *wanted_headers)
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	952 {
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	953 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(t->box);
19672 e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	954 struct pop3_migration_mail_storage *mstorage =
e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	955 POP3_MIGRATION_CONTEXT(t->box->storage);
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	956
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	957 if ((wanted_fields & (MAIL_FETCH_UIDL_BACKEND \|
19672 e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	958 MAIL_FETCH_POP3_ORDER)) != 0 &&
e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	959 (mstorage->all_mailboxes \|\| t->box->inbox_user)) {
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	960 /* Start POP3 UIDL syncing before the search, so we'll do it
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	961 before we start sending any FETCH BODY[]s to IMAP. It
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	962 shouldn't matter much, except this works around a bug in
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	963 Yahoo IMAP where it sometimes breaks its state when doing
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	964 a FETCH BODY[] followed by FETCH BODY[HEADER].. */
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	965 (void)pop3_migration_uidl_sync_if_needed(t->box);
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	966 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	967
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	968 return mbox->module_ctx.super.search_init(t, args, sort_program,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	969 wanted_fields, wanted_headers);
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	970 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	971
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	972 static void pop3_migration_mailbox_allocated(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	973 {
20746 6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	974 struct pop3_migration_mail_storage *mstorage =
6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	975 POP3_MIGRATION_CONTEXT(box->storage);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	976 struct mailbox_vfuncs *v = box->vlast;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	977 struct pop3_migration_mailbox *mbox;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	978
20746 6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	979 if (mstorage == NULL)
6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	980 return;
6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	981
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	982 mbox = p_new(box->pool, struct pop3_migration_mailbox, 1);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	983 mbox->module_ctx.super = *v;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	984 box->vlast = &mbox->module_ctx.super;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	985
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	986 v->search_init = pop3_migration_mailbox_search_init;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	987
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	988 MODULE_CONTEXT_SET(box, pop3_migration_storage_module, mbox);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	989 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	990
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	991 static void pop3_migration_mail_storage_destroy(struct mail_storage *storage)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	992 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	993 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	994 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	995
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	996 if (array_is_created(&mstorage->pop3_uidl_map))
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	997 array_free(&mstorage->pop3_uidl_map);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	998
15272 ca10d2e8d8e3 lib-storage: Moved mail attribute dict to struct mail_storage. Timo Sirainen <tss@iki.fi> parents: 15199 diff changeset	999 mstorage->module_ctx.super.destroy(storage);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1000 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1001
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1002 static void pop3_migration_mail_storage_created(struct mail_storage *storage)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1003 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1004 struct pop3_migration_mail_storage *mstorage;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1005 struct mail_storage_vfuncs *v = storage->vlast;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1006 const char *pop3_box_vname;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1007
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1008 pop3_box_vname = mail_user_plugin_getenv(storage->user,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1009 "pop3_migration_mailbox");
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	1010 if (pop3_box_vname == NULL) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	1011 if (storage->user->mail_debug)
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	1012 i_debug("pop3_migration: No pop3_migration_mailbox setting - disabled");
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1013 return;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	1014 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1015
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1016 mstorage = p_new(storage->pool, struct pop3_migration_mail_storage, 1);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1017 mstorage->module_ctx.super = *v;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1018 storage->vlast = &mstorage->module_ctx.super;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1019 v->destroy = pop3_migration_mail_storage_destroy;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1020
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1021 mstorage->pop3_box_vname = p_strdup(storage->pool, pop3_box_vname);
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	1022 mstorage->all_mailboxes =
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	1023 mail_user_plugin_getenv(storage->user,
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	1024 "pop3_migration_all_mailboxes") != NULL;
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	1025 mstorage->ignore_missing_uidls =
1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	1026 mail_user_plugin_getenv(storage->user,
1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	1027 "pop3_migration_ignore_missing_uidls") != NULL;
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	1028 mstorage->ignore_extra_uidls =
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	1029 mail_user_plugin_getenv(storage->user,
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	1030 "pop3_migration_ignore_extra_uidls") != NULL;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	1031 mstorage->skip_size_check =
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	1032 mail_user_plugin_getenv(storage->user,
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	1033 "pop3_migration_skip_size_check") != NULL;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	1034 mstorage->skip_uidl_cache =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	1035 mail_user_plugin_getenv(storage->user,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	1036 "pop3_migration_skip_uidl_cache") != NULL;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1037
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1038 MODULE_CONTEXT_SET(storage, pop3_migration_storage_module, mstorage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1039 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1040
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1041 static struct mail_storage_hooks pop3_migration_mail_storage_hooks = {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1042 .mail_allocated = pop3_migration_mail_allocated,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1043 .mailbox_allocated = pop3_migration_mailbox_allocated,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1044 .mail_storage_created = pop3_migration_mail_storage_created
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1045 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1046
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1047 void pop3_migration_plugin_init(struct module *module)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1048 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1049 mail_storage_hooks_add(module, &pop3_migration_mail_storage_hooks);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1050 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1051
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1052 void pop3_migration_plugin_deinit(void)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1053 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1054 mail_storage_hooks_remove(&pop3_migration_mail_storage_hooks);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1055 }

Mercurial > dovecot > core-2.2

annotate src/plugins/pop3-migration/pop3-migration-plugin.c @ 22453:6eb5a6930614