dovecot/core-2.2: src/plugins/pop3-migration/pop3-migration-plugin.c annotate

annotate src/plugins/pop3-migration/pop3-migration-plugin.c @ 22713:cb108f786fb4

Updated copyright notices to include the year 2018.

author	Stephan Bosch <stephan.bosch@dovecot.fi>
date	Mon, 01 Jan 2018 22:42:08 +0100
parents	6eb5a6930614
children

rev	line source
22713 cb108f786fb4 Updated copyright notices to include the year 2018. Stephan Bosch <stephan.bosch@dovecot.fi> parents: 22453 diff changeset	1 /* Copyright (c) 2007-2018 Dovecot authors, see the included COPYING file */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	2
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	3 #include "lib.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	4 #include "array.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	5 #include "istream.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	6 #include "istream-header-filter.h"
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	7 #include "str.h"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	8 #include "sha1.h"
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	9 #include "message-size.h"
19691 69502e2013f3 pop3-migration: Use message_header_hash_more() instead of duplicating the code Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19672 diff changeset	10 #include "message-header-hash.h"
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	11 #include "message-header-parser.h"
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	12 #include "mail-cache.h"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	13 #include "mail-namespace.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	14 #include "mail-search-build.h"
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	15 #include "index-storage.h"
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	16 #include "index-mail.h"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	17 #include "pop3-migration-plugin.h"
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	18
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	19 #define POP3_MIGRATION_CONTEXT(obj) \
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	20 MODULE_CONTEXT(obj, pop3_migration_storage_module)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	21 #define POP3_MIGRATION_MAIL_CONTEXT(obj) \
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	22 MODULE_CONTEXT(obj, pop3_migration_mail_module)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	23
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	24 struct msg_map_common {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	25 /* sha1(header) - set only when needed */
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	26 unsigned char hdr_sha1[SHA1_RESULTLEN];
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	27 unsigned int hdr_sha1_set:1;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	28 };
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	29
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	30 struct pop3_uidl_map {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	31 struct msg_map_common common;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	32
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	33 uint32_t pop3_seq;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	34 uint32_t imap_uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	35
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	36 /* UIDL */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	37 const char *pop3_uidl;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	38 /* LIST size */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	39 uoff_t size;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	40 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	41
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	42 struct imap_msg_map {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	43 struct msg_map_common common;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	44
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	45 uint32_t uid, pop3_seq;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	46 uoff_t psize;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	47 const char *pop3_uidl;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	48 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	49
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	50 struct pop3_migration_mail_storage {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	51 union mail_storage_module_context module_ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	52
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	53 const char *pop3_box_vname;
14920 a097ef0a9d6d Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Timo Sirainen <tss@iki.fi> parents: 14835 diff changeset	54 ARRAY(struct pop3_uidl_map) pop3_uidl_map;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	55
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	56 unsigned int all_mailboxes:1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	57 unsigned int pop3_all_hdr_sha1_set:1;
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	58 unsigned int ignore_missing_uidls:1;
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	59 unsigned int ignore_extra_uidls:1;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	60 unsigned int skip_size_check:1;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	61 unsigned int skip_uidl_cache:1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	62 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	63
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	64 struct pop3_migration_mailbox {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	65 union mailbox_module_context module_ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	66
14920 a097ef0a9d6d Array API changed: ARRAY_DEFINE(name, type) -> ARRAY(type) name Timo Sirainen <tss@iki.fi> parents: 14835 diff changeset	67 ARRAY(struct imap_msg_map) imap_msg_map;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	68 unsigned int first_unfound_idx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	69
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	70 struct mail_cache_field cache_field;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	71
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	72 unsigned int cache_field_registered:1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	73 unsigned int uidl_synced:1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	74 unsigned int uidl_sync_failed:1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	75 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	76
19614 87d9089d2c76 pop3-migration: Ignore Return-Path: header when matching UIDLs by header hashes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	77 /* NOTE: these headers must be sorted */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	78 static const char *hdr_hash_skip_headers[] = {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	79 "Content-Length",
19614 87d9089d2c76 pop3-migration: Ignore Return-Path: header when matching UIDLs by header hashes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19552 diff changeset	80 "Return-Path", /* Yahoo IMAP has Return-Path, Yahoo POP3 doesn't */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	81 "Status",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	82 "X-IMAP",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	83 "X-IMAPbase",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	84 "X-Keywords",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	85 "X-Message-Flag",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	86 "X-Status",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	87 "X-UID",
19618 591fa1448786 pop3-migration: Ignore X-Yahoo-Newman-Property: header Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19617 diff changeset	88 "X-UIDL",
591fa1448786 pop3-migration: Ignore X-Yahoo-Newman-Property: header Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19617 diff changeset	89 "X-Yahoo-Newman-Property"
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	90 };
15199 73916b7be94e Plugin ABI version checking improvements. Timo Sirainen <tss@iki.fi> parents: 14921 diff changeset	91 const char *pop3_migration_plugin_version = DOVECOT_ABI_VERSION;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	92
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	93 static MODULE_CONTEXT_DEFINE_INIT(pop3_migration_storage_module,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	94 &mail_storage_module_register);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	95 static MODULE_CONTEXT_DEFINE_INIT(pop3_migration_mail_module,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	96 &mail_module_register);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	97
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	98 static int imap_msg_map_uid_cmp(const struct imap_msg_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	99 const struct imap_msg_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	100 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	101 if (map1->uid < map2->uid)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	102 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	103 if (map1->uid > map2->uid)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	104 return 1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	105 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	106 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	107
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	108 static int pop3_uidl_map_pop3_seq_cmp(const struct pop3_uidl_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	109 const struct pop3_uidl_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	110 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	111 if (map1->pop3_seq < map2->pop3_seq)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	112 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	113 if (map1->pop3_seq > map2->pop3_seq)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	114 return 1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	115 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	116 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	117
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	118 static int pop3_uidl_map_uidl_cmp(const struct pop3_uidl_map *map1,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	119 const struct pop3_uidl_map *map2)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	120 {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	121 return strcmp(map1->pop3_uidl, map2->pop3_uidl);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	122 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	123
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	124 static int imap_msg_map_uidl_cmp(const struct imap_msg_map *map1,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	125 const struct imap_msg_map *map2)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	126 {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	127 return null_strcmp(map1->pop3_uidl, map2->pop3_uidl);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	128 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	129
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	130 static int pop3_uidl_map_hdr_cmp(const struct pop3_uidl_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	131 const struct pop3_uidl_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	132 {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	133 return memcmp(map1->common.hdr_sha1, map2->common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	134 sizeof(map1->common.hdr_sha1));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	135 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	136
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	137 static int imap_msg_map_hdr_cmp(const struct imap_msg_map *map1,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	138 const struct imap_msg_map *map2)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	139 {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	140 return memcmp(map1->common.hdr_sha1, map2->common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	141 sizeof(map1->common.hdr_sha1));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	142 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	143
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	144 struct pop3_hdr_context {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	145 bool have_eoh;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	146 bool stop;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	147 };
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	148
19624 259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	149 static bool header_name_is_valid(const char *name)
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	150 {
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	151 unsigned int i;
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	152
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	153 for (i = 0; name[i] != '\0'; i++) {
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	154 if ((uint8_t)name[i] <= 0x20 \|\| name[i] >= 0x7f)
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	155 return FALSE;
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	156 }
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	157 return TRUE;
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	158 }
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	159
22265 66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	160 static bool header_value_want_skip(const struct message_header_line *hdr)
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	161 {
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	162 for (size_t i = 0; i < hdr->value_len; i++) {
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	163 if (hdr->value[i] != ' ' && hdr->value[i] != '\t')
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	164 return FALSE;
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	165 }
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	166 /* "header: \r\n \r\n" - Zimbra's BODY[HEADER] strips this line away. */
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	167 return TRUE;
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	168 }
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	169
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	170 static void
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	171 pop3_header_filter_callback(struct header_filter_istream *input ATTR_UNUSED,
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	172 struct message_header_line *hdr,
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	173 bool matched, struct pop3_hdr_context ctx)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	174 {
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	175 if (hdr == NULL)
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	176 return;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	177 if (hdr->eoh) {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	178 ctx->have_eoh = TRUE;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	179 if (ctx->stop) {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	180 /* matched is handled differently for eoh by
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	181 istream-header-filter. a design bug I guess.. */
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	182 *matched = FALSE;
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	183 }
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	184 } else {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	185 if (strspn(hdr->name, "\r") == hdr->name_len) {
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	186 /* CR+CR+LF - some servers stop the header processing
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	187 here while others don't. To make sure they can be
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	188 matched correctly we want to stop here entirely. */
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	189 ctx->stop = TRUE;
22451 e2b9cadf91e4 pop3-migration: Strip away invalid header lines. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22450 diff changeset	190 } else if (!hdr->continued && hdr->middle_len == 0) {
e2b9cadf91e4 pop3-migration: Strip away invalid header lines. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22450 diff changeset	191 /* not a valid "key: value" header -
e2b9cadf91e4 pop3-migration: Strip away invalid header lines. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22450 diff changeset	192 Zimbra's BODY[HEADER] strips this line away. */
e2b9cadf91e4 pop3-migration: Strip away invalid header lines. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22450 diff changeset	193 *matched = TRUE;
22265 66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	194 } else if (hdr->continued && header_value_want_skip(hdr)) {
66e02e3235d3 pop3-migration: Drop lines with only spaces or tabs from comparison Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22263 diff changeset	195 *matched = TRUE;
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	196 }
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	197 if (ctx->stop)
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	198 *matched = TRUE;
19624 259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	199 else if (!header_name_is_valid(hdr->name)) {
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	200 /* Yahoo IMAP drops headers with invalid names, while
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	201 Yahoo POP3 preserves them. Drop them all. */
259d971afa5a pop3-migration: Filter out headers with invalid names. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19623 diff changeset	202 *matched = TRUE;
22263 096f0c8b9cb1 pop3-migration: Replace trailing whitespace removal with new header hashing version Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22195 diff changeset	203 }
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	204 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	205 }
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	206
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	207 int pop3_migration_get_hdr_sha1(uint32_t mail_seq, struct istream *input,
20328 7e016f5e8cb4 [LEN] to [static LEN] on some function parameters Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 19693 diff changeset	208 unsigned char sha1_r[STATIC_ARRAY SHA1_RESULTLEN],
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	209 bool *have_eoh_r)
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	210 {
19623 7e08d7ae86be pop3-migration: Convert all non-ASCII in headers to '?' Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19619 diff changeset	211 const unsigned char *data;
19691 69502e2013f3 pop3-migration: Use message_header_hash_more() instead of duplicating the code Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19672 diff changeset	212 size_t size;
20769 efadeff4ab8d lib-mail: message_header_hash_more() now allows input in any slices. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20767 diff changeset	213 struct message_header_hash_context hash_ctx;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	214 struct sha1_ctxt sha1_ctx;
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	215 struct pop3_hdr_context hdr_ctx;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	216
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20769 diff changeset	217 i_zero(&hdr_ctx);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	218 /* hide headers that might change or be different in IMAP vs. POP3 */
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	219 input = i_stream_create_header_filter(input, HEADER_FILTER_HIDE_BODY \|
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	220 HEADER_FILTER_EXCLUDE \| HEADER_FILTER_NO_CR,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	221 hdr_hash_skip_headers,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	222 N_ELEMENTS(hdr_hash_skip_headers),
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	223 pop3_header_filter_callback, &hdr_ctx);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	224
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	225 sha1_init(&sha1_ctx);
22263 096f0c8b9cb1 pop3-migration: Replace trailing whitespace removal with new header hashing version Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22195 diff changeset	226 i_zero(&hash_ctx);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	227 while (i_stream_read_data(input, &data, &size, 0) > 0) {
22263 096f0c8b9cb1 pop3-migration: Replace trailing whitespace removal with new header hashing version Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22195 diff changeset	228 message_header_hash_more(&hash_ctx, &hash_method_sha1, &sha1_ctx,
096f0c8b9cb1 pop3-migration: Replace trailing whitespace removal with new header hashing version Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22195 diff changeset	229 MESSAGE_HEADER_HASH_MAX_VERSION,
19691 69502e2013f3 pop3-migration: Use message_header_hash_more() instead of duplicating the code Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19672 diff changeset	230 data, size);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	231 i_stream_skip(input, size);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	232 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	233 if (input->stream_errno != 0) {
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	234 i_error("pop3_migration: Failed to read header for msg %u: %s",
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	235 mail_seq, i_stream_get_error(input));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	236 i_stream_unref(&input);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	237 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	238 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	239 sha1_result(&sha1_ctx, sha1_r);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	240 i_stream_unref(&input);
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	241
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	242 *have_eoh_r = hdr_ctx.have_eoh;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	243 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	244 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	245
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	246 static unsigned int get_cache_idx(struct mail *mail)
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	247 {
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	248 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(mail->box);
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	249
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	250 if (mbox->cache_field_registered)
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	251 return mbox->cache_field.idx;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	252
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	253 mbox->cache_field.name = "pop3-migration.hdr";
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	254 mbox->cache_field.type = MAIL_CACHE_FIELD_FIXED_SIZE;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	255 mbox->cache_field.field_size = SHA1_RESULTLEN;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	256 mail_cache_register_fields(mail->box->cache, &mbox->cache_field, 1);
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	257 mbox->cache_field_registered = TRUE;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	258 return mbox->cache_field.idx;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	259 }
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	260
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	261 static int
20328 7e016f5e8cb4 [LEN] to [static LEN] on some function parameters Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 19693 diff changeset	262 get_hdr_sha1(struct mail *mail, unsigned char sha1_r[STATIC_ARRAY SHA1_RESULTLEN])
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	263 {
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	264 struct istream *input;
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	265 const char *errstr;
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	266 enum mail_error error;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	267 bool have_eoh;
20730 a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	268 int ret;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	269
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	270 if (mail_get_hdr_stream(mail, NULL, &input) < 0) {
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	271 errstr = mailbox_get_last_internal_error(mail->box, &error);
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	272 i_error("pop3_migration: Failed to get header for msg %u: %s",
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	273 mail->seq, errstr);
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	274 return error == MAIL_ERROR_EXPUNGED ? 0 : -1;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	275 }
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	276 if (pop3_migration_get_hdr_sha1(mail->seq, input, sha1_r, &have_eoh) < 0)
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	277 return -1;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	278 if (have_eoh) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	279 struct index_mail imail = (struct index_mail )mail;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	280
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	281 index_mail_cache_add_idx(imail, get_cache_idx(mail),
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	282 sha1_r, SHA1_RESULTLEN);
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	283 return 1;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	284 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	285
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	286 /* The empty "end of headers" line is missing. Either this means that
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	287 the headers ended unexpectedly (which is ok) or that the remote
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	288 server is buggy. Some servers have problems with
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	289
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	290 1) header line continuations that contain only whitespace and
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	291 2) headers that have no ":". The header gets truncated when such
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	292 line is reached.
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	293
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	294 At least Oracle IMS IMAP FETCH BODY[HEADER] handles 1) by not
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	295 returning the whitespace line and 2) by returning the line but
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	296 truncating the rest. POP3 TOP instead returns the entire header.
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	297 This causes the IMAP and POP3 hashes not to match.
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	298
18909 ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	299 If there's LF+CR+CR+LF in the middle of headers, Courier IMAP's
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	300 FETCH BODY[HEADER] stops after that, but Courier POP3's TOP doesn't.
ab441df52e86 pop3-migration: Truncate header if there's line containing only CR(s). Timo Sirainen <tss@iki.fi> parents: 18908 diff changeset	301
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	302 So we'll try to avoid this by falling back to full FETCH BODY[]
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	303 (and/or RETR) and we'll parse the header ourself from it. This
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	304 should work around any similar bugs in all IMAP/POP3 servers. */
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	305 if (mail_get_stream_because(mail, NULL, NULL, "pop3-migration", &input) < 0) {
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	306 errstr = mailbox_get_last_internal_error(mail->box, &error);
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	307 i_error("pop3_migration: Failed to get body for msg %u: %s",
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	308 mail->seq, errstr);
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	309 return error == MAIL_ERROR_EXPUNGED ? 0 : -1;
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	310 }
20744 cf26a89ceb9a pop3-migration: Avoid unnecessarily using stream's hdr_size. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20730 diff changeset	311 ret = pop3_migration_get_hdr_sha1(mail->seq, input, sha1_r, &have_eoh);
20730 a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	312 if (ret == 0) {
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	313 if (!have_eoh)
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	314 i_warning("pop3_migration: Truncated email with UID %u stored as truncated", mail->uid);
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	315 struct index_mail imail = (struct index_mail )mail;
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	316 index_mail_cache_add_idx(imail, get_cache_idx(mail),
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	317 sha1_r, SHA1_RESULTLEN);
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	318 return 1;
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	319 } else {
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	320 return -1;
a21e038e54c7 pop3-migration-plugin: Add to index after successful retry Aki Tuomi <aki.tuomi@dovecot.fi> parents: 20560 diff changeset	321 }
17122 18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	322 }
18d0ce83bf7a pop3-migration: Work around IMAP/POP3 server bugs which truncate the header too early. Timo Sirainen <tss@iki.fi> parents: 16500 diff changeset	323
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	324 static bool
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	325 get_cached_hdr_sha1(struct mail mail, buffer_t cache_buf,
20328 7e016f5e8cb4 [LEN] to [static LEN] on some function parameters Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 19693 diff changeset	326 unsigned char sha1_r[STATIC_ARRAY SHA1_RESULTLEN])
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	327 {
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	328 struct index_mail imail = (struct index_mail )mail;
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	329
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	330 buffer_set_used_size(cache_buf, 0);
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	331 if (index_mail_cache_lookup_field(imail, cache_buf,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	332 get_cache_idx(mail)) > 0 &&
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	333 cache_buf->used == SHA1_RESULTLEN) {
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	334 memcpy(sha1_r, cache_buf->data, cache_buf->used);
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	335 return TRUE;
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	336 }
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	337 return FALSE;
19635 3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	338 }
3712091cf4d0 pop3-migration: Add calculated header hashes to local cache. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19625 diff changeset	339
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	340 static struct mailbox pop3_mailbox_alloc(struct mail_storage storage)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	341 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	342 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	343 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	344 struct mail_namespace *ns;
21878 e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	345 struct mailbox *box;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	346
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	347 ns = mail_namespace_find(storage->user->namespaces,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	348 mstorage->pop3_box_vname);
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	349 i_assert(ns != NULL);
21878 e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	350 box = mailbox_alloc(ns->list, mstorage->pop3_box_vname,
e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	351 MAILBOX_FLAG_READONLY \| MAILBOX_FLAG_POP3_SESSION);
e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	352 mailbox_set_reason(box, "pop3_migration");
e78a42ead488 global: Add mailbox_set_reason() calls Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21814 diff changeset	353 return box;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	354 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	355
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	356 static int pop3_map_read(struct mail_storage storage, struct mailbox pop3_box)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	357 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	358 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	359 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	360 struct mailbox_transaction_context *t;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	361 struct mail_search_args *search_args;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	362 struct mail_search_context *ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	363 struct mail *mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	364 struct pop3_uidl_map *map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	365 const char *uidl;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	366 uoff_t size = (uoff_t)-1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	367 int ret = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	368
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	369 if (array_is_created(&mstorage->pop3_uidl_map)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	370 /* already read these, just reset the imap_uids */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	371 array_foreach_modifiable(&mstorage->pop3_uidl_map, map)
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	372 map->imap_uid = 0;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	373 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	374 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	375 i_array_init(&mstorage->pop3_uidl_map, 128);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	376
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	377 if (mailbox_sync(pop3_box, 0) < 0) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	378 i_error("pop3_migration: Couldn't sync mailbox %s: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	379 pop3_box->vname, mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	380 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	381 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	382
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	383 t = mailbox_transaction_begin(pop3_box, 0);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	384 search_args = mail_search_build_init();
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	385 mail_search_build_add_all(search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	386 ctx = mailbox_search_init(t, search_args, NULL,
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	387 mstorage->skip_size_check ? 0 :
18912 64568a033fc8 pop3-migration: Fetch physical sizes instead of virtual sizes so pop3c uses LIST Timo Sirainen <tss@iki.fi> parents: 18911 diff changeset	388 MAIL_FETCH_PHYSICAL_SIZE, NULL);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	389 mail_search_args_unref(&search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	390
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	391 while (mailbox_search_next(ctx, &mail)) {
18911 4bebfbb32410 pop3-migration: Use LIST instead of RETRs to get the messages' sizes. Timo Sirainen <tss@iki.fi> parents: 18909 diff changeset	392 /* get the size with LIST instead of RETR */
4bebfbb32410 pop3-migration: Use LIST instead of RETRs to get the messages' sizes. Timo Sirainen <tss@iki.fi> parents: 18909 diff changeset	393 mail->lookup_abort = MAIL_LOOKUP_ABORT_READ_MAIL;
4bebfbb32410 pop3-migration: Use LIST instead of RETRs to get the messages' sizes. Timo Sirainen <tss@iki.fi> parents: 18909 diff changeset	394
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	395 if (mstorage->skip_size_check)
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	396 ;
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	397 else if (mail_get_physical_size(mail, &size) < 0) {
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	398 i_error("pop3_migration: Failed to get size for msg %u: %s",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	399 mail->seq,
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	400 mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	401 ret = -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	402 break;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	403 }
21649 78f38a28086e lib-storage, pop3-migration: Reset lookup_abort before continuing to search Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21390 diff changeset	404 mail->lookup_abort = MAIL_LOOKUP_ABORT_NEVER;
78f38a28086e lib-storage, pop3-migration: Reset lookup_abort before continuing to search Timo Sirainen <timo.sirainen@dovecot.fi> parents: 21390 diff changeset	405
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	406 if (mail_get_special(mail, MAIL_FETCH_UIDL_BACKEND, &uidl) < 0) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	407 i_error("pop3_migration: Failed to get UIDL for msg %u: %s",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	408 mail->seq,
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	409 mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	410 ret = -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	411 break;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	412 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	413 if (*uidl == '\0') {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	414 i_warning("pop3_migration: UIDL for msg %u is empty",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	415 mail->seq);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	416 continue;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	417 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	418
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	419 map = array_append_space(&mstorage->pop3_uidl_map);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	420 map->pop3_seq = mail->seq;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	421 map->pop3_uidl = p_strdup(storage->pool, uidl);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	422 map->size = size;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	423 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	424
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	425 if (mailbox_search_deinit(&ctx) < 0) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	426 i_error("pop3_migration: Failed to search all POP3 mails: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	427 mailbox_get_last_internal_error(pop3_box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	428 ret = -1;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	429 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	430 (void)mailbox_transaction_commit(&t);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	431 return ret;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	432 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	433
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	434 static void
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	435 pop3_map_read_cached_hdr_hashes(struct mailbox_transaction_context *t,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	436 struct mail_search_args *search_args,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	437 struct array *msg_map)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	438 {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	439 struct mail_search_context *ctx;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	440 struct mail *mail;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	441 struct msg_map_common *map;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	442 buffer_t *cache_buf;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	443
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	444 ctx = mailbox_search_init(t, search_args, NULL, 0, NULL);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	445 cache_buf = buffer_create_dynamic(pool_datastack_create(), SHA1_RESULTLEN);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	446
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	447 while (mailbox_search_next(ctx, &mail)) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	448 map = array_idx_modifiable_i(msg_map, mail->seq-1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	449
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	450 if (get_cached_hdr_sha1(mail, cache_buf, map->hdr_sha1))
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	451 map->hdr_sha1_set = TRUE;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	452 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	453
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	454 if (mailbox_search_deinit(&ctx) < 0) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	455 i_warning("pop3_migration: Failed to search all cached POP3 header hashes: %s - ignoring",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	456 mailbox_get_last_internal_error(t->box, NULL));
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	457 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	458 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	459
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	460 static void map_remove_found_seqs(struct mail_search_arg *search_arg,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	461 struct array *msg_map, uint32_t seq1)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	462 {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	463 const struct msg_map_common *map;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	464 uint32_t seq, count = array_count_i(msg_map);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	465
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	466 i_assert(search_arg->type == SEARCH_SEQSET);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	467
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	468 for (seq = seq1; seq <= count; seq++) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	469 map = array_idx_i(msg_map, seq-1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	470 if (map->hdr_sha1_set)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	471 seq_range_array_remove(&search_arg->value.seqset, seq);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	472 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	473 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	474
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	475 static int
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	476 map_read_hdr_hashes(struct mailbox box, struct array msg_map, uint32_t seq1)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	477 {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	478 struct mailbox_transaction_context *t;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	479 struct mail_search_args *search_args;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	480 struct mail_search_context *ctx;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	481 struct mail *mail;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	482 struct msg_map_common *map;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	483 int ret = 0;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	484
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	485 t = mailbox_transaction_begin(box, 0);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	486 /* get all the cached hashes */
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	487 search_args = mail_search_build_init();
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	488 mail_search_build_add_seqset(search_args, seq1, array_count_i(msg_map));
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	489 pop3_map_read_cached_hdr_hashes(t, search_args, msg_map);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	490 /* read all the non-cached hashes. doing this in two passes allows
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	491 us to set wanted_fields=MAIL_FETCH_STREAM_HEADER, which allows
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	492 prefetching to work without downloading all the headers even
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	493 for mails that already are cached. */
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	494 map_remove_found_seqs(search_args->args, msg_map, seq1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	495 ctx = mailbox_search_init(t, search_args, NULL,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	496 MAIL_FETCH_STREAM_HEADER, NULL);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	497 mail_search_args_unref(&search_args);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	498
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	499 while (mailbox_search_next(ctx, &mail)) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	500 map = array_idx_modifiable_i(msg_map, mail->seq-1);
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	501
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	502 if ((ret = get_hdr_sha1(mail, map->hdr_sha1)) < 0) {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	503 ret = -1;
19647 8f2ba9ebc463 pop3-migration: If reading message header for hashing fails, fail immediately. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19646 diff changeset	504 break;
8f2ba9ebc463 pop3-migration: If reading message header for hashing fails, fail immediately. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19646 diff changeset	505 }
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	506 if (ret > 0)
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	507 map->hdr_sha1_set = TRUE;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	508 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	509
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	510 if (mailbox_search_deinit(&ctx) < 0) {
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	511 i_error("pop3_migration: Failed to search all mail headers: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	512 mailbox_get_last_internal_error(box, NULL));
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	513 ret = -1;
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	514 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	515 (void)mailbox_transaction_commit(&t);
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	516 return ret < 0 ? -1 : 0;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	517 }
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	518
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	519 static int
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	520 pop3_map_read_hdr_hashes(struct mail_storage storage, struct mailbox pop3_box,
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	521 unsigned first_seq)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	522 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	523 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	524 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	525
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	526 if (mstorage->pop3_all_hdr_sha1_set)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	527 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	528 if (mstorage->all_mailboxes) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	529 /* we may be matching against multiple mailboxes.
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	530 read all the hashes only once. */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	531 first_seq = 1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	532 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	533
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	534 if (map_read_hdr_hashes(pop3_box, &mstorage->pop3_uidl_map.arr,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	535 first_seq) < 0)
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	536 return -1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	537
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	538 if (first_seq == 1)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	539 mstorage->pop3_all_hdr_sha1_set = TRUE;
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	540 return 0;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	541 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	542
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	543 static int imap_map_read(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	544 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	545 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	546 struct index_mailbox_context *ibox = INDEX_STORAGE_CONTEXT(box);
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	547 struct pop3_migration_mail_storage *mstorage =
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	548 POP3_MIGRATION_CONTEXT(box->storage);
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	549 const unsigned int uidl_cache_idx =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	550 ibox->cache_fields[MAIL_CACHE_POP3_UIDL].idx;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	551 struct mailbox_status status;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	552 struct mailbox_transaction_context *t;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	553 struct mail_search_args *search_args;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	554 struct mail_search_context *ctx;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	555 struct mail *mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	556 struct imap_msg_map *map;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	557 uoff_t psize = (uoff_t)-1;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	558 string_t *uidl;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	559 int ret = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	560
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	561 mailbox_get_open_status(box, STATUS_MESSAGES, &status);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	562
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	563 i_assert(!array_is_created(&mbox->imap_msg_map));
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	564 p_array_init(&mbox->imap_msg_map, box->pool, status.messages);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	565
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	566 t = mailbox_transaction_begin(box, 0);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	567 search_args = mail_search_build_init();
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	568 mail_search_build_add_all(search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	569 ctx = mailbox_search_init(t, search_args, NULL,
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	570 mstorage->skip_size_check ? 0 :
14402 87cd04f35f82 pop3-replication: Don't request virtual size from IMAP mailbox, we don't need it. Timo Sirainen <tss@iki.fi> parents: 14401 diff changeset	571 MAIL_FETCH_PHYSICAL_SIZE, NULL);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	572 mail_search_args_unref(&search_args);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	573
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	574 uidl = t_str_new(64);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	575 while (mailbox_search_next(ctx, &mail)) {
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	576 if (mstorage->skip_size_check)
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	577 ;
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	578 else if (mail_get_physical_size(mail, &psize) < 0) {
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	579 i_error("pop3_migration: Failed to get psize for imap uid %u: %s",
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	580 mail->uid,
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	581 mailbox_get_last_internal_error(box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	582 ret = -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	583 break;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	584 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	585
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	586 if (!mstorage->skip_uidl_cache) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	587 str_truncate(uidl, 0);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	588 (void)mail_cache_lookup_field(mail->transaction->cache_view,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	589 uidl, mail->seq, uidl_cache_idx);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	590 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	591
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	592 map = array_append_space(&mbox->imap_msg_map);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	593 map->uid = mail->uid;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	594 map->psize = psize;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	595 map->pop3_uidl = p_strdup_empty(box->pool, str_c(uidl));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	596 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	597
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	598 if (mailbox_search_deinit(&ctx) < 0) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	599 i_error("pop3_migration: Failed to search all IMAP mails: %s",
21814 759962e70148 global: Log internal storage error on failure Martti Rannanjärvi <martti.rannanjarvi@dovecot.fi> parents: 21649 diff changeset	600 mailbox_get_last_internal_error(box, NULL));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	601 ret = -1;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	602 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	603 (void)mailbox_transaction_commit(&t);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	604 return ret;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	605 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	606
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	607 static int imap_map_read_hdr_hashes(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	608 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	609 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	610
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	611 return map_read_hdr_hashes(box, &mbox->imap_msg_map.arr,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	612 mbox->first_unfound_idx+1);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	613 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	614
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	615 static void pop3_uidl_assign_cached(struct mailbox *box)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	616 {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	617 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	618 struct pop3_migration_mail_storage *mstorage =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	619 POP3_MIGRATION_CONTEXT(box->storage);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	620 struct pop3_uidl_map *pop3_map;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	621 struct imap_msg_map *imap_map;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	622 unsigned int imap_idx, pop3_idx, pop3_count, imap_count;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	623 int ret;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	624
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	625 if (mstorage->skip_uidl_cache)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	626 return;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	627
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	628 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_uidl_cmp);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	629 array_sort(&mbox->imap_msg_map, imap_msg_map_uidl_cmp);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	630
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	631 pop3_map = array_get_modifiable(&mstorage->pop3_uidl_map, &pop3_count);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	632 imap_map = array_get_modifiable(&mbox->imap_msg_map, &imap_count);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	633
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	634 /* see if we can match the messages using sizes */
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	635 for (imap_idx = pop3_idx = 0; imap_idx < imap_count; imap_idx++) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	636 if (imap_map[imap_idx].pop3_uidl == NULL)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	637 continue;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	638
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	639 ret = 1;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	640 for (; pop3_idx < pop3_count; pop3_idx++) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	641 ret = strcmp(imap_map[imap_idx].pop3_uidl,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	642 pop3_map[pop3_idx].pop3_uidl);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	643 if (ret >= 0)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	644 break;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	645 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	646 if (ret == 0) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	647 imap_map[imap_idx].pop3_seq =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	648 pop3_map[pop3_idx].pop3_seq;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	649 pop3_map[pop3_idx].imap_uid = imap_map[imap_idx].uid;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	650 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	651 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	652 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	653
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	654 static bool pop3_uidl_assign_by_size(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	655 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	656 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	657 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	658 POP3_MIGRATION_CONTEXT(box->storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	659 struct pop3_uidl_map *pop3_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	660 struct imap_msg_map *imap_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	661 unsigned int i, pop3_count, imap_count, count;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	662 unsigned int size_match = 0, uidl_match = 0;
19644 defab5b9af2a pop3-migration: Don't even try to match message sizes with pop3_migration_skip_size_check=yes Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19640 diff changeset	663
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	664 pop3_map = array_get_modifiable(&mstorage->pop3_uidl_map, &pop3_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	665 imap_map = array_get_modifiable(&mbox->imap_msg_map, &imap_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	666 count = I_MIN(pop3_count, imap_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	667
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	668 /* see if we can match the messages using sizes */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	669 for (i = 0; i < count; i++) {
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	670 if (imap_map[i].pop3_uidl != NULL) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	671 /* some of the UIDLs were already found cached. */
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	672 if (strcmp(pop3_map[i].pop3_uidl, imap_map[i].pop3_uidl) == 0) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	673 uidl_match++;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	674 continue;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	675 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	676 /* mismatch - can't trust the sizes */
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	677 break;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	678 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	679
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	680 if (pop3_map[i].size != imap_map[i].psize \|\|
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	681 mstorage->skip_size_check)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	682 break;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	683 if (i+1 < count && pop3_map[i].size == pop3_map[i+1].size) {
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	684 /* two messages with same size, don't trust them */
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	685 break;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	686 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	687
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	688 size_match++;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	689 pop3_map[i].imap_uid = imap_map[i].uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	690 imap_map[i].pop3_uidl = pop3_map[i].pop3_uidl;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	691 imap_map[i].pop3_seq = pop3_map[i].pop3_seq;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	692 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	693 mbox->first_unfound_idx = i;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	694 if (box->storage->user->mail_debug) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	695 i_debug("pop3_migration: cached uidls=%u, size matches=%u, total=%u",
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	696 uidl_match, size_match, count);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	697 }
22453 6eb5a6930614 pop3-migration: Fail if all IMAP mails were matched by size, but POP3 had extra mails Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22452 diff changeset	698 return i == count && imap_count == pop3_count;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	699 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	700
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	701 static int
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	702 pop3_uidl_assign_by_hdr_hash(struct mailbox box, struct mailbox pop3_box)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	703 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	704 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	705 POP3_MIGRATION_CONTEXT(box->storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	706 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	707 struct pop3_uidl_map *pop3_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	708 struct imap_msg_map *imap_map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	709 unsigned int pop3_idx, imap_idx, pop3_count, imap_count;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	710 unsigned int first_seq, missing_uids_count;
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	711 uint32_t first_missing_idx = 0, first_missing_seq = (uint32_t)-1;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	712 int ret;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	713
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	714 first_seq = mbox->first_unfound_idx+1;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	715 if (pop3_map_read_hdr_hashes(box->storage, pop3_box, first_seq) < 0 \|\|
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	716 imap_map_read_hdr_hashes(box) < 0)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	717 return -1;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	718
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	719 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_hdr_cmp);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	720 array_sort(&mbox->imap_msg_map, imap_msg_map_hdr_cmp);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	721
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	722 pop3_map = array_get_modifiable(&mstorage->pop3_uidl_map, &pop3_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	723 imap_map = array_get_modifiable(&mbox->imap_msg_map, &imap_count);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	724
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	725 pop3_idx = imap_idx = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	726 while (pop3_idx < pop3_count && imap_idx < imap_count) {
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	727 if (!pop3_map[pop3_idx].common.hdr_sha1_set \|\|
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	728 pop3_map[pop3_idx].imap_uid != 0) {
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	729 pop3_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	730 continue;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	731 }
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	732 if (!imap_map[imap_idx].common.hdr_sha1_set \|\|
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	733 imap_map[imap_idx].pop3_uidl != NULL) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	734 imap_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	735 continue;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	736 }
19640 b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	737 ret = memcmp(pop3_map[pop3_idx].common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	738 imap_map[imap_idx].common.hdr_sha1,
b6dafead7c40 pop3-migration: Cached header hashes weren't actually being used for imapc. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19635 diff changeset	739 sizeof(pop3_map[pop3_idx].common.hdr_sha1));
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	740 if (ret < 0)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	741 pop3_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	742 else if (ret > 0)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	743 imap_idx++;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	744 else {
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	745 pop3_map[pop3_idx].imap_uid = imap_map[imap_idx].uid;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	746 imap_map[imap_idx].pop3_uidl =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	747 pop3_map[pop3_idx].pop3_uidl;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	748 imap_map[imap_idx].pop3_seq =
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	749 pop3_map[pop3_idx].pop3_seq;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	750 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	751 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	752 missing_uids_count = 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	753 for (pop3_idx = 0; pop3_idx < pop3_count; pop3_idx++) {
19693 dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	754 if (pop3_map[pop3_idx].imap_uid != 0) {
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	755 /* matched */
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	756 } else if (!pop3_map[pop3_idx].common.hdr_sha1_set) {
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	757 /* we treated this mail as expunged - ignore */
dacdabcd8085 pop3-migration: Ignore mails returned as expunged by pop3c. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19691 diff changeset	758 } else {
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	759 uint32_t seq = pop3_map[pop3_idx].pop3_seq;
6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	760 if (first_missing_seq > seq) {
6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	761 first_missing_seq = seq;
19617 7b0a093df334 pop3-migration: When logging the first missing POP3 mail, it logged a wrong mail Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19616 diff changeset	762 first_missing_idx = pop3_idx;
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	763 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	764 missing_uids_count++;
18908 b51dfee18fd2 pop3-migration: Show the first message's number and UIDL which wasn't found from IMAP. Timo Sirainen <tss@iki.fi> parents: 18624 diff changeset	765 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	766 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	767 if (missing_uids_count > 0 && !mstorage->all_mailboxes) {
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	768 string_t *str = t_str_new(128);
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	769 bool all_imap_mails_found = FALSE;
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	770
af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	771 str_printfa(str, "pop3_migration: %u POP3 messages have no "
af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	772 "matching IMAP messages (first POP3 msg %u UIDL %s)",
20767 6140ec208b4b pop3-migration: "first POP3 msg" warning didn't actually show the first one. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20746 diff changeset	773 missing_uids_count, first_missing_seq,
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	774 pop3_map[first_missing_idx].pop3_uidl);
19619 c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	775 if (imap_count + missing_uids_count == pop3_count) {
c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	776 str_append(str, " - all IMAP messages were found "
c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	777 "(POP3 contains more than IMAP INBOX - you may want to set pop3_migration_all_mailboxes=yes)");
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	778 all_imap_mails_found = TRUE;
19619 c1ba198e8b50 pop3-migration: If we matched all the IMAP (but not all POP3) messages, log about it. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19618 diff changeset	779 }
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	780 if (all_imap_mails_found && mstorage->ignore_extra_uidls) {
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	781 /* pop3 had more mails than imap. maybe it was just
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	782 that a new mail was just delivered. */
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	783 } else if (!mstorage->ignore_missing_uidls) {
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	784 str_append(str, " - set pop3_migration_ignore_missing_uidls=yes");
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	785 if (all_imap_mails_found)
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	786 str_append(str, " or pop3_migration_ignore_extra_uidls=yes");
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	787 i_error("%s to continue anyway", str_c(str));
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	788 return -1;
1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	789 }
19616 af9448bcbfd9 pop3-migration: When failing because some messages couldn't be matched, show the first message's number and UIDL Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19615 diff changeset	790 i_warning("%s", str_c(str));
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	791 } else if (box->storage->user->mail_debug) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	792 i_debug("pop3_migration: %u mails matched by headers", pop3_count);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	793 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	794 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_pop3_seq_cmp);
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	795 array_sort(&mbox->imap_msg_map, imap_msg_map_uid_cmp);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	796 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	797 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	798
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	799 static void imap_uidls_add_to_cache(struct mailbox *box)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	800 {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	801 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	802 struct mailbox_transaction_context *t;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	803 struct mail *mail;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	804 struct index_mail *imail;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	805 struct imap_msg_map *imap_map;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	806 unsigned int i, count;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	807 unsigned int field_idx;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	808
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	809 t = mailbox_transaction_begin(box, 0);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	810 mail = mail_alloc(t, 0, NULL);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	811 imail = (struct index_mail *)mail;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	812 field_idx = imail->ibox->cache_fields[MAIL_CACHE_POP3_UIDL].idx;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	813
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	814 imap_map = array_get_modifiable(&mbox->imap_msg_map, &count);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	815 for (i = 0; i < count; i++) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	816 if (imap_map[i].pop3_uidl == NULL)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	817 continue;
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	818
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	819 if (!mail_set_uid(mail, imap_map[i].uid))
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	820 i_unreached();
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	821 if (mail_cache_field_can_add(t->cache_trans, mail->seq, field_idx)) {
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	822 index_mail_cache_add_idx(imail, field_idx,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	823 imap_map[i].pop3_uidl, strlen(imap_map[i].pop3_uidl)+1);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	824 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	825 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	826 mail_free(&mail);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	827 (void)mailbox_transaction_commit(&t);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	828 }
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	829
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	830 static int pop3_migration_uidl_sync(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	831 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	832 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	833 struct pop3_migration_mail_storage *mstorage =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	834 POP3_MIGRATION_CONTEXT(box->storage);
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	835 struct mailbox *pop3_box;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	836
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	837 pop3_box = pop3_mailbox_alloc(box->storage);
14835 ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	838 /* the POP3 server isn't connected to yet. handle all IMAP traffic
ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	839 first before connecting, so POP3 server won't disconnect us due to
ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	840 idling. */
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	841 if (imap_map_read(box) < 0 \|\|
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	842 pop3_map_read(box->storage, pop3_box) < 0) {
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	843 mailbox_free(&pop3_box);
14835 ab6a4455b27d pop3-migration: Avoid disconnection from POP3 server due to idling. Timo Sirainen <tss@iki.fi> parents: 14423 diff changeset	844 return -1;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	845 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	846
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	847 pop3_uidl_assign_cached(box);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	848
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	849 array_sort(&mstorage->pop3_uidl_map, pop3_uidl_map_pop3_seq_cmp);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	850 array_sort(&mbox->imap_msg_map, imap_msg_map_uid_cmp);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	851
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	852 if (!pop3_uidl_assign_by_size(box)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	853 /* everything wasn't assigned, figure out the rest with
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	854 header hashes */
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	855 if (pop3_uidl_assign_by_hdr_hash(box, pop3_box) < 0) {
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	856 mailbox_free(&pop3_box);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	857 return -1;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	858 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	859 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	860
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	861 if (!mstorage->skip_uidl_cache)
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	862 imap_uidls_add_to_cache(box);
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	863
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	864 mbox->uidl_synced = TRUE;
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	865 mailbox_free(&pop3_box);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	866 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	867 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	868
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	869 static int pop3_migration_uidl_sync_if_needed(struct mailbox *box)
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	870 {
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	871 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(box);
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	872
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	873 if (mbox->uidl_synced)
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	874 return 0;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	875
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	876 if (mbox->uidl_sync_failed \|\|
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	877 pop3_migration_uidl_sync(box) < 0) {
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	878 mbox->uidl_sync_failed = TRUE;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	879 mail_storage_set_error(box->storage, MAIL_ERROR_TEMP,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	880 "POP3 UIDLs couldn't be synced");
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	881 return -1;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	882 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	883 return 0;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	884 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	885
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	886 static int
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	887 pop3_migration_get_special(struct mail *_mail, enum mail_fetch_field field,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	888 const char **value_r)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	889 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	890 struct mail_private mail = (struct mail_private )_mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	891 union mail_module_context *mmail = POP3_MIGRATION_MAIL_CONTEXT(mail);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	892 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(_mail->box);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	893 struct imap_msg_map map_key, *map;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	894
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	895 if (field == MAIL_FETCH_UIDL_BACKEND \|\|
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	896 field == MAIL_FETCH_POP3_ORDER) {
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	897 if (pop3_migration_uidl_sync_if_needed(_mail->box) < 0)
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	898 return -1;
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	899
21389 59437f8764c6 global: Replaced all instances of memset(p, 0, sizeof(p)) with the new i_zero() macro. Stephan Bosch <stephan.bosch@dovecot.fi>* parents: 20769 diff changeset	900 i_zero(&map_key);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	901 map_key.uid = _mail->uid;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	902 map = array_bsearch(&mbox->imap_msg_map, &map_key,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	903 imap_msg_map_uid_cmp);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	904 if (map != NULL && map->pop3_uidl != NULL) {
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	905 if (field == MAIL_FETCH_UIDL_BACKEND)
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	906 *value_r = map->pop3_uidl;
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	907 else
6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	908 *value_r = t_strdup_printf("%u", map->pop3_seq);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	909 return 0;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	910 }
14401 6df2235fd6af pop3-migration: Migrate also POP3 ordering. Timo Sirainen <tss@iki.fi> parents: 14398 diff changeset	911 /* not found from POP3 server, fallback to default */
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	912 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	913 return mmail->super.get_special(_mail, field, value_r);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	914 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	915
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	916 static void pop3_migration_mail_allocated(struct mail *_mail)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	917 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	918 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	919 POP3_MIGRATION_CONTEXT(_mail->box->storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	920 struct mail_private mail = (struct mail_private )_mail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	921 struct mail_vfuncs *v = mail->vlast;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	922 union mail_module_context *mmail;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	923 struct mail_namespace *ns;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	924
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	925 if (mstorage == NULL \|\|
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	926 (!mstorage->all_mailboxes && !_mail->box->inbox_user)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	927 /* assigns UIDLs only for INBOX */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	928 return;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	929 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	930
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	931 ns = mail_namespace_find(_mail->box->storage->user->namespaces,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	932 mstorage->pop3_box_vname);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	933 if (ns == mailbox_get_namespace(_mail->box)) {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	934 /* we're accessing the pop3-migration namespace itself */
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	935 return;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	936 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	937
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	938 mmail = p_new(mail->pool, union mail_module_context, 1);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	939 mmail->super = *v;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	940 mail->vlast = &mmail->super;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	941
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	942 v->get_special = pop3_migration_get_special;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	943 MODULE_CONTEXT_SET_SELF(mail, pop3_migration_mail_module, mmail);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	944 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	945
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	946 static struct mail_search_context *
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	947 pop3_migration_mailbox_search_init(struct mailbox_transaction_context *t,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	948 struct mail_search_args *args,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	949 const enum mail_sort_type *sort_program,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	950 enum mail_fetch_field wanted_fields,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	951 struct mailbox_header_lookup_ctx *wanted_headers)
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	952 {
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	953 struct pop3_migration_mailbox *mbox = POP3_MIGRATION_CONTEXT(t->box);
19672 e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	954 struct pop3_migration_mail_storage *mstorage =
e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	955 POP3_MIGRATION_CONTEXT(t->box->storage);
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	956
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	957 if ((wanted_fields & (MAIL_FETCH_UIDL_BACKEND \|
19672 e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	958 MAIL_FETCH_POP3_ORDER)) != 0 &&
e7849490acd9 pop3-migration: Fix to 5407a86 - don't sync POP3 UIDLs when we don't need to Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19654 diff changeset	959 (mstorage->all_mailboxes \|\| t->box->inbox_user)) {
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	960 /* Start POP3 UIDL syncing before the search, so we'll do it
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	961 before we start sending any FETCH BODY[]s to IMAP. It
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	962 shouldn't matter much, except this works around a bug in
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	963 Yahoo IMAP where it sometimes breaks its state when doing
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	964 a FETCH BODY[] followed by FETCH BODY[HEADER].. */
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	965 (void)pop3_migration_uidl_sync_if_needed(t->box);
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	966 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	967
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	968 return mbox->module_ctx.super.search_init(t, args, sort_program,
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	969 wanted_fields, wanted_headers);
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	970 }
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	971
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	972 static void pop3_migration_mailbox_allocated(struct mailbox *box)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	973 {
20746 6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	974 struct pop3_migration_mail_storage *mstorage =
6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	975 POP3_MIGRATION_CONTEXT(box->storage);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	976 struct mailbox_vfuncs *v = box->vlast;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	977 struct pop3_migration_mailbox *mbox;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	978
20746 6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	979 if (mstorage == NULL)
6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	980 return;
6c827dbef85e pop3-migration: Fixed crash when not specifying pop3_migration_mailbox setting. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 20744 diff changeset	981
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	982 mbox = p_new(box->pool, struct pop3_migration_mailbox, 1);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	983 mbox->module_ctx.super = *v;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	984 box->vlast = &mbox->module_ctx.super;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	985
19654 ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	986 v->search_init = pop3_migration_mailbox_search_init;
ff56bb2c713c pop3-migration: Perform UIDL syncing a bit earlier to work around Yahoo IMAP bug. Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19647 diff changeset	987
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	988 MODULE_CONTEXT_SET(box, pop3_migration_storage_module, mbox);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	989 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	990
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	991 static void pop3_migration_mail_storage_destroy(struct mail_storage *storage)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	992 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	993 struct pop3_migration_mail_storage *mstorage =
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	994 POP3_MIGRATION_CONTEXT(storage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	995
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	996 if (array_is_created(&mstorage->pop3_uidl_map))
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	997 array_free(&mstorage->pop3_uidl_map);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	998
15272 ca10d2e8d8e3 lib-storage: Moved mail attribute dict to struct mail_storage. Timo Sirainen <tss@iki.fi> parents: 15199 diff changeset	999 mstorage->module_ctx.super.destroy(storage);
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1000 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1001
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1002 static void pop3_migration_mail_storage_created(struct mail_storage *storage)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1003 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1004 struct pop3_migration_mail_storage *mstorage;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1005 struct mail_storage_vfuncs *v = storage->vlast;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1006 const char *pop3_box_vname;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1007
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1008 pop3_box_vname = mail_user_plugin_getenv(storage->user,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1009 "pop3_migration_mailbox");
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	1010 if (pop3_box_vname == NULL) {
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	1011 if (storage->user->mail_debug)
3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	1012 i_debug("pop3_migration: No pop3_migration_mailbox setting - disabled");
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1013 return;
18624 3903badc4ee0 pop3-migration: Added more debug and error logging. Timo Sirainen <tss@iki.fi> parents: 18137 diff changeset	1014 }
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1015
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1016 mstorage = p_new(storage->pool, struct pop3_migration_mail_storage, 1);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1017 mstorage->module_ctx.super = *v;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1018 storage->vlast = &mstorage->module_ctx.super;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1019 v->destroy = pop3_migration_mail_storage_destroy;
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1020
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1021 mstorage->pop3_box_vname = p_strdup(storage->pool, pop3_box_vname);
16500 c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	1022 mstorage->all_mailboxes =
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	1023 mail_user_plugin_getenv(storage->user,
c903fbcbf5d2 pop3-migration: struct mailbox must be freed before mail_storage is destroyed. Timo Sirainen <tss@iki.fi> parents: 15715 diff changeset	1024 "pop3_migration_all_mailboxes") != NULL;
17124 1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	1025 mstorage->ignore_missing_uidls =
1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	1026 mail_user_plugin_getenv(storage->user,
1c298212ab70 pop3-migration: If UIDLs can't be set, fail unless pop3_migration_ignore_missing_uidls=yes. Timo Sirainen <tss@iki.fi> parents: 17123 diff changeset	1027 "pop3_migration_ignore_missing_uidls") != NULL;
22452 566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	1028 mstorage->ignore_extra_uidls =
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	1029 mail_user_plugin_getenv(storage->user,
566bd6b3b348 pop3-migration: Add pop3_migration_ignore_extra_uidls=yes setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22451 diff changeset	1030 "pop3_migration_ignore_extra_uidls") != NULL;
19615 152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	1031 mstorage->skip_size_check =
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	1032 mail_user_plugin_getenv(storage->user,
152ed3cd2bad pop3-migration: Added pop3_migration_skip_size_check setting Timo Sirainen <timo.sirainen@dovecot.fi> parents: 19614 diff changeset	1033 "pop3_migration_skip_size_check") != NULL;
22450 1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	1034 mstorage->skip_uidl_cache =
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	1035 mail_user_plugin_getenv(storage->user,
1ff8f91dafba pop3-migration: Try to assign UIDLs based on dovecot.index.cache Timo Sirainen <timo.sirainen@dovecot.fi> parents: 22449 diff changeset	1036 "pop3_migration_skip_uidl_cache") != NULL;
14398 78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1037
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1038 MODULE_CONTEXT_SET(storage, pop3_migration_storage_module, mstorage);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1039 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1040
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1041 static struct mail_storage_hooks pop3_migration_mail_storage_hooks = {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1042 .mail_allocated = pop3_migration_mail_allocated,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1043 .mailbox_allocated = pop3_migration_mailbox_allocated,
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1044 .mail_storage_created = pop3_migration_mail_storage_created
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1045 };
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1046
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1047 void pop3_migration_plugin_init(struct module *module)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1048 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1049 mail_storage_hooks_add(module, &pop3_migration_mail_storage_hooks);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1050 }
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1051
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1052 void pop3_migration_plugin_deinit(void)
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1053 {
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1054 mail_storage_hooks_remove(&pop3_migration_mail_storage_hooks);
78317179b4af Added pop3-migration plugin for getting POP3 UIDLs from POP3 server. Timo Sirainen <tss@iki.fi> parents: diff changeset	1055 }

Mercurial > dovecot > core-2.2

annotate src/plugins/pop3-migration/pop3-migration-plugin.c @ 22713:cb108f786fb4